Deep learning based automatic video content label generation method

Abstract

The present disclosure provides a deep learning-based video content automatic label generation method, which comprises time sequence segmentation processing of an input teaching video, extraction of basic features of each video segment, and construction of a standardized segment feature sequence; a pre-trained deep neural network is used to determine the semantic dominant object of each video segment based on the segment feature sequence to generate a dominant object label sequence; a multi-level semantic segmentation is performed on the video timeline to obtain a series of semantic units with consistent internal dominant objects and continuous semantics; for each semantic unit, its teaching activity label is predicted, and the label is bound and output with the corresponding semantic dominant object and time interval. The method can accurately perceive and explicitly process the frequent switching of semantic dominant objects in the teaching video, and solve the defects of mixed label semantics and inability to accurately reflect the real structure of the teaching process caused by neglecting subject transformation.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and artificial intelligence, and in particular to a method for automatically generating video content tags based on deep learning. Background Technology

[0002] With the popularization of online education and digital teaching, universities have generated a massive amount of classroom and experimental teaching video resources. To facilitate subsequent teaching analysis, resource retrieval, and personalized learning, it is necessary to automatically generate accurate and structured semantic tags for these video contents. Existing automatic video tagging technologies typically rely on the overall or segmented analysis of video content, inferring semantic tags by recognizing information such as scenes, objects, actions, or speech. However, these general methods face significant challenges when applied to specific scenarios like university classrooms.

[0003] Classroom teaching is a dynamic and interactive process, where semantic dominance frequently and rapidly shifts between different subjects, such as teachers lecturing, students operating, and teachers and students asking and answering questions. Current mainstream solutions, when performing video semantic analysis, typically assume, or implicitly assume, that the dominant entity within a video segment or semantic unit is continuous and stable. Their modeling process tends to aggregate temporally adjacent segments with potentially continuous visual features into the same semantic unit. This leads to a situation where, in actual teaching, when the subject in front of the camera rapidly switches from teacher to student, or from theoretical explanation to experimental demonstration, existing technologies cannot accurately capture and respect this fundamental shift in semantic dominance. The generated video tags often mix unequal semantics such as lecturing, experimental operation, and classroom interaction, which are not at the same level and are dominated by different subjects. This makes the tags unable to accurately depict the actual roles played by each participant at different times during the teaching process and the boundaries of their specific activities. Consequently, teaching analysis and resource retrieval based on such tags suffer from systematic distortion, significantly reducing their practical value.

[0004] Therefore, there is an urgent need for an automatic tag generation method that can accurately perceive and explicitly handle the problem of frequent switching of semantic dominant objects in teaching videos, in order to solve the core defect of existing technologies that ignore subject changes, resulting in semantically mixed tags and failing to accurately reflect the true structure of the teaching process. Summary of the Invention

[0005] In view of this, in order to solve the problems caused by the existing technology, this application provides a method for automatic video content tag generation based on deep learning.

[0006] In a first aspect, this disclosure provides a method for automatically generating video content tags based on deep learning, the method comprising: S1. Perform time-series fragmentation processing on the input teaching video, extract the basic features of each video segment, and construct a standardized segment feature sequence; S2. Using a pre-trained deep neural network, determine the semantic dominant object of each video segment based on the segment feature sequence, and generate a dominant object tag sequence; S3. Based on the dominant object label sequence, perform multi-level semantic segmentation on the video timeline. The semantic segmentation includes primary segmentation of the timeline based on the dominant object label and its probability distribution to obtain continuous semantic segments with consistent dominant objects; and fine-grained segmentation within the continuous semantic segments of the same dominant object based on the changes in visual and action features to obtain a series of semantic units. S4. For each semantic unit, predict its teaching activity label, and bind and output the label with the corresponding semantic dominant object and time interval.

[0007] Optionally, S1 includes: The instructional videos are divided into consecutive short segments according to fixed time windows; Standardize the spatial scale and pixel values ​​of each video segment; The average brightness and motion intensity features of each standardized segment are extracted to form the segment feature sequence.

[0008] Optionally, S2 includes: Each video segment's image frames are constructed as temporal tensors and input into a deep neural network to predict the probability distribution of their belonging to each preset dominant object category; The initial dominant object label and determination confidence level for each segment are generated based on the probability distribution. The initial dominant object markers are subjected to time consistency correction to generate the final dominant object marker sequence.

[0009] Optionally, the time consistency correction includes: For segments whose confidence level is below the threshold, smoothing is performed based on the probability distribution of multiple segments in their time neighborhood.

[0010] Optionally, the semantic segmentation further includes: After the initial segmentation, semantic segments shorter than a preset threshold are suppressed and merged into adjacent semantic segments with the same dominant object.

[0011] Optionally, the primary segmentation includes: When adjacent segments have different dominant object labels, or the labels are the same but the change intensity of the dominant object probability distribution exceeds a preset threshold, the segments are split.

[0012] Optionally, S4 includes: The visual content of each semantic unit is organized into a fixed-length tensor and input into the label generation model to obtain the probability that it belongs to various types of teaching activity labels. Based on a preset confidence threshold, the valid teaching activity label for the semantic unit is determined from the probability. The effective teaching activity tags of each semantic unit are bound to its semantic dominant object and time start and end points to form structured tag entries; Tag entries that are adjacent in time and have the same tag as the dominant object are merged to form the final output video tag set.

[0013] Optionally, the semantically dominant object includes the teacher, the student, and the blackboard or projector.

[0014] In a second aspect, this disclosure provides an electronic device including a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to implement the method of the first aspect described above.

[0015] Thirdly, this disclosure provides a computer storage medium storing a computer program that, when executed, implements the method described in the first aspect.

[0016] The beneficial effects of this disclosure are that, compared with the prior art, this disclosure has the following advantages: 1) Addressing the issue that existing technologies, by assuming the semantically dominant entity within video segments is continuous and stable, fail to accurately capture the high-frequency, rapid alternation of dominance among multiple subjects such as teachers, students, blackboards, or projectors in teaching scenarios, this paper constructs segment temporal tensors and utilizes pre-trained deep neural networks to perform end-to-end subject probability prediction for each video segment. Initial judgment results are then smoothly corrected based on temporal consistency, resulting in a labeled sequence that accurately reflects the actual semantic dominance in each time period. This allows the system to explicitly perceive and respect the inherent subject switching during teaching, laying a fundamental foundation for generating structurally realistic labels.

[0017] 2) To address the problem that existing technologies, by neglecting subject transformation, generate video tags that are a mixture of different levels of semantically inequivalent meanings dominated by different subjects, making the tags unable to depict the real teaching process, a semantic unit segmentation method based on the consistency of the dominant object is proposed. This method uses the dominant object label as a hard constraint to perform initial segmentation of the video timeline, ensuring that the stages dominated by different subjects are strictly separated. Then, noise fragmentation is eliminated through short-segment suppression, and fine-grained segmentation is performed within the same subject based on visual and action features. Finally, the video is divided into a series of semantically pure and internally coherent semantic units, each containing only one or more closely related teaching activities under a single dominant object, fundamentally eliminating semantic mixing in the tags.

[0018] 3) To address the systematic distortion in teaching analysis and resource retrieval based on tags generated by existing technologies, a tag generation and output method based on dominant object semantic units is proposed. In the tag generation stage, teaching activity tags are predicted individually for each segmented semantic unit, and a dominant object semantic consistency constraint is applied to ensure logical self-consistency of the tags. In the output stage, the determined tags are bound to the dominant object and a precise time interval, and adjacent tags of the same type can be merged to form a concise structured summary. The final output is a tag set that unambiguously describes when, by whom, and what kind of teaching activity is being conducted. This allows video retrieval, teaching process statistics, and analysis based on this tag set to accurately reflect the actual teaching process, significantly improving the practical value and reliability of automated teaching video analysis. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 A flowchart of the video content automatic tag generation method based on deep learning provided in this disclosure embodiment is shown; Figure 2 A flowchart illustrating the semantically dominant object determination and tag sequence generation provided in an embodiment of this disclosure is shown. Figure 3 A flowchart illustrating the multi-level semantic segmentation process provided in an embodiment of this disclosure is shown.

[0021] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0022] The present disclosure will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solutions of the present disclosure more clearly, and should not be used to limit the scope of protection of the present disclosure.

[0023] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0025] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0026] Figure 1 A flowchart of the video content automatic tag generation method based on deep learning provided in the embodiments of this disclosure is shown below. Figure 1 As shown, the process may include the following steps: S1: Perform time-series fragmentation processing on the input teaching video, extract the basic features of each video segment, and construct a standardized segment feature sequence.

[0027] The input raw classroom or experimental teaching videos are preprocessed to convert them into a series of temporally continuous, uniformly formatted video clips carrying basic visual and motion information, laying the data foundation for subsequent deep semantic analysis. This is achieved through the following sub-steps.

[0028] S1.1: Perform timeline analysis on the original teaching video and perform discrete frame sampling at a fixed frame rate to generate an image frame sequence with time-series labels.

[0029] Acquire the raw video stream of classroom or experimental teaching to be processed. This video stream is represented in the system as a continuous function of time. At the same time, the frame rate r is read from the metadata of the video file. This parameter represents the number of image frames contained per second, with a common range of 15 to 120 frames per second.

[0030] The core of the processing is to convert the continuous video stream into a discrete sequence of image frames with time-stamped labels. This ensures that all subsequent operations are based on a unique and accurate time reference, avoiding timestamp drift errors caused by video container formats or player decoding. This is achieved by uniformly discretely sampling at the inherent frame rate *r* of the video. Specifically, starting from the beginning of the video, one frame is captured every 1 / *r* seconds, resulting in a time-ordered frame sequence. Where N is the total number of frames in the video. This represents the k-th frame image in the sequence.

[0031] Assigning a precise timestamp to each frame is crucial. The timestamp corresponding to the k-th frame is... It is calculated using its frame number k and frame rate r, and the calculation formula is as follows: Here, k is an integer index starting from 1. The unit is seconds, and its value ranges from 0 to (N-1) / r, where N is the total number of video frames. Through this frame-sequence-based calculation method, the timestamp strictly corresponds to the physical sampling time of the video and is independent of the time displayed by the player, thus providing a reliable basis for subsequent stable and repeatable segmentation according to fixed time windows. Ultimately, a frame sequence F and its corresponding timestamp sequence are obtained. .

[0032] S1.2: Based on a preset fixed time window, the entire video is divided into a series of continuous short segments, resulting in a set of video segments with clearly defined time intervals.

[0033] After obtaining the discrete frame sequence F and the timestamp sequence T, the entire video needs to be divided into a series of consecutive short segments along the time axis. This disclosure uses windows of fixed time length for segmentation to adapt to the semantic rhythms common in teaching videos, such as teacher explanations, student demonstrations, and classroom interactions, which typically last for several seconds.

[0034] Set a fixed time window length As a segmentation unit. Depending on the application scenario... Different values ​​can be selected: to capture macroscopic semantic beats such as explanations, demonstrations, and interactions, a value of 1.0 to 5.0 seconds is typically used; if it is needed to capture fine movements in experimental operations, a value of 0.5 to 2.0 seconds can be used. This value range is set based on two main considerations: first, a coherent semantic action or expression in a teaching activity usually lasts for several seconds; and second, it is to match the ability of subsequent deep neural networks to process temporal information. Too short a video clip will result in overly fragmented video segments, loss of contextual information, and increased computational burden; Excessive length may result in different semantically dominant phases being mixed within a single segment, violating the assumption that the dominant object within a segment remains consistent in subsequent steps, thus affecting segmentation accuracy. The total timeline of the video is then started from time 0, with... The time intervals are divided into M consecutive time intervals, where the time interval corresponding to the j-th segment is... j ranges from 1 to M.

[0035] The total number of segments M is determined by the total video duration. With time window The decision is made, and the calculation formula is as follows: ,in This indicates a rounding up operation to ensure the entire video duration is covered. The duration is typically between 60 seconds and 14,400 seconds (4 hours).

[0036] Based on the timestamp sequence T, for each time interval Locate the corresponding frame in frame sequence F. Specifically, find all frames that satisfy... The conditional frame index k, these indices form a continuous interval, denoted as Then, the index is extracted from the frame sequence F. arrive Combine all consecutive frames to form the j-th video segment. By using this segmentation method based on a fixed time window, we obtained a set of video clips. Each segment is bound to a unique time interval, which ensures that all subsequent analyses are performed at the segment level, thus effectively preventing feature information from being incorrectly mixed together across different subject stages when the semantic dominant object changes.

[0037] S1.3: Perform spatial scale normalization and pixel value standardization on all image frames within each video segment to eliminate external visual differences.

[0038] The video clip set S obtained directly from step S1.2 may contain images with different original resolutions, and their brightness and contrast may also differ due to variations in the lighting conditions of the classroom where the footage was taken and the model of the camera used. To ensure the comparability of the underlying features for subsequent calculations across videos, each frame within each clip must be normalized.

[0039] The processing consists of two main parts: spatial scale normalization and pixel value normalization. First, spatial scale normalization is performed, transforming each segment... Each frame of the image The dimensions are uniformly adjusted to the preset target width W and height H. W and H are constants, and their recommended range is usually [value missing] to achieve a balance between computational complexity and image detail preservation. , A common approach is to adjust it to a square shape, such as... or .

[0040] Next, pixel value normalization is performed. For each pixel, its original pixel value... (In 8-bit representation, the range is 0 to 255) This will undergo a linear transformation, mapping it to a new numerical range. The normalization formula is: .in, It is a preset pixel mean constant. These are preset pixel standard deviation constants. and This is typically obtained by statistically analyzing all image pixels on a large training dataset and using it as a fixed value during the inference phase. For single-channel grayscale images or brightness channels, The suggested value range is [80, 180]. The recommended value range is [20, 80]. For three-channel color images, it is necessary to apply the respective values ​​to the red, green, and blue channels separately. and The values ​​are standardized. In the formula... It is a very small positive number, for example arrive This is used to prevent the denominator from being zero and to ensure numerical stability.

[0041] After the above processing, we obtain the normalized set of fragment images. It eliminates non-semantic visual differences introduced by different external conditions, enabling the basic visual and motion features extracted in subsequent steps to truly reflect information related to the teaching content, thereby achieving effective comparison and analysis across devices and classrooms.

[0042] S1.4: Calculate the average brightness and motion intensity of each standardized video segment, combine them into a segment-level low-dimensional joint feature vector, and thus construct a standardized segment feature sequence.

[0043] Obtaining standardized video clips Building upon this foundation, this sub-step calculates two readily available and physically meaningful low-dimensional features for each segment: average brightness and motion intensity, and combines them to form a joint feature representation for each segment. These features provide direct numerical basis for rapidly evaluating temporal changes and making preliminary judgments about scene states in subsequent steps.

[0044] Calculate the average brightness index for each segment j. This index reflects the overall brightness and darkness of all frames within a segment, and can characterize macroscopic visual states such as the switching on and off of a projector on a lectern, the appearance or erasure of writing on a blackboard, and changes in local lighting on an experimental table. The calculation formula is:

[0045] in, This represents the coordinates of the k-th frame image after normalization. The pixel grayscale value at point j can be taken as the luminance channel value for a color image. The numerator is the sum of the grayscale values ​​of all pixels in all frames within segment j, and the denominator is the product of the total number of frames in the segment and the number of pixels in a single frame, i.e., the total number of pixels, thus obtaining the average luminance.

[0046] Secondly, calculate the motion intensity index for each segment j. This metric measures the intensity of motion changes in a scene by calculating the average pixel difference between adjacent frames within a segment. It can characterize scenes with significant motion, such as a teacher walking, a student standing up and going on stage, or performing an experiment. If segment j contains more than 1 frame (i.e., ... The calculation formula is:

[0047] If segment j contains only one frame (i.e.) Then define its action intensity. .

[0048] in, The absolute value is used to measure the magnitude of change at each pixel position. The numerator is the sum of the absolute differences between all pixels in each pair of adjacent frames within the segment, and the denominator is the product of the logarithm of the inter-frame differences used in the calculation and the number of pixels in a single frame. The technical advantage of calculating B(j) and A(j) is that it generates a low-dimensional, easily computed, and physically meaningful joint feature vector for each segment, providing a direct and reliable numerical basis for subsequent rapid assessment of temporal changes and preliminary judgment of scene status, effectively eliminating the interference of non-semantic visual differences.

[0049] For each fragment j and Combined into a feature vector ,Right now The feature vectors of all segments constitute a set. This set X constitutes the standardized fragment feature sequence. This joint feature set X provides the lowest-level, rapidly computable temporal change signal for the subsequent semantic dominant object determination module, enabling the system to have a reliable one-dimensional temporal change basis even in teaching scenarios where the subject frequently switches.

[0050] In the technical solution of this disclosure, the original teaching video is processed by temporal fragmentation, scale normalization, and pixel standardization, and the joint features of average brightness and motion intensity at the fragment level are extracted to construct a standardized fragment feature sequence. This step eliminates non-semantic interference introduced by different shooting equipment and lighting conditions, transforming the continuous video stream into discrete data units suitable for deep temporal analysis, and providing a stable, reliable, and computable data foundation for subsequent accurate perception of the dynamic switching of semantically dominant objects.

[0051] S2: Using a pre-trained deep neural network, determine the semantic dominant object of each video segment based on the segment feature sequence, and generate a dominant object tag sequence.

[0052] Using a deep learning model, based on the constructed standardized segment feature sequence, the dominant semantic subject of each video segment generated in step 1 is automatically determined, and the segment is labeled with the corresponding dominant object, laying the foundation for subsequent semantic segmentation based on the dominant object. Figure 2 A flowchart illustrating the semantically dominant object determination and tag sequence generation process provided in an embodiment of this disclosure is shown. Figure 2 As shown, this is achieved through the following sub-steps.

[0053] S2.1: Construct the image frame sequence of each video segment into a temporal tensor with a unified time dimension, and use it as input to the deep model.

[0054] In step 1, we obtained the normalized set of video clip images. And the basic feature set X for each segment. To utilize deep learning models for semantic analysis, the image data needs to be converted into a standardized tensor format that the model can directly accept. This sub-step involves constructing a unified temporal tensor for each video segment.

[0055] Specifically, for the j-th video segment, its corresponding normalized multi-frame image sequence is: We arrange all the frames in this sequence in chronological order to form a three-dimensional temporal tensor. In this tensor, the first dimension represents the time axis, indexing different frames; the second and third dimensions represent the spatial height and width of the image, corresponding to the row and column coordinates of pixels, respectively.

[0056] To meet the input size consistency requirement of deep learning models, the tensors of all segments need to be unified to a fixed length in the time dimension. A maximum frame count limit is set. The recommended value range is [8, 32] frames, to strike a balance between preserving sufficient temporal information and controlling computational complexity. For a segment j, if it actually contains the number of frames... Greater than Then only the earlier ones in chronological order are retained. Frame; if Less than Then, by repeatedly copying the last frame of the segment until the time dimension length reaches [a certain value], [the process continues]. This padding method can preserve visual information at the end of the segment to some extent. Ultimately, we obtain a set of segment-level temporal tensors. This step transforms the original video frame sequence into well-structured input data suitable for efficient parallel processing by deep neural networks, thus preparing the data for subsequent intelligent judgment.

[0057] S2.2: Input the segment temporal tensor into a deep neural network to predict the probability distribution that the segment belongs to the teacher, student, or blackboard / projection.

[0058] After obtaining the temporal tensor set Z, it is input into a deep neural network specifically designed for semantically dominant object determination. middle. This is a classification model based on temporal visual features. Its network structure includes convolutional modules for extracting spatiotemporal features, pooling layers for aggregating temporal information, and fully connected classification layers that output probabilities for three main subjects. Specifically, the convolutional modules may contain, for example, 2 to 4 three-dimensional convolutional layers, interspersed with batch normalization layers and non-linear activation functions, to progressively extract local spatiotemporal features from fragmented temporal tensors. The pooling layers are used to reduce the dimensionality and aggregate information in the temporal and spatial dimensions of the feature maps output by the convolutional modules. For example, temporal global average pooling or one-dimensional convolution combined with pooling can be used to convert variable-length spatiotemporal features into fixed-length feature vectors. Optionally, a channel attention module or spatiotemporal attention mechanism can be introduced before the fully connected layers to enhance the model's attention to key visual regions and moments. Finally, the fully connected classification layer receives the aforementioned fixed-length feature vectors and outputs a three-dimensional vector, corresponding to the original predicted scores for the three dominant objects: teacher, student, and blackboard / projection.

[0059] This model learns the mapping from image sequences to semantically dominant categories through end-to-end training. The model is trained on a large number of manually annotated teaching video clips. Each training sample consists of a standardized segment temporal tensor (input) generated in step S1 and its corresponding manually annotated semantically dominant object label (output). The training data covers a rich variety of visual modes in classroom teaching, including but not limited to teacher-led lectures, student-led operations, close-ups of blackboard / projection, and complex scenarios such as teacher-student interactions and rapid camera cuts. By accessing a massive amount of samples, the model can automatically learn to assess the semantic dominance of a segment based on comprehensive visual context features such as relative size, posture, focus position, and voice information when teachers and students are in the same frame. Similarly, for student reaction shots interspersed during teacher lectures, the model can also determine that the segment is student-dominated based on its independent visual content (such as student close-ups). During training, the cross-entropy loss function is optimized to enable the model to distinguish between three dominant modes: teacher, student, and blackboard / projection. After sufficient training, It has the ability to identify three types of classroom semantic dominant patterns from the input image temporal tensor.

[0060] For each segment of the temporal tensor ,Model The model processes and outputs a raw score vector containing three values, corresponding to the teacher, student, and blackboard / projector subjects, respectively. This output design itself constitutes an implicit weighting mechanism. Instead of making hard, either-or decisions, the model assigns a matching score to each of the three possible dominant subjects. To transform these scores into easily understandable and comparable probabilistic forms, the last layer of the model uses the Softmax function for probability normalization. The specific calculation formula is as follows: Where q represents the subject category identifier, and its value is one of T, S, or B, namely teacher, student, or blackboard / projector; is the model's raw output score for fragment j with respect to subject q; e is the base of the natural logarithm. The physical meaning of this Softmax function is to transform the raw score output by the deep neural network into a probability distribution. Its technical effect is to quantify the probability of dominance by the three subjects (teacher, student, blackboard / projector) and sum them to 1, facilitating subsequent maximum probability decision-making and confidence calculation, thus achieving data-driven, interpretable semantic dominance determination. Through this calculation, we obtain three probability values. , and These represent the likelihood that the segment is dominated by the teacher, student, or blackboard / projector, and satisfy the following conditions: When teachers and students are in the same frame or the scene is blurred, the probability values ​​of the two categories may be similar (e.g., , This precisely reflects the uncertainty of the model in determining dominance, which will be further addressed through subsequent confidence calculations and time consistency correction steps.

[0061] The predicted probabilities of all segments are organized into a set. This step leverages the powerful feature learning capabilities of deep learning models to replace traditional methods based on manual rules or thresholds, enabling the recognition of semantically dominant objects to stem from an automated, data-driven understanding of video temporal visual patterns.

[0062] S2.3: Generate an initial dominant object label for each segment based on the maximum predicted probability and calculate its decision confidence.

[0063] After obtaining the three subject probabilities for each segment, they need to be transformed into explicit, discrete dominant object labels to facilitate subsequent time series analysis and segmentation operations. This sub-step performs a simple maximization decision, generating an initial decision result for each segment and evaluating the reliability of that decision.

[0064] For the j-th segment, compare its three probability values. , and The subject category corresponding to the highest probability value is selected as the initial semantic dominant object label for that segment. The physical meaning of this formula is to perform maximum a posteriori probability estimation to determine the most likely dominant element in the current segment. Its technical effect is to generate a discrete initial semantic label for each segment, providing the basic elements for constructing the dominant element sequence. This decision-making process is expressed by the formula: ,in This indicates that the argument that maximizes the value of the expression is the subject category with the highest probability.

[0065] However, relying solely on the maximum probability value for a decision may mask uncertainties in certain segments of the model. For example, when the three probability values ​​are very close, the lead of the maximum probability is small, and the decision may be unreliable. To quantify this reliability, we simultaneously calculate a dominant confidence level. The confidence level is defined as the maximum probability. With the second highest probability The difference between them, i.e. . The value of is between 0 and 1 (excluding 1). The physical meaning of this formula is to measure the discriminative and deterministic nature of the model's decision results. Its technical effect is to generate a confidence score sequence to identify which segments' decision results may have uncertainty (low confidence), thus providing a crucial basis for subsequent time consistency correction processing. The larger the value, the more confident the model is in believing that the segment belongs to a dominant category, and the more obvious the discrimination; conversely, the smaller the value, the lower the confidence. The smaller the value, the greater the uncertainty in the model's judgment of the segment, which may be in the transition phase of subject switching or the content of the image is blurry.

[0066] Through this step, we obtain the initial semantically dominant object tag sequence. and dominant confidence sequence This provides a direct basis for further refined processing of low-confidence segments.

[0067] S2.4: Based on the decision confidence, introduce a time neighborhood window to perform probabilistic smoothing on low-confidence segments, and output the final smoothed dominant object label sequence.

[0068] In real classroom teaching videos, due to rapid camera cuts, brief obstructions, or the simultaneous appearance of multiple subjects in the frame, deep learning models may exhibit occasional jitter or errors in their independent judgments of individual segments. This means that the dominant object labels of adjacent segments may jump unreasonably and frequently on the timeline. To improve the temporal smoothness and reasonableness of the final label sequence, this sub-step introduces temporal consistency constraints to correct the initial judgment results.

[0069] The correction process is based on the dominant confidence sequence C. First, a confidence threshold is set. The recommended value range is [0.05, 0.30]. This threshold is used to define the reliability of the model's judgment results. Its value is based on statistical analysis of prediction results from a large number of samples: when the difference between the highest probability and the second highest probability is lower than this threshold, it indicates that the model has high uncertainty regarding the dominant object attribution of the current segment, which may be in a transitional zone of subject switching or where the image content is blurry. Setting a lower threshold... (e.g., 0.05) will make the system more sensitive to uncertainty and retain more segments to be corrected; setting a higher value will make the system more sensitive to uncertainty and retain more segments to be corrected. A value of 0.30 makes the system more stringent, smoothing only segments with low confidence. Generally, a suitable value is chosen while ensuring sensitive capture of subject transitions. Values ​​(e.g., 0.10-0.20) help strike a balance between smoothing noise and preserving detail, thus achieving robust temporal consistency correction. For the j-th segment, if its confidence level... If so, then the model's initial judgment on this segment is considered. It is highly reliable, therefore it was directly adopted as the final label, i.e., let .

[0070] For confidence level If a segment is deemed indeterminate, its initial assessment is considered potentially unstable. To leverage the temporal continuity of the video to aid decision-making, the system introduces a temporal neighborhood window to re-evaluate the segment. This window is centered on the current segment index j and extends R segments to both sides, forming a neighborhood of total length 2R+1, where R is a constant for the window radius, with a suggested value range of [1,5].

[0071] Within this neighborhood window, instead of simply counting labels, we perform a time-weighted average of the probabilities of various subjects to smooth out accidental fluctuations. Specifically, for subject q, its corrected smoothed probability... The calculation formula is

[0072] in, It is the original probability of the i-th segment in the neighborhood with respect to the subject q; It is a time weighting function, usually designed to be inversely proportional to the distance, for example, taking... This means that the closer a segment is to the center segment j, the greater its probability contributes to the smoothing result. When the index... or At that time, the corresponding and It is not involved in the summation calculation. The physical meaning of this formula is to use the probability information within the current segment's time neighborhood to perform a weighted average to smooth out random fluctuations. Its technical effect is to introduce prior knowledge of temporal continuity, correct the judgment results of low-confidence segments, effectively suppress non-physical jumps in the label sequence caused by instantaneous interference (such as occlusion, jitter), and thus output a more temporally smoother and more robust final dominant object label sequence.

[0073] The smoothing probabilities after correction for the three types of subjects were calculated. , , Then, the maximization decision is performed again, and the subject category with the highest smoothing probability is taken as the final semantic dominant object label for that segment. .

[0074] After the above correction process, the final output is a smoothed semantically dominant object tag sequence that better conforms to the common sense of temporal consistency. This step adds an engineering-controllable temporal logic check layer on top of the core deep learning judgment, effectively suppressing non-physical jitter in the label sequence caused by instantaneous interference, and ensuring the overall robustness of the dominant object judgment.

[0075] In the technical solution of this disclosure, by constructing a segment temporal tensor and utilizing a pre-trained deep neural network for end-to-end subject probability prediction, automated, data-driven identification of the semantic dominant object for each video segment is achieved. Furthermore, based on probability maximization and temporal consistency correction, a smooth and robust dominant object label sequence is output. This step explicitly captures the alternation of dominance among teachers, students, blackboard / projector, etc., on the timeline during the teaching process, overcoming the implicit assumption of subject continuity in existing technologies, and providing a direct and accurate basis for subsequent semantic segmentation based on the dominant object.

[0076] S3: Based on the dominant object label sequence, perform multi-level semantic segmentation on the video timeline. The semantic segmentation includes primary segmentation of the timeline based on the dominant object label and its probability distribution to obtain continuous semantic segments with consistent dominant objects; and fine-grained segmentation within the continuous semantic segments of the same dominant object based on the changes in visual and action features to obtain a series of semantic units.

[0077] Intelligent segmentation is performed on video segment sequences marked with a dominant object, generating multiple semantically clean and coherent semantic units. Consistency of the semantic dominant object is used as a hard constraint for segmentation, ensuring that teaching stages dominated by different subjects are not incorrectly grouped together. Furthermore, within consecutive segments with consistent dominant objects, fine-grained segmentation is performed based on changes in visual and motor features to distinguish different teaching activity stages under the same subject. Figure 3A flowchart illustrating a multi-level semantic segmentation process provided in an embodiment of this disclosure is shown. For example... Figure 3 As shown, this is achieved through the following sub-steps.

[0078] S3.1: Calculate the intensity of change in the probability distribution of the dominant object between adjacent segments to obtain a temporal change sequence that reflects the magnitude of semantic transfer.

[0079] Step 2 outputs the final dominant object label sequence O and the subject probability sequence P for each segment. To more precisely detect the boundaries of semantic changes, it not only focuses on whether the dominant object label changes, but also measures the overall magnitude of the difference in subject probability distribution between adjacent segments. This magnitude of difference reflects the strength of the deep learning model's perception of semantic center shifts, helping to distinguish between explicit topic shifts and uncertain, subtle fluctuations.

[0080] Therefore, a sequence D called the dominant object change intensity is calculated. For the j-th segment ( ), its intensity of change Defined as the sum of the absolute differences between the current segment and the previous segment in terms of the probabilities of the three subject categories. The calculation formula is:

[0081] in, , , These represent the probabilities that the j-th segment belongs to the teacher, student, or blackboard / projection subject, respectively, and are given by the deep learning model in step 2. The value range is [0,2]. The physical meaning of this formula is to calculate the Manhattan distance between adjacent segments on the three subject probability distributions, thus quantifying the magnitude of semantic centroid shift. When the subject probability distributions of adjacent segments are exactly the same, When the distributions are completely opposite (e.g., from a pure teacher probability of 1 to a pure student probability of 1), . The larger the value, the more drastic the change in the dominant semantics perceived by the model.

[0082] By calculating the intensity of change between all adjacent segments, the sequence of intensity of change of the dominant object is obtained. This sequence will provide a continuous and quantitative basis for subsequent segmentation decisions, so that the segmentation point may not only appear where the dominant object label changes, but also where the label remains unchanged but the probability distribution has shifted significantly, thus capturing potential semantic transitions earlier.

[0083] S3.2: Based on whether the dominant object marker changes, and combined with the change intensity threshold, a primary segmentation is performed to obtain a continuous semantic segment with a consistent dominant object.

[0084] This sub-step uses the dominant object label sequence O and the change intensity sequence D to perform the first segmentation of the video timeline, generating a series of continuous primary semantic segments. The segmentation strategy is based on the principle of dominant object consistency, supplemented by a change intensity threshold as an auxiliary triggering condition, to ensure timely segmentation even if the label has not yet been flipped when the subject probability changes significantly.

[0085] In practice, a threshold for the intensity of change of the dominant object is set. The recommended value range is [0.40, 1.20]. This threshold is used to determine whether the difference in the probability distribution of the dominant object between adjacent segments constitutes a meaningful semantic transfer, thereby triggering primary segmentation. Its value is based on the following: during a stable phase of the dominant object, the probability distribution of adjacent segments differs little due to the continuity of the image content, and the intensity of change is relatively small. Typically below 0.40; however, when a clear subject switch or scene transition occurs, It will exhibit a relatively large peak value. Therefore, Setting the threshold above 0.40 effectively filters out minor fluctuations within stable segments, only segmenting when significant semantic changes are detected. A higher threshold results in more conservative segmentation, generating fewer but more stable semantic segments; a lower threshold leads to more sensitive segmentation, potentially producing finer-grained segments but requiring subsequent suppression. Starting from the second segment, each position j ( If any of the following conditions are met, then a semantic boundary exists between fragment j and j-1, and segmentation should be performed at this point: The dominant object tag has changed, that is .

[0086] The dominant object tags are the same, that is However, the intensity of change .

[0087] Condition 1 is a hard constraint, ensuring that stages with different dominant subjects are necessarily separated. Condition 2 is a soft constraint, designed to prevent situations where subject probabilities have clearly shifted but haven't yet led to a switch in the maximum category from being incorrectly aggregated into the same semantic segment. For example, in a long teacher-led explanation stage, if a student suddenly makes a large movement causing a sharp increase in the student's subject probability (but not exceeding the teacher's probability), the intensity of this change... It is likely that the problem can be solved by cutting at this point through condition 2, thereby separating the explanation stage from the student interference stage and improving the purity of the semantic unit.

[0088] After the above scanning and segmentation, we divide the video segment sequence into K consecutive primary semantic segments, each semantic segment... Use a fragment index range It means that, among them and These are the start and end segment indices of the segment, respectively. These primary semantic segments constitute a set. Within each semantic segment, the dominant object label is consistent, and the intensity of change between adjacent segments within the segment is below a threshold. (Except at segment boundaries). The length (number of segments contained) of the k-th semantic segment is: .

[0089] S3.3: Based on the semantic segment length and its dominant probability strength, suppress overly short semantic segments and merge them into adjacent stable semantic segments.

[0090] Because classroom environments may contain distractions such as momentary occlusion, camera shake, or rapid transitions, initial segmentation may produce some extremely short semantic segments. These short segments typically do not represent meaningful independent teaching phases but rather noise or transitional states. Retaining them would lead to semantic unit fragmentation, affecting the practicality of label generation and subsequent analysis. Therefore, this sub-step aims to filter out these meaningless short segments and merge them into adjacent, more stable longer segments.

[0091] Set a minimum segment length threshold The suggested range for this value is [2, 6] segments. For each segment in the primary semantic segment set R... If its length If so, mark it as a short segment to be processed.

[0092] When processing short segments, it's not simply a matter of merging them with adjacent segments. Instead, the dominant probability strength of the short segment and its adjacent segments needs to be considered to ensure the rationality of the merging decision. First, calculate the short segment... Dominant probability strength It reflects the dominant object within that section. The average prediction confidence level. The calculation formula is:

[0093] in, It is fragment j to its dominant object The predicted probability. The physical meaning of this formula is to calculate the semantic segment. Inside, to its dominant object The average predicted probability reflects the overall confidence level of the dominant decision in that segment. Its technical effect is to provide an objective metric for the short-segment suppression and merging steps. This enables the system to distinguish between meaningless noisy short segments and meaningful independent short segments based on the stability of the probability within the segment (rather than just the length), thereby making more reasonable merging decisions and maintaining the integrity of the true semantic boundaries while eliminating fragmentation.

[0094] Then, examine the short passage. The left and right adjacent segments. The merging strategy is as follows: If the dominant object of adjacent segments is the same, then the shorter segment will be... Merging into dominant probability strength In the higher adjacent segment; if the strengths on both sides are equal, then merge into the previous (left) adjacent segment.

[0095] If the dominant objects of the left and right adjacent segments are different, then the shorter segment is calculated. Given the average probability of the dominant object in the left and right adjacent segments (i.e., the average probability of all segments within the short segment), the short segment is merged into the adjacent segment with the higher average probability.

[0096] If the dominant objects of the left and right adjacent segments are different, then the shorter segment It may be in the transition zone between two stable segments. At this point, compare the proximity of the probability of the dominant object in the shorter segment with that in the adjacent segments on the left and right, and merge it into the side with the higher dominant probability.

[0097] An important principle is that merging stable segments across different dominant objects is not allowed. That is, if the left and right adjacent segments are stable long segments of different dominant objects, then the short segment in the middle must be merged into one of them, and the segments of different dominant objects cannot be merged at the same time.

[0098] After the above suppression and absorption processes, a purified set of semantic segments was obtained. ,in In the new set of semantic segments, the length of each segment is no less than [amount missing]. Alternatively, a short but highly dominant short phase can be identified as a meaningful independent short phase. This step leverages the probabilistic stability of the deep learning model output to effectively eliminate oversegmentation caused by noise while maintaining the integrity of the real subject switching boundaries.

[0099] S3.4: Within the semantic segments of the same dominant object, fine-grained divisions are made based on changes in average brightness and action intensity to distinguish different stages of teaching activities.

[0100] After processing in step S3.3, each semantic segment They all have the same dominant object tag internally. However, within a relatively long period dominated by the same subject, there may be multiple different stages of teaching behavior. For example, during teacher-led periods, different activities may occur sequentially, such as explaining theories, presenting PPTs, and writing derivations on the blackboard; during student-led periods, different steps may be included, such as answering questions on stage, group discussions, and experimental operations. To more precisely depict the teaching process, this document further subdivides each semantic segment into stages, ensuring that it does not cross different dominant subjects.

[0101] The features relied upon for segmentation are the set of basic features X extracted in step 1, specifically the average brightness of each segment. and intensity of movement The rationale for selecting these two fundamental features for internal segmentation lies in the fact that, within the specific context of classroom teaching, when the dominant semantic object remains unchanged, the transitions between different teaching stages are often accompanied by visual pattern changes that can be captured by the camera and possess a certain degree of distinctiveness. While these features are simple, they effectively reflect the visual changes brought about by different teaching behaviors within the same subject. For example, a teacher switching from verbal explanation to writing on the blackboard typically leads to a significant change in the average brightness of the image, such as the difference in brightness between the blackboard area and the teacher's face area; a switch from static explanation while standing behind the podium to dynamic guidance while moving among the students inevitably causes an increase in the overall intensity of movement in the image. Similarly, in student-led experiments, different operational steps may correspond to different tool movements, changes in movement intensity, or changes in the state of the experimental apparatus, which may cause localized brightness changes. Therefore, changes in brightness and movement intensity, within the context of classroom teaching videos, are strong correlation signals indicating transitions in teaching activities within the same subject.

[0102] For semantic segments For any two adjacent segments j-1 and j within the same segment (where j starts from the starting segment index plus 1 and ends at the ending segment index), calculate the characteristic change between them. The calculation formula is:

[0103] in, and These are weighting constants, representing the importance of brightness change and motion change in the total change, respectively, satisfying... Recommended range , ε is a very small positive number, for example... arrive This is used to prevent the denominator from being zero and to ensure numerical stability. The physical meaning of this formula is to calculate the weighted sum of the normalized relative changes in brightness and the relative changes in motion intensity, so as to comprehensively measure the degree of visual feature changes between adjacent segments within the same dominant object.

[0104] A larger value indicates a more significant change in the visual appearance or motion intensity of adjacent segments. Considering the aforementioned characteristics of the teaching scenario, this change is highly likely to correspond to a transition in the teaching stage. A feature change threshold should be set. The recommended value range is [0.10, 0.60]. For each position j within a semantic segment, if... If the segment is at that position (between segments j-1 and j), then it is segmented, breaking down a long semantic segment into smaller semantic units. This provides a fine-grained basis for segmenting long time periods within the same subject, while ensuring that different dominant objects are not crossed. This is achieved through thresholding. judge The system can identify internal transition points that belong to the same subject but involve different teaching behaviors, such as when a teacher switches from lecturing to writing on the blackboard or when a student switches from answering questions to discussing. This allows the system to generate more refined and semantically coherent semantic units.

[0105] After scanning and subdividing all semantic segments, L final semantic units are obtained, forming a set. Each semantic unit It must contain at least one complete video segment and inherit the dominant object tag of its semantic segment. And the corresponding time intervals. These semantic units not only ensure the strict separation of different dominant object stages, but also achieve the characterization of different teaching stages under the same subject through internal subdivision, providing an ideal structured input for subsequent accurate tag generation.

[0106] In the technical solution of this disclosure, multi-level semantic segmentation is performed based on the dominant object label sequence and combined with the intensity of dominant object change and visual action feature changes. First, primary segmentation is performed with the consistency of the dominant object as the fundamental constraint to ensure strict separation of stages for different dominant subjects. Then, short-segment suppression and absorption processes are used to eliminate fragmentation caused by noise. Finally, fine-grained stage division is performed within the same subject based on visual action features. This process generates multiple semantically pure, clearly defined, and internally coherent semantic units, realistically reflecting the temporal structure of multiple subject alternations and different teaching stages within each subject during the teaching process.

[0107] S4: For each semantic unit, predict its teaching activity label, and bind and output the label with the corresponding semantic dominant object and time interval.

[0108] For each independent semantic unit identified in step 3, generate a teaching activity label that accurately describes its content. Since the semantic units already guarantee the consistency of their internal semantic dominant objects, this step can be safely processed individually for each unit without worrying about confusion between different semantic subjects. This is achieved through the following sub-steps.

[0109] S4.1: Organize all image frames contained in each semantic unit into a fixed-length temporal tensor, which serves as the input to the label generation model.

[0110] The final set of semantic units output in step 3 It provides the semantic structure of the video. Each semantic unit It includes the corresponding video segment index range. Dominant object tag And precise start and end times To generate labels for each unit, its visual content needs to be converted into a uniform format that the label generation deep learning model can process.

[0111] According to semantic units fragment index range The normalized fragment frame sequence obtained from step 1 Extract all video clips belonging to that unit (i.e., from the first segment) The segment to the first (Segments). Then, all the image frames contained in these segments are spliced ​​together strictly in their original chronological order to form a long, continuous frame sequence that fully represents the visual content of the semantic unit.

[0112] However, different semantic units may contain different numbers of frames. To meet the requirement of fixed input size for deep learning models, the frame sequence of each unit needs to be length normalized. Therefore, a maximum upper limit on the number of frames per semantic unit is set. The recommended range is [16, 64] frames. For a semantic unit... If its actual frame count exceeds Then, a time-uniform sampling method is used to extract from the original sequence. Frames are used to preserve key timing information; if the actual number of frames is insufficient... Then, the last frame of the sequence is repeatedly copied until the number of frames reaches [a certain threshold]. This approach maintains the temporal integrity of the cell content as much as possible while standardizing the input size.

[0113] The processed frame sequence is constructed into a four-dimensional tensor. The four dimensions of this tensor have fixed meanings: the first dimension is the temporal index, corresponding to the sampled frame sequence; the second and third dimensions are the spatial height and width, respectively; and the fourth dimension is the color channels (e.g., an RGB image has 3 channels). Through this step, each structured semantic unit is transformed into a uniform-sized tensor that can be directly input into a deep learning model. The tensors of all semantic units constitute a set. This ensures that subsequent tag generation is performed at the level of each pure semantic unit, rather than reverting to the original segment level or coarse full video level, laying the foundation for generating high-precision tags.

[0114] S4.2: Input the semantic unit tensor into the label to generate a deep neural network, predict the probability that it belongs to various teaching activity labels, and apply the dominant object semantic consistency constraint.

[0115] Obtain the set of input tensors at the semantic unit level. Then, it is input into a pre-trained label-generating deep neural network. middle. This is a deep temporal model suitable for multi-label classification, typically consisting of a feature encoder and a label classifier. It can process visual sequences of variable length and output multi-dimensional probabilities corresponding to a predefined set of instructional activity labels. The feature encoder extracts high-level semantic features from the temporal visual tensor of semantic units. Its structure can be, for example, a backbone network based on a 3D CNN, a two-stream network combined with optical flow input, or a temporal Transformer encoder. This encoder outputs a feature representation that incorporates temporal contextual information. The label classifier is typically a multilayer perceptron with one or more fully connected layers. Its input is the feature vector output by the feature encoder, and its output dimension is the same as the number K of predefined instructional activity labels. Each output neuron corresponds to the raw prediction score for one label. Optionally, a temporal attention layer or self-attention mechanism can be added between the feature encoder and the label classifier, allowing the model to dynamically focus on the most critical time segment within the semantic unit for label determination. For multi-label classification tasks, the final activation function of the label classifier is usually the Sigmoid function, or it can be normalized using the Softmax function before thresholding.

[0116] The model is trained using supervised learning. Its training dataset consists of a large number of precisely segmented semantic units from instructional videos. Each semantic unit sample contains its visual content tensor (input) and one or more manually labeled instructional activity tags (output). Through this training... The model learned to map the visual temporal patterns of semantic units to specific teaching activity labels. The network parameters of the model were obtained by optimizing the multi-label classification loss on this type of data.

[0117] Model The output layer is a vector of length K, where K represents the total number of predefined teaching video tag sets, with a recommended range of [50, 500], covering a wide range of teaching activity types such as teacher theoretical explanations, student experimental operations, teacher-student Q&A, blackboard derivation, and courseware presentations. This tag set is a predefined open-ended vocabulary used to describe classroom teaching activities. The tags can be logically parallel or organized into a lightweight hierarchical relationship based on pedagogical knowledge; for example, teacher activities could include theoretical explanations, blackboard derivations, and experimental demonstrations. The content and size of the set (K value) can be customized and expanded according to the knowledge of specific application domains without changing the core process of this method, thus ensuring the adaptability and scalability of the method.

[0118] For input tensor The model will output an original score vector, denoted as... ,in The original score represents the degree to which the model considers semantic unit l to match the k-th label.

[0119] To obtain comparable probability distributions, we normalize the original scores using the Softmax function, calculated as follows:

[0120] in, This represents the predicted probability that the l-th semantic unit belongs to the k-th label, with a value ranging from 0 to 1. For the same unit l, the sum of the probabilities of all labels is 1. The physical meaning of this Softmax function is to optimize the label generation model. The output raw matching scores are normalized to a probability distribution. The technical effect is that the degree of matching between semantic units and all predefined teaching activity labels can be quantified and compared, providing a numerical basis for subsequent selection of effective labels based on confidence thresholds.

[0121] To improve the semantic relevance of generated tags and avoid errors, a key constraint is introduced in this step: the dominant object semantic consistency constraint. Each semantic unit... Each has its own dominant object marker. (Teacher T, Student S, or Blackboard / Projector B). The system will, based on the preset correspondence between labels and the dominant object, exclude those that do not conform to the criteria. The predicted probability corresponding to the label is set to zero, so as to retain only those labels. The probability of a logically consistent subset of labels. For example, for a unit where the dominant object is the student (S), the probability of labels clearly dominated by the teacher, such as teacher writing on the blackboard, will be set to zero. This constraint is not implemented by modifying the model, but is processed after the model output, ensuring that the generated labels are semantically strictly consistent with the dominant role of the unit, and preventing contradictory situations such as student actions being labeled as teacher demonstrations. Ultimately, a set of label probability distributions at the semantic unit level is obtained. This probability distribution directly supports multi-label classification scenarios, i.e., a semantic unit. It may correspond to one or more high-probability teaching activity tags, which objectively reflects that there may be multiple behaviors combined in teaching activities, such as giving lectures and pointing out projections at the same time.

[0122] S4.3: Select valid labels from the probability distribution based on the preset label confidence threshold to ensure that each semantic unit obtains at least one descriptive label.

[0123] After obtaining the tag probability distribution for each semantic unit, it needs to be transformed into one or more discrete, deterministic content tags. Directly selecting the single tag with the highest probability may be too absolute and cannot handle situations where a unit contains multiple related activities, such as a presentation and instructions PowerPoint slides. Therefore, this sub-step uses a probability threshold-based filtering mechanism to determine effective tags.

[0124] Set a label confidence threshold The recommended value range is [0.30, 0.80]. This threshold is an important control parameter, and its value directly affects the performance of the label generation model and the quality of the final output labels. When the value is high, the system becomes more conservative, only outputting labels that the model is very confident about. The labels have high precision, but the recall may decrease, making it suitable for generating catalog labels for formal course resources; when... Lower values ​​result in richer and more diverse tag output, improving recall, but may introduce some uncertain tags. This approach is suitable for resource retrieval and fine-grained instructional analysis scenarios. This suggested range is based on the tag generation model. The predicted probability distribution and the matching relationship between the actual labels on the validation set are statistically analyzed, which can cover different application needs from high precision to high recall.

[0125] For the l-th semantic unit, iterate through its label probability distribution. For each label k, if its probability If a threshold-based filtering mechanism is used, the tag is determined to be valid for unit l and added to the set of valid tags for that unit. This threshold-based filtering mechanism is the core of this method for handling multi-tag output. By independently judging all tags, a semantic unit can simultaneously obtain zero, one, or more valid tags, thus naturally supporting multi-tag scenarios. This judgment process can be represented by an indicator function. express: ,when ,otherwise .

[0126] There is a special case where the probabilities of all labels for a certain semantic unit are below a threshold. This results in an empty set of valid labels. To prevent units from having no labels, the system has a fallback mechanism: in this case, the system will select the label with the highest probability, i.e., This serves as a catch-all label for the unit. This ensures that each semantic unit has at least one label describing its content.

[0127] After the above screening, each semantic unit They all obtained a definite set of tags. This set may contain one or more labels. Optionally, the system can assign labels according to predicted probabilities. The tags within the set are sorted from highest to lowest order to reflect the relative relevance of each tag to the unit content. The set of tags for all units constitutes... This step transforms the continuous probabilities output by the model into a defined set of semantic labels that meet the needs of different applications through an adjustable threshold mechanism.

[0128] S4.4: Bind the definite label of each semantic unit to its dominant object tag and precise start and end time, and merge adjacent similar label entries to form the final structured video label set.

[0129] The generated semantic tags need to be integrated with their contextual information in the video to output a structured result that is ultimately usable by the user. The specific tag for each semantic unit must be closely linked to its identity information, namely the dominant object and the time interval in which it occurs, in order to form a meaningful video summary.

[0130] For each semantic unit Determine the tag set Dominant object tag and the precise time interval inherited from step 3. Package the data to form a complete video tag entry, in the format of (tag, dominant object, start time, end time).

[0131] However, directly outputting L independent entries can lead to label fragmentation. For example, two adjacent semantic units may have the exact same set of labels and dominant objects, and their time intervals may be very short (not exceeding a single time window). This may be due to overly detailed internal subdivisions in step 3. To output a more concise and user-friendly tag structure, this step performs an optional merging operation. The system scans the time-sorted tag entry sequence, and if adjacent entries simultaneously meet the following three conditions, they are merged into a single continuous tag interval: ① The tag sets are completely identical; ②The dominant object tags are the same; ③ The interval between the start time of the next purpose and the end time of the previous purpose shall not exceed [a certain value]. Here A threshold is set for tag merging. Its value can be the same as the fixed time window length for segmentation in step S1.2, or it can be set independently according to the simplicity requirements of the output tags. It is usually set to 1.0 to 5.0 seconds.

[0132] Start time of the new label range after merging (The superscript m indicates the merged interval) Take the minimum start time of all merged units, and the end time. Take the maximum value of the end times of all merged units, i.e. and The label and the dominant object remain unchanged.

[0133] After final binding and optional merging, the system outputs a video-level tag set. This collection comprehensively depicts the true semantic structure of multiple subjects taking turns dominating in instructional videos: each tag clearly indicates the time frame, who is leading, and what kind of teaching activity occurred. This structured output not only supports precise timeline-based positioning and browsing but also enables multi-dimensional statistical analysis based on the dominant subject and activity type, greatly enhancing the usability of classroom video resources.

[0134] In the technical solution of this disclosure embodiment, for each segmented semantic unit, a specialized tag generation deep model is used to predict its teaching activity tags. Before output, a semantic consistency constraint on the dominant object is applied to ensure that the tags logically match the dominant role of the unit. Valid tags are determined through a probability threshold mechanism, and the tags are bound to the dominant object and a precise time interval for output, forming a structured video semantic description. This step ultimately produces a set of tags that precisely correspond to the video timeline and clearly label the dominant object and activity content. This allows the automatically generated tags to unambiguously depict complex teaching interaction processes, greatly improving the retrieval and analyzability of video resources.

[0135] According to embodiments of this disclosure, an electronic device is also provided, which may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor can invoke logical instructions stored in the memory to execute the methods provided in the above embodiments.

[0136] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, this disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] It should be understood that the above embodiments are only used to illustrate the technical solutions of this disclosure, and not to limit them; although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for automatically generating video content tags based on deep learning, characterized in that, The method includes: S1. Perform time-series fragmentation processing on the input teaching video, extract the basic features of each video segment, and construct a standardized segment feature sequence; S2. Using a pre-trained deep neural network, determine the semantic dominant object of each video segment based on the segment feature sequence, and generate a dominant object tag sequence; S3. Based on the dominant object label sequence, perform multi-level semantic segmentation on the video timeline. The semantic segmentation includes primary segmentation of the timeline based on the dominant object label and its probability distribution to obtain continuous semantic segments with consistent dominant objects; and fine-grained segmentation within the continuous semantic segments of the same dominant object based on the changes in visual and action features to obtain a series of semantic units. S4. For each semantic unit, predict its teaching activity label, and bind and output the label with the corresponding semantic dominant object and time interval.

2. The video content automatic tag generation method based on deep learning according to claim 1, characterized in that, S1 includes: The instructional videos are divided into consecutive short segments according to fixed time windows; Standardize the spatial scale and pixel values ​​of each video segment; The average brightness and motion intensity features of each standardized segment are extracted to form the segment feature sequence.

3. The video content automatic tag generation method based on deep learning according to claim 1, characterized in that, S2 includes: Each video segment's image frames are constructed as temporal tensors and input into a deep neural network to predict the probability distribution of their belonging to each preset dominant object category; The initial dominant object label and determination confidence level for each segment are generated based on the probability distribution. The initial dominant object markers are subjected to time consistency correction to generate the final dominant object marker sequence.

4. The video content automatic tag generation method based on deep learning according to claim 3, characterized in that, The time consistency correction includes: For segments whose confidence level is below the threshold, smoothing is performed based on the probability distribution of multiple segments in their time neighborhood.

5. The method for automatically generating video content tags based on deep learning according to claim 1, characterized in that, The semantic segmentation also includes: After the initial segmentation, semantic segments shorter than a preset threshold are suppressed and merged into adjacent semantic segments with the same dominant object.

6. The method for automatically generating video content tags based on deep learning according to claim 1, characterized in that, The primary segmentation includes: When adjacent segments have different dominant object labels, or the labels are the same but the change intensity of the dominant object probability distribution exceeds a preset threshold, the segments are split.

7. The method for automatically generating video content tags based on deep learning according to claim 1, characterized in that, S4 includes: The visual content of each semantic unit is organized into a fixed-length tensor and input into the label generation model to obtain the probability that it belongs to various types of teaching activity labels. Based on a preset confidence threshold, the valid teaching activity label for the semantic unit is determined from the probability. The effective teaching activity tags of each semantic unit are bound to its semantic dominant object and time start and end points to form structured tag entries; Tag entries that are adjacent in time and have the same tag as the dominant object are merged to form the final output video tag set.

8. The method for automatically generating video content tags based on deep learning according to claim 1, characterized in that, The semantically dominant objects include teachers, students, and blackboards or projectors.

9. An electronic device, characterized in that, The electronic device includes a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to implement the deep learning-based automatic video content tagging method according to any one of claims 1-8.

10. A computer storage medium, characterized in that, It stores a computer program, which, when executed, implements the video content automatic tag generation method based on deep learning according to any one of claims 1-8.

Images