A video content labeling method based on big data

Abstract

The application discloses a video content labeling method based on big data. The method comprises the following steps: extracting the multi-modal features of a video, constructing an initial semantic representation containing scene, object, action and attribute nodes, and establishing a hierarchical semantic graph containing top-level scenes, intermediate actions and bottom-level object attributes. Information transmission and aggregation are carried out under the hierarchical, interlayer and time sequence attention mechanism by using a graph neural network, and the labeling result is predicted after updating the node features. The labeling result is evaluated in terms of frame, shot and video quality, and when the result is lower than a threshold, manual review is triggered, and the result is fed back to the model for incremental training. Meanwhile, the application also integrates cross-modal adversarial reasoning, adaptive modal weighting, dynamic threshold adjustment and error mode analysis functions. The application realizes efficient, accurate, controllable and self-optimizing video content labeling.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, and more specifically to a video content annotation method based on big data. Background Technology

[0002] With the explosive growth of video data, video content annotation has become a core foundation for applications such as video retrieval, intelligent recommendation, autonomous driving, and medical image analysis, and its quality directly affects the reliability of subsequent tasks.

[0003] Existing technologies mainly suffer from the following problems: First, purely manual annotation is costly, inefficient, and inconsistent, and long videos easily lead to attention fatigue. Second, rule-based methods struggle to cover diverse scenarios, resulting in low recall and a lack of adaptability. Third, single-modal deep learning methods ignore multimodal information in videos, leading to insufficient semantic integrity and poor robustness. Fourth, simple multimodal fusion methods cannot effectively model complex dependencies between modalities, exhibiting poor adaptability and drastically reduced performance in the presence of noise. Fifth, weakly supervised learning faces the problem of label noise, making it difficult to obtain fine-grained semantics. Sixth, transfer learning-based methods suffer from domain bias and limited generalization ability.

[0004] In terms of quality control, existing technologies lack a reliability assessment mechanism for labeled results, making errors prone to cascading propagation. Furthermore, most methods rely on post-hoc verification and lack real-time monitoring. Regarding graph modeling, existing methods primarily use static external knowledge bases, failing to model the hierarchical semantic structure within videos and exhibiting simplistic node-edge relationships. Finally, quality control lacks a multi-granularity evaluation system at the frame, shot, and video levels. Manual review often employs random sampling, making it difficult to proactively identify high-risk samples and resulting in low efficiency.

[0005] In summary, existing video annotation technologies suffer from problems such as low annotation efficiency, poor semantic hierarchy, insufficient multimodal fusion, weak domain transferability, and insufficient quality controllability, and urgently need improvement. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a video content annotation method based on big data, which overcomes the technical defects of existing video annotation technologies, such as low annotation efficiency, poor semantic hierarchy, insufficient multimodal fusion, weak domain transfer capability, and insufficient quality controllability.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides an adaptive video content annotation method based on hierarchical semantic graphs, comprising the following steps: The process involves: acquiring multimodal feature data of the video to be labeled, including visual, audio, and text features; constructing an initial semantic representation of the video based on the multimodal feature data, including scene nodes, object nodes, action nodes, and attribute nodes; constructing a hierarchical semantic graph, comprising a top-level scene layer, an intermediate action layer, and a bottom-level object attribute layer, with nodes connected by semantic edges to form a hierarchical semantic relationship; using a graph neural network to perform semantic information transfer and feature aggregation on the hierarchical semantic graph to generate updated feature representations for each node, including intra-layer attention, inter-layer attention, and temporal attention mechanisms; performing label prediction based on the updated node feature representations to output the labeling results of the video content; and performing multi-granularity quality evaluation on the labeling results, including frame-level evaluation, shot-level evaluation, and video-level evaluation. When the quality evaluation result is lower than a preset threshold, a manual review process is triggered, and the review result is fed back to the model for incremental training.

[0008] In the video content annotation method of the present invention, the step of constructing the initial semantic representation of the video based on the multimodal feature data specifically includes: performing scene classification detection on the visual features; if the detection confidence is higher than a first preset threshold, then establishing a corresponding node in the scene layer; if the detection confidence is lower than the first preset threshold but higher than a second preset threshold, then marking it as a scene to be confirmed and entering the manual review queue; if the detection confidence is lower than the second preset threshold, then determining it as an unknown scene type; detecting action categories based on motion information in the visual features; if the action detection confidence is higher than a third preset threshold, then confirming the establishment of the action node; if the action detection confidence is between the third preset threshold and the fourth preset threshold, then combining the audio features for auxiliary judgment; establishing the object node based on the target detection result in the visual features, and extracting object attribute features to establish the attribute node.

[0009] In the video content annotation method of the present invention, the rules for establishing semantic edges between nodes of the hierarchical semantic graph include: if the scene node and the action node have a co-occurrence relationship in the spatiotemporal dimension, then a scene-action edge is established; if the action node and the object node have a subject-verb-object relationship, then an action-object edge is established; if the object node and the attribute node have a subordinate relationship, then an object-attribute edge is established.

[0010] In the video content annotation method of this invention, the steps of using graph neural networks for semantic information transmission and feature aggregation specifically include: calculating the intra-layer attention weights between adjacent nodes in the same layer; if the feature similarity between two nodes is higher than a fifth preset threshold, a higher intra-layer attention weight is assigned; if the feature similarity is lower than the fifth preset threshold but higher than a sixth preset threshold, the weights are adjusted through semantic edge type; calculating the inter-layer attention weights between nodes in different layers to establish a cross-layer information transmission channel; calculating the temporal attention weights of nodes in different temporal units and performing temporal aggregation of node features; and aggregating the features of adjacent nodes based on the attention weights to update the feature representation of the current node.

[0011] In the video content annotation method of the present invention, the step of predicting labels based on the updated node feature representation specifically includes: for the scene node, if the cosine similarity between its feature vector and the scene category library is higher than a seventh preset threshold, then the corresponding category label is output; if the similarity is lower than an eighth preset threshold, then it is marked as an unknown category and manual review is triggered; for the action node, if the similarity between the action feature and the best match of the action category library is higher than a ninth preset threshold, then the action label is output; for the object attribute node, if the category of the object node has been determined and the attributes have been completely extracted, then the object label and its attribute label are output.

[0012] The video content annotation method described in this invention also includes a cross-modal adversarial reasoning step: after acquiring the multimodal feature data, a modality discriminator is constructed to distinguish feature representations from different modalities; adversarial training is used to align the feature representations of each modality in a shared semantic space; when a modality is missing, the semantic representation of the missing modality is inferred based on the complete modality using a cross-modal reasoning network.

[0013] The video content annotation method described in this invention further includes an adaptive modality weighting step: evaluating the quality of each modality feature, wherein the quality evaluation includes visual quality evaluation of the visual features, audio quality evaluation of the audio features, and text quality evaluation of the text features; and dynamically adjusting the fusion weights of each modality based on the quality evaluation results.

[0014] In the video content annotation method of the present invention, the step of performing multi-granularity quality evaluation on the annotation results specifically includes: at the frame level, performing semantic conflict detection and temporal consistency verification on the annotation results of each frame; if there is a semantic conflict between multiple labels annotated in the same frame, the confidence score of that frame is reduced; at the shot level, performing keyframe annotation consistency rate calculation and object trajectory consistency verification on the annotation results of each shot; if the annotation consistency rate of keyframes in a shot is higher than the tenth preset threshold, the confidence of that shot is high; at the video level, performing logical verification of the transitions and scene transition frequency verification on the annotation results of the entire video; if the temporal logic of scene transitions is abnormal, the confidence of the entire video is reduced.

[0015] The video content annotation method described in this invention also includes an active closed-loop learning step: calculating the comprehensive uncertainty score of all annotated samples, wherein the uncertainty score comprehensively considers label boundary distance, multi-label competition degree and annotation consistency; constructing a sampling priority queue based on the comprehensive uncertainty score, and giving priority to manually reviewing high uncertainty samples; and using the reviewed annotation data for incremental model training.

[0016] In the video content annotation method of the present invention, the triggering conditions for incremental training of the model include: statistically analyzing the proportion of correctly corrected samples; if the proportion of correctly corrected samples is higher than the eleventh preset threshold and the number of samples being reviewed reaches the twelfth preset threshold, then a complete model fine-tuning process is executed; if the proportion of correctly corrected samples is lower than the eleventh preset threshold, then the cause of the error is analyzed and the model is adjusted accordingly before fine-tuning is performed.

[0017] The present invention has the following advantages over the prior art: By constructing a hierarchical semantic graph, video content is mapped to graph structure nodes. Graph neural networks are used to transmit semantic information between levels, achieving adaptive annotation from coarse to fine, from scene to action to object attributes. This significantly improves the semantic hierarchy and consistency of the annotation results. The hierarchical semantic graph can explicitly express the hierarchical structure of video content, making the annotation results more interpretable. Detailed Implementation

[0018] The following embodiments are only used to illustrate the technical solutions of the present invention more clearly, and are therefore only examples and should not be used to limit the scope of protection of the present invention.

[0019] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0020] In a first aspect, the present invention provides an adaptive video content annotation method based on hierarchical semantic graphs, comprising the following steps: The process involves: acquiring multimodal feature data of the video to be labeled, including visual, audio, and text features; constructing an initial semantic representation of the video based on the multimodal feature data, including scene nodes, object nodes, action nodes, and attribute nodes; constructing a hierarchical semantic graph, comprising a top-level scene layer, an intermediate action layer, and a bottom-level object attribute layer, with nodes connected by semantic edges to form a hierarchical semantic relationship; using a graph neural network to perform semantic information transfer and feature aggregation on the hierarchical semantic graph to generate updated feature representations for each node, including intra-layer attention, inter-layer attention, and temporal attention mechanisms; performing label prediction based on the updated node feature representations to output the labeling results of the video content; and performing multi-granularity quality evaluation on the labeling results, including frame-level evaluation, shot-level evaluation, and video-level evaluation. When the quality evaluation result is lower than a preset threshold, a manual review process is triggered, and the review result is fed back to the model for incremental training.

[0021] In the video content annotation method of the present invention, the step of constructing the initial semantic representation of the video based on the multimodal feature data specifically includes: performing scene classification detection on the visual features; if the detection confidence is higher than a first preset threshold, then establishing a corresponding node in the scene layer; if the detection confidence is lower than the first preset threshold but higher than a second preset threshold, then marking it as a scene to be confirmed and entering the manual review queue; if the detection confidence is lower than the second preset threshold, then determining it as an unknown scene type; detecting action categories based on motion information in the visual features; if the action detection confidence is higher than a third preset threshold, then confirming the establishment of the action node; if the action detection confidence is between the third preset threshold and the fourth preset threshold, then combining the audio features for auxiliary judgment; establishing the object node based on the target detection result in the visual features, and extracting object attribute features to establish the attribute node.

[0022] In the video content annotation method of the present invention, the rules for establishing semantic edges between nodes of the hierarchical semantic graph include: if the scene node and the action node have a co-occurrence relationship in the spatiotemporal dimension, then a scene-action edge is established; if the action node and the object node have a subject-verb-object relationship, then an action-object edge is established; if the object node and the attribute node have a subordinate relationship, then an object-attribute edge is established.

[0023] In the video content annotation method of this invention, the steps of using graph neural networks for semantic information transmission and feature aggregation specifically include: calculating the intra-layer attention weights between adjacent nodes in the same layer; if the feature similarity between two nodes is higher than a fifth preset threshold, a higher intra-layer attention weight is assigned; if the feature similarity is lower than the fifth preset threshold but higher than a sixth preset threshold, the weights are adjusted through semantic edge type; calculating the inter-layer attention weights between nodes in different layers to establish a cross-layer information transmission channel; calculating the temporal attention weights of nodes in different temporal units and performing temporal aggregation of node features; and aggregating the features of adjacent nodes based on the attention weights to update the feature representation of the current node.

[0024] In the video content annotation method of the present invention, the step of predicting labels based on the updated node feature representation specifically includes: for the scene node, if the cosine similarity between its feature vector and the scene category library is higher than a seventh preset threshold, then the corresponding category label is output; if the similarity is lower than an eighth preset threshold, then it is marked as an unknown category and manual review is triggered; for the action node, if the similarity between the action feature and the best match of the action category library is higher than a ninth preset threshold, then the action label is output; for the object attribute node, if the category of the object node has been determined and the attributes have been completely extracted, then the object label and its attribute label are output.

[0025] The video content annotation method described in this invention also includes a cross-modal adversarial reasoning step: after acquiring the multimodal feature data, a modality discriminator is constructed to distinguish feature representations from different modalities; adversarial training is used to align the feature representations of each modality in a shared semantic space; when a modality is missing, the semantic representation of the missing modality is inferred based on the complete modality using a cross-modal reasoning network.

[0026] The video content annotation method described in this invention further includes an adaptive modality weighting step: evaluating the quality of each modality feature, wherein the quality evaluation includes visual quality evaluation of the visual features, audio quality evaluation of the audio features, and text quality evaluation of the text features; and dynamically adjusting the fusion weights of each modality based on the quality evaluation results.

[0027] In the video content annotation method of the present invention, the step of performing multi-granularity quality evaluation on the annotation results specifically includes: at the frame level, performing semantic conflict detection and temporal consistency verification on the annotation results of each frame; if there is a semantic conflict between multiple labels annotated in the same frame, the confidence score of that frame is reduced; at the shot level, performing keyframe annotation consistency rate calculation and object trajectory consistency verification on the annotation results of each shot; if the annotation consistency rate of keyframes in a shot is higher than the tenth preset threshold, the confidence of that shot is high; at the video level, performing logical verification of the transitions and scene transition frequency verification on the annotation results of the entire video; if the temporal logic of scene transitions is abnormal, the confidence of the entire video is reduced.

[0028] The video content annotation method described in this invention also includes an active closed-loop learning step: calculating the comprehensive uncertainty score of all annotated samples, wherein the uncertainty score comprehensively considers label boundary distance, multi-label competition degree and annotation consistency; constructing a sampling priority queue based on the comprehensive uncertainty score, and giving priority to manually reviewing high uncertainty samples; and using the reviewed annotation data for incremental model training.

[0029] In the video content annotation method of the present invention, the triggering conditions for incremental training of the model include: statistically analyzing the proportion of correctly corrected samples; if the proportion of correctly corrected samples is higher than the eleventh preset threshold and the number of samples being reviewed reaches the twelfth preset threshold, then a complete model fine-tuning process is executed; if the proportion of correctly corrected samples is lower than the eleventh preset threshold, then the cause of the error is analyzed and the model is adjusted accordingly before fine-tuning is performed.

[0030] Secondly, the present invention provides a video content annotation system based on big data, including a feature extraction module, a semantic graph construction module, a graph neural network inference module, an annotation output module, a quality assessment module, and a feedback learning module.

[0031] The feature extraction module is used to obtain multimodal feature data of the video to be labeled; The semantic graph construction module is used to construct a hierarchical semantic graph based on the multimodal feature data; The graph neural network inference module is used to perform semantic information transmission and feature aggregation on the hierarchical semantic graph. The annotation output module is used to output annotation results based on the updated node feature representation; The quality assessment module is used to assess the quality of the annotation results; The feedback learning module is used to trigger manual review when the quality assessment result is lower than a preset threshold, and then feed the review result back to the system for model updates.

[0032] The feature extraction module is responsible for receiving the video data to be labeled and extracting multimodal features from the video. Specifically, this module first performs shot segmentation on the video to determine the basic editing units; then, it selects keyframes for each shot and extracts the visual features of the keyframes; simultaneously, it extracts the corresponding audio signals for each shot, extracting audio features such as MFCC; if the video has subtitles or title text, it extracts the corresponding text features. The feature extraction module outputs feature vector sequences of the three modalities as the basis for subsequent processing.

[0033] The specific workflow of the feature extraction module includes the following decision logic: First, the system performs a format check on the input video, determining whether parameters such as the video's encoding format, resolution, and frame rate meet the processing requirements. If the video parameters are outside the preset range, the system performs format preprocessing to convert the video into a unified standard format. The format preprocessing logic is as follows: if the video resolution is higher than 4K, it is downsampled to 1080P to reduce computational overhead; if the video resolution is lower than 720P, it is marked as a low-resolution video and the super-resolution enhancement module is enabled in subsequent processing; if the video frame rate is higher than 60fps, it is downsampled to 30fps to maintain a uniform temporal sampling rate.

[0034] Then, the system performs shot segmentation on the video. The shot segmentation module uses a segmentation algorithm based on inter-frame differences, specifically including the following judgment logic: calculating the color histogram difference and edge change difference between adjacent frames; if the difference value is greater than the first segmentation threshold, it is determined to be a shot transition boundary; if the difference value is between the first and second segmentation thresholds, further analysis of motion information is performed; if the motion information shows smooth motion, it is determined to be the same shot; if the motion information shows abrupt change, it is determined to be a shot transition; if the difference value is less than the second segmentation threshold, it is determined to be a frame within the same shot. After shot segmentation is completed, the system extracts keyframes for each shot, using a combination of uniform sampling and saliency detection.

[0035] The system extracts visual, audio, and text features separately. Visual feature extraction uses a pre-trained convolutional neural network model to output feature vectors. The judgment logic for visual feature extraction is as follows: first, it detects whether the video is a color video; if it is a grayscale video, only the luminance channel features are extracted; if there are text regions in the video, the text region detection module is activated to extract text features; if there are face regions in the video, the face detection module is activated to extract face features; if there are specific object categories in the video, the object detection module is activated to extract object features.

[0036] Audio feature extraction includes the extraction of MFCC features, spectral features, and prosodic features. The judgment logic for audio feature extraction is as follows: first, check if the audio channel exists; if the audio signal exists, perform preprocessing; if human voice is detected, activate the speech recognition module to extract speech content features; if background music is detected, activate the music feature extraction module; if environmental sound effects are detected, activate the environmental sound classification module.

[0037] Text feature extraction includes subtitle text extraction, title text extraction, and bullet screen text extraction. The judgment logic for text feature extraction is as follows: prioritize extracting subtitle text with accurate timing; if the subtitle text is missing or incomplete, supplement it with title text and descriptive text; if bullet screen text exists, analyze the sentiment and topic distribution of the bullet screen text as supplementary semantic information.

[0038] The semantic graph construction module receives multimodal feature vector sequences and constructs a hierarchical semantic graph for the video. The hierarchical semantic graph has a three-layer structure: the top layer is the scene layer, which contains scene category nodes, such as indoor, outdoor, seaside, city, countryside, and forest; the middle layer is the action layer, which contains action category nodes, such as walking, talking, exercising, driving, shopping, and cooking; and the bottom layer is the object attribute layer, which contains object category nodes and attribute category nodes, such as object nodes like people, vehicles, buildings, animals, and items, as well as attribute nodes like color, size, material, state, and emotion.

[0039] The construction process of the hierarchical semantic graph includes the following detailed judgment logic: The judgment logic for scene layer construction: First, scene classification and detection are performed on visual features. The system employs a multi-scale scene classification model to extract global scene features and local region features. If the detection confidence is higher than a first preset threshold, a corresponding node is created at the scene layer and the node status is marked as confirmed. If the detection confidence is between the first and second preset thresholds, it is marked as a scene to be confirmed and enters the manual review queue for approval. If the detection confidence is lower than the second preset threshold, it is determined to be an unknown scene type. The system queries the knowledge base to see if similar scenes exist. If similar scenes exist, similar nodes are created and marked as speculative; if no similar scenes exist, they are marked as unrecognizable.

[0040] During scene detection, if multiple scene categories are detected to exist simultaneously, the system executes multi-scene coexistence processing logic: determine the confidence difference between each scene; if the difference is less than the third preset threshold, multiple scene nodes are established and marked as multi-scene coexistence; if the difference is greater than the third preset threshold, the scene with the highest confidence is selected as the primary scene, and the rest are secondary scenes.

[0041] The system also performs scene timing consistency analysis. The judgment logic is as follows: if the scene label of adjacent shots or adjacent frames changes drastically, scene anomaly detection is triggered; if the scene change is consistent with the narrative logic of the video content, it is determined to be a normal transition; if the scene change is inconsistent with the narrative logic, it is marked as a suspicious scene and entered into the review queue.

[0042] The judgment logic for constructing the action layer: Then, the action category is detected based on motion information in visual features. The system employs an action detection method that combines optical flow features and spatiotemporal convolutional networks. If the action detection confidence score is higher than a fourth preset threshold, the action node is confirmed and its status is marked as confirmed.

[0043] The detailed judgment logic of motion detection is as follows: First, the system extracts the motion region in the video and determines the motion foreground through frame difference and saliency detection; if the area of ​​the motion region is less than the fourth motion threshold, it is judged as a minor motion and no motion node is established; if the area of ​​the motion region is between the fourth motion threshold and the fifth motion threshold, it is judged as a local motion, an motion node is established but marked as low confidence; if the area of ​​the motion region is greater than the fifth motion threshold, it is judged as a significant motion, an motion node is established and marked as high confidence.

[0044] Secondly, the system analyzes the motion trajectory to identify the specific action type. The judgment logic is as follows: extract the trajectory features of the moving target; if the trajectory shows linear features, it is determined to be a walking or running action; if the trajectory shows curved features, it is determined to be a meandering or circling action; if the trajectory shows broken line features, it is determined to be a turning or changing direction action; if the trajectory features are unclear, the attitude estimation results are used for auxiliary judgment.

[0045] If the action detection confidence is between the fourth and sixth preset thresholds, then the multimodal auxiliary judgment logic is executed: the system synchronously analyzes the audio features. If there are footsteps in the audio and the rhythm is stable, the action confidence is increased. If there are dialogues in the audio, the lip-sync detection results are combined to determine whether there is a conversation. If there is a clear spatiotemporal correspondence between the audio and the visual action, the action label is confirmed.

[0046] If the confidence level is still insufficient after auxiliary judgment, it is marked as an ambiguous action and added to the confirmation queue. The system further analyzes the context information: if there is a preparation action in the previous frame of the current action, the confidence level is increased based on the temporal context; if there is a closing action in the subsequent frame of the current action, the confidence level is also increased; if there is a lack of context support, it remains in an ambiguous state and is marked as awaiting manual confirmation.

[0047] The judgment logic for constructing the object property layer: Finally, object nodes are established based on the object detection results from visual features, and attribute nodes are established by extracting object attribute features. Object detection employs a multi-scale detection network, capable of identifying common object categories in the video.

[0048] The judgment logic for establishing an object node is as follows: if the confidence of the target detection box is higher than the seventh preset threshold, then the object node is confirmed to be established; if the detection confidence is between the seventh preset threshold and the eighth preset threshold, then the object node is established but marked as speculative; if the detection confidence is lower than the eighth preset threshold, then the object node is not established but recorded as a potential object.

[0049] The judgment logic for establishing attribute nodes is as follows: First, extract attributes from the established object nodes; if the completeness of the extracted object attributes is higher than the ninth preset threshold, then establish the corresponding attribute node; if the completeness of the extracted attributes is between the ninth preset threshold and the tenth preset threshold, then establish some attribute nodes and mark them as incomplete; if the completeness of the extracted attributes is lower than the tenth preset threshold, then only establish object nodes, do not establish attribute nodes, and mark the attributes as missing in the annotation results.

[0050] The specific judgment logic for attribute extraction is as follows: For color attributes, the system analyzes the color distribution of the object region. If the color distribution is concentrated, the dominant color is extracted as the attribute value; if the color distribution is scattered, the first three dominant colors are extracted as the attribute values. For size attributes, the system calculates the actual size of the object based on the size of the target detection box and the known resolution of the video. If the calculation result is within a reasonable range, a size attribute node is established. For material attributes, the system analyzes the texture features of the object. If the texture features match a known material library, a material attribute node is established. For state attributes, the system analyzes the temporal change features of the object. If there are significant changes, a state attribute node is established. For emotion attributes, the system analyzes the facial expressions and posture features of the object. If obvious emotional expression is detected, an emotion attribute node is established.

[0051] The semantic graph construction module is also responsible for establishing semantic edges between nodes at each layer. The establishment of semantic edges follows these rules: The logic for establishing scene-action edges is as follows: If scene nodes and action nodes have a co-occurrence relationship in the spatiotemporal dimension, then a scene-action edge is established. Specifically, the judgment logic is as follows: If the temporal interval in which the action occurs falls within the continuous temporal interval of the scene, and the spatial location conforms to the spatial range of the scene, then a strong correlation edge is established; if the temporal interval partially overlaps or the spatial location partially coincides, then a weak correlation edge is established; if neither the temporal interval nor the spatial location coincides, then no edge is established.

[0052] The logic for establishing action-object edges is as follows: If an action node and an object node have a subject-verb-object relationship, then an action-object edge is established. Specifically, the logic is: analyze the executor and receiver of the action; if the object is the executor of the action, then a subject edge is established; if the object is the receiver of the action, then an object edge is established; if the object is merely an accompaniment to the action, then an accompaniment edge is established.

[0053] The logic for establishing object-attribute edges is as follows: If an object node and an attribute node have a subordinate relationship, then an object-attribute edge is established. Specifically, the judgment logic is: the attribute must be an inherent characteristic or state of the object; if there is a clear belonging relationship between the attribute and the object, then a strong association edge is established; if there is an indirect relationship between the attribute and the object, then a weak association edge is established.

[0054] The graph neural network inference module receives hierarchical semantic graphs and uses a graph attention network for semantic information transmission and feature aggregation. The core idea of ​​this module is to enable nodes to perceive contextual information through a message-passing mechanism, thereby achieving more accurate label prediction.

[0055] The graph neural network inference module employs a multi-level graph attention mechanism. This multi-level attention mechanism includes three components: intra-level attention, inter-level attention, and temporal attention.

[0056] The judgment logic of the attention mechanism within a hierarchy: Intra-layer attention is used to calculate the attention weights between adjacent nodes within the same layer. First, the system calculates the similarity of node features. The similarity calculation uses the cosine similarity method, mapping node features to a unit sphere and then calculating the inner product.

[0057] After similarity calculation, the attention weight allocation stage begins. The judgment logic is as follows: if the feature similarity between two nodes is higher than the eleventh preset threshold, a higher attention weight is assigned; if the feature similarity is between the eleventh and twelfth preset thresholds, the weight is adjusted according to the semantic edge type: if the semantic edge type is a strong association type, the weight is adjusted to the first level; if the semantic edge type is a weak association type, the weight is adjusted to the second level; if the semantic edge type is an indirect association type, the weight is adjusted to the third level; if the feature similarity is lower than the twelfth preset threshold, the weight is adjusted to the low level.

[0058] In the weight allocation process, the system also considers the importance of nodes. The judgment logic for importance assessment is as follows: count the number of times a node is referenced. If the number of references is higher than the thirteenth preset threshold, the node is an important node and is assigned the first weight bonus coefficient; if the number of references is between the thirteenth and fourteenth preset thresholds, it is assigned the second weight bonus coefficient; if the number of references is lower than the fourteenth preset threshold, it is assigned the third weight bonus coefficient.

[0059] The judgment logic of inter-layer attention mechanism: Interlayer attention is used to establish information transmission channels between nodes at different layers. The system adopts a cross-layer connection approach, enabling lower-level nodes to receive semantic information from higher-level nodes, and higher-level nodes to receive detailed information from lower-level nodes.

[0060] The calculation logic for inter-layer attention weights is as follows: First, analyze the degree of semantic association between nodes. If there is a subject-predicate relationship between the bottom object node and the intermediate action node, a strong semantic association is established, and the first inter-layer attention weight is assigned; if there is a spatial subordinate relationship between the bottom object node and the top scene node, a medium semantic association is established, and the second inter-layer attention weight is assigned; if there is only a weak semantic association between the bottom object node and the top scene node, the third inter-layer attention weight is assigned.

[0061] The system also performs inter-layer information flow control. The judgment logic is as follows: bottom-up information flow is mainly used to convey the detailed features of objects. If the feature information of the bottom object nodes is high, the amount of information transmitted upwards is increased. Top-down information flow is mainly used to convey the context information of the scene. If the confidence of the top scene nodes is high, the amount of information transmitted downwards is increased.

[0062] The judgment logic of the temporal attention mechanism: Considering the temporal characteristics of video content, the system also introduces a temporal attention mechanism. Temporal attention is used to model the semantic changes of nodes at different temporal positions.

[0063] The calculation logic for temporal attention weights is as follows: First, the video is temporally sliced, dividing it into multiple temporal units. Then, the activation level of nodes in each temporal unit is analyzed. The judgment logic is as follows: if the activation level of a node in a certain temporal unit is higher than the fifteenth preset threshold, then that temporal unit is the node's active period; if the activation level is between the fifteenth and sixteenth preset thresholds, then that temporal unit is the node's transition period; if the activation level is lower than the sixteenth preset threshold, then that temporal unit is the node's silent period.

[0064] The system performs temporal aggregation of node features based on temporal attention weights. The decision logic is as follows: node features in active and transitional periods are assigned higher weights for aggregation; node features in inactive periods are assigned lower weights or ignored. Through the temporal attention mechanism, the system can capture the temporal variation patterns of nodes, improving the temporal consistency of the annotation results.

[0065] After calculating the attention weights, the system aggregates the features of neighboring nodes based on these attention weights and updates the feature representation of the current node. Feature aggregation uses a weighted summation method, summing the features of neighboring nodes according to their attention weights, and then applying an activation function to obtain the updated node features.

[0066] The updated features are then passed to the corresponding nodes in the next layer. After processing by a multi-layer graph attention network, each node obtains a feature representation that incorporates global contextual information, providing richer semantic features for subsequent label prediction.

[0067] The annotation output module is responsible for outputting annotation results based on the updated node feature representations. This module performs a classification judgment for each node: The logic for determining scene node label output: For scene nodes, the system calculates the cosine similarity between their feature vectors and the features of each category in the scene category library. The judgment logic is as follows: if the best match similarity is higher than the seventeenth preset threshold, the corresponding category label is output and the confidence level is marked as high; if the similarity is between the seventeenth and eighteenth preset thresholds, the main category label is output and the confidence level is marked as medium in the annotation results, while the secondary high confidence category is output as a reference; if the similarity is lower than the eighteenth preset threshold, it is marked as an unknown category and manual review is triggered.

[0068] The system also performs conflict detection for scene labels. The judgment logic is as follows: if mutually exclusive scene labels exist in the same video, the conflict handling mechanism is triggered: analyze the temporal distribution of the two scene labels; if the temporal distributions do not overlap, retain both labels and mark them as scene transitions in the annotation results; if the temporal distributions overlap, mark them as abnormal annotations and enter the manual review queue.

[0069] The logic for determining the output of action node labels: For action nodes, the judgment logic is as follows: if the similarity between the action feature and the best match in the action category library is higher than the nineteenth preset threshold, then the action label is output and the confidence level is marked as high.

[0070] The system also performs multi-label processing for action tags. The judgment logic is as follows: if the action feature matches multiple categories and the similarity difference is less than the twentieth preset threshold, then the multi-label annotation result is output; if the action feature matches multiple categories but the similarity difference is greater than the twentieth preset threshold, then the best matching category is selected and output as a single label in the annotation result.

[0071] The logic for judging the temporal consistency of action tags is as follows: if the confidence fluctuation of the same action tag in different temporal units is less than the twenty-first preset threshold, then the temporal consistency of the action tag is good; if the confidence fluctuation is greater than the twenty-first preset threshold, it is marked as temporally unstable and enters the suspicious queue.

[0072] The logic for judging the output of object attribute node tags: For object attribute nodes, if the object category is determined and the attributes are fully extracted, the object label and its attribute labels are output. The judgment logic is as follows: first, the object category label is output; if the confidence level of the attribute node is higher than the twenty-second preset threshold, the corresponding attribute label is output; if the confidence level of the attribute node is between the twenty-second and twenty-third preset thresholds, the attribute label is output but marked as speculative; if the confidence level of the attribute node is lower than the twenty-third preset threshold, no attribute label is output and the attribute is marked as missing in the annotation results.

[0073] The system also performs object-attribute consistency checks. The judgment logic is as follows: if there is a semantic conflict between the object label and the attribute label, the conflict handling mechanism is triggered: analyze the conflict type, if it is an impossible combination, mark it as an incorrect label and enter the review queue; if it is a low-probability combination, retain the label but mark it as low confidence.

[0074] The system also includes a cross-modal adversarial reasoning module to handle modality loss and noise interference scenarios, thereby improving the system's robustness.

[0075] The cross-modal adversarial inference module works as follows: First, after the feature extraction module extracts multimodal features, the cross-modal adversarial inference module maps the features of each modality to a shared semantic space. The mapping process employs an adversarial training strategy, specifically including the construction of a modality discriminator and a feature generator.

[0076] The judgment logic of the modality discriminator: The modality discriminator receives feature vectors from different modalities and determines which modality the current feature vector originates from. The modality discriminator employs a multi-classifier structure and outputs the probability distribution of each modality.

[0077] The judgment logic is as follows: if the modality discriminator can accurately distinguish (discrimination accuracy is higher than 95%), it means that the feature representations of each modality have not yet been aligned in the semantic space and adversarial training needs to be strengthened; if the discrimination accuracy of the modality discriminator is between 80% and 95%, it means that the feature representations of some modalities have been aligned, but further training is still needed; if the discrimination accuracy of the modality discriminator is lower than 80% and close to the level of random guessing (for example, the random accuracy is about 33% in the case of three modalities), it means that the feature representations of each modality have been effectively aligned.

[0078] Optimization logic for the feature generator: The optimization goal of the feature generator is to enable the generated feature vectors to deceive the modality discriminator, preventing it from accurately determining the modality source. The feature generator adopts an encoder-decoder architecture, mapping the original features of each modality to a shared semantic space.

[0079] The optimization logic of the feature generator is as follows: First, calculate the distance between the generated features and the target semantic space. If the distance is greater than the 24th preset threshold, strengthen the learning of the semantic space mapping; if the distance is less than the 24th preset threshold, reduce the mapping strength to avoid overfitting. Second, analyze the distribution of the generated features across categories. If the category distribution is too concentrated (a certain category accounts for more than 70%), introduce a category balancing loss for adjustment; if the category distribution is relatively uniform (the difference in the proportion of each category is less than 20%), maintain the current mapping strategy.

[0080] Reasoning logic of cross-modal reasoning networks: When a certain modality is missing, the cross-modal reasoning network initiates the reasoning mechanism.

[0081] The logic for modality missing detection is as follows: First, the completeness of each modality feature is checked. If the visual feature is empty or the feature dimension is abnormal (e.g., the dimension is 0 or much smaller than the normal dimension), it is determined to be a visual modality missing; if the audio feature is empty or the energy is too low (e.g., the average energy is lower than the noise threshold), it is determined to be an audio modality missing; if the text feature is empty or the number of characters is 0, it is determined to be a text modality missing.

[0082] The judgment logic for cross-modal reasoning is as follows: If visual features are missing but audio and text features are complete, an audio-text cross-modal inference network is used to infer visual semantics. The specific inference process is as follows: The system first analyzes the semantic information in the audio features, such as identifying scene description words in the speech content (e.g., "we are at the beach" corresponds to the "beach" scene); then it combines scene keywords in the text features (e.g., "office" in the subtitles corresponds to the "indoor" scene); finally, it fuses the audio and text semantics to generate a predicted visual semantic representation. If the inference confidence is higher than the 25th preset threshold (e.g., 0.70), the inference result is used as a substitute for the visual modality; if the inference confidence is lower than the 25th preset threshold but higher than the 26th preset threshold (e.g., 0.50), it is marked as a predicted result in the annotation results and its confidence level is reduced; if the inference confidence is lower than the 26th preset threshold, it is determined that inference is impossible, marked as a missing modality, and manual supplementary annotation is triggered.

[0083] If audio features are missing but visual and text features are complete, a visual-text cross-modal inference network is used to infer audio semantics. The specific inference process is as follows: the system first analyzes motion information (e.g., lip movements and posture) in visual features to infer the possible sound types; then it combines this with dialogue content or sound effect descriptions in text features; finally, it fuses visual and text semantics to generate a predicted audio semantic representation. If the inference confidence is higher than the 25th preset threshold, the inference result is used as a substitute for the audio modality; if the inference confidence is between the 25th and 26th preset thresholds, it is marked as a prediction; if the inference confidence is lower than the 26th preset threshold, it is determined that inference is impossible.

[0084] If text features are missing but visual and audio features are complete, a visual-audio cross-modal inference network is used to infer the text semantics. The specific inference process is as follows: The system first analyzes the text regions in the visual features; if text is present, OCR recognition is performed; if no text regions exist but a dialogue scene exists, speech recognition is performed by combining the voice from the audio; if text cannot be inferred from visual and audio, descriptive text is generated based on the semantic content of the visual and audio. If the inference confidence is higher than the 25th preset threshold, the inference result is used as a substitute for the text modality; if the inference confidence is between the 25th and 26th preset thresholds, it is marked as speculation; if the inference confidence is lower than the 26th preset threshold, it is determined that inference is impossible.

[0085] Adaptive weighting for multimodal fusion: The system also features an adaptive modal weighting module that dynamically adjusts the fusion weights of each modality based on their reliability in specific scenarios.

[0086] The adaptive modality weighting judgment logic is as follows: First, the quality of each modality is evaluated. The judgment logic for visual quality assessment is: if the video image is clear (contrast above the threshold), without severe occlusion, and without motion blur, the visual quality is high; if there is slight blur or occlusion, the visual quality is medium; if there is severe blur or occlusion, the visual quality is low. The judgment logic for audio quality assessment is: if the audio signal-to-noise ratio is higher than 30dB, without obvious noise, and without clipping distortion, the audio quality is high; if the signal-to-noise ratio is between 20dB and 30dB, the audio quality is medium; if the signal-to-noise ratio is lower than 20dB, the audio quality is low. The judgment logic for text quality assessment is: if the text completeness is high (character recognition accuracy higher than 95%) and without obvious errors, the text quality is high; if the completeness is between 80% and 95%, the text quality is medium; if the completeness is lower than 80%, the text quality is low.

[0087] Then, the fusion weights for each modality are calculated based on the quality assessment results. The judgment logic is as follows: if a modality has high quality, its fusion weight is 1.2 times the baseline weight; if its quality is medium, its fusion weight is 1.0 times the baseline weight; if its quality is low, its fusion weight is 0.6 times the baseline weight. The normalization of the fusion weights is as follows: the sum of the weights of each modality should be equal to a preset normalization constant (usually 3.0).

[0088] Thirdly, the present invention provides a video annotation method based on cross-modal adversarial reasoning, as a further improvement to the above technical solution, comprising: After acquiring multimodal feature data, a cross-modal feature alignment operation is further performed. Specifically, this includes constructing a modality discriminator to distinguish feature representations from different modalities; using adversarial training to achieve distribution alignment of feature representations of each modality in a shared semantic space; and when a certain modality is missing, using a cross-modal inference network to infer the semantic representation of the missing modality based on the complete modality to ensure the robustness of the annotation system.

[0089] By introducing a cross-modal adversarial learning mechanism, feature distribution alignment between modalities is established, enabling effective fusion of feature representations from different modalities in a shared semantic space and improving the utilization efficiency of multimodal information. The cross-modal inference network can infer the semantic representation of a missing modality based on the complete modality when a modality is missing, significantly improving the system's robustness.

[0090] Fourthly, this invention provides a labeling quality control method based on active closed-loop learning, as a further improvement to the above technical solution, comprising: A multi-granularity quality assessment module is designed to calculate the confidence level of the annotation results at three granularities: each frame, each shot, and the entire video. Based on the confidence level ranking, low-confidence samples are prioritized for manual review. A closed-loop iterative mechanism of "machine annotation - quality assessment - manual review - model update" is established to achieve continuous optimization of annotation quality.

[0091] By designing an active closed-loop learning mechanism, real-time evaluation and dynamic control of annotation quality were achieved, effectively reducing the cascading propagation of erroneous annotations, significantly reducing the workload of manual review, and markedly improving annotation accuracy. The multi-granularity quality evaluation system can comprehensively monitor annotation quality from multiple dimensions, including frame-level, shot-level, and video-level.

[0092] In summary, this invention can adapt to complex scenarios such as modality loss, noise interference, and domain migration, exhibiting strong robustness and generalization ability. The incremental learning mechanism enables the system to continuously learn from manual review, thereby constantly improving annotation performance.

[0093] The quality assessment module and the feedback learning module work together to achieve an active closed-loop learning mechanism, enabling continuous optimization of annotation quality.

[0094] The core of the active closed-loop learning mechanism lies in actively selecting the most valuable samples for manual annotation, rather than random selection.

[0095] Multi-granularity quality assessment module: The quality assessment module calculates the confidence score of the annotation results at three granularities: frame level, shot level, and video level.

[0096] Frame-level evaluation logic: Frame-level evaluation checks the consistency of the annotation results for each frame in the video.

[0097] The semantic conflict detection logic is as follows: First, a conflict rule base is established, containing mutually exclusive label pairs (such as indoor-outdoor, black-white, still-moving, etc.). Then, it checks whether there are conflict label pairs in the current frame. If a conflict label pair exists, conflict analysis is triggered: the confidence difference between the conflict labels is judged. If the difference between the high-confidence label and the low-confidence label is greater than the 27th preset threshold, the high-confidence label is retained and the low-confidence label is marked as suspicious; if the difference is less than the 27th preset threshold, the frame is marked as requiring manual review.

[0098] The logic for temporal consistency verification is as follows: The system maintains the annotation state trajectory of the previous and next frames. If there is a significant difference between the annotation of the current frame and the annotation of the previous frame (the scene label changes or the number of objects changes beyond a threshold), change detection is triggered: The rationality of the change is analyzed. If the change conforms to the shot switching pattern (such as scene switching), it is judged as a normal change and the change point is recorded; if the change does not conform to the shot switching pattern (such as the scene label fluctuating drastically within a single frame), it is judged as abnormal and marked as suspicious.

[0099] The judgment logic of the confidence distribution test is as follows: analyze the confidence distribution of each label in the current frame. If the number of high confidence labels (confidence greater than 0.8) is too small (e.g., less than 2), the overall confidence of the frame is reduced; if there are a large number of medium confidence labels (confidence between 0.5 and 0.8), it is marked as a frame with blurred boundaries.

[0100] The judgment logic of lens-level evaluation: Shot-level evaluation verifies the completeness of the annotation results for each shot in the video.

[0101] The logic for calculating the keyframe annotation consistency rate is as follows: First, select keyframes within the shot, typically the first, middle, and last frames. Then, calculate the number of identical labels between keyframes. The consistency rate is calculated as: number of identical labels divided by the total number of labels. If the consistency rate is higher than the 28th preset threshold, the shot has high confidence; if the consistency rate is between the 28th and 29th preset thresholds, it is marked as requiring manual verification; if the consistency rate is lower than the 29th preset threshold, it is determined to be an annotation anomaly and needs to be reprocessed.

[0102] The logic for verifying the consistency of object trajectories within a shot is as follows: analyze whether the labeling of the same object within the shot is consistent in different frames. If the object label changes (e.g., from "person" to "unknown"), trajectory analysis is triggered: determine the reason for the change. If the object is significantly obstructed within the shot or leaves the frame, the change is reasonable; if the object does not change significantly but the label changes, it is marked as an incorrect labeling.

[0103] The logic for checking scene continuity is as follows: check whether the scene labels of all frames within the shot are consistent. If multiple different scene labels exist, scene analysis is triggered: determine whether it is a normal scene transition (such as a transition from indoors to outdoors). If so, retain multiple scene labels; if it is not a transition but a sudden change without a shot change, mark it as suspicious.

[0104] The judgment logic of video level assessment: Video-level evaluation performs a logical check on the annotation results of the entire video.

[0105] The logic for verifying the flow of the video is as follows: Analyze whether the scene annotations at the beginning and end of the video conform to the norm. If the beginning of the video is annotated as an indoor scene or the title screen, it conforms to the norm; if the beginning is annotated as an outdoor scene, it is marked as abnormal. If the end of the video is annotated as an indoor scene or the ending screen, it conforms to the norm; if the end is annotated as an abnormal scene (such as a battle scene), it is marked as abnormal.

[0106] The logic for the scene transition frequency test is as follows: count the number of times the scene label changes in the video. The judgment logic is: if the scene transition frequency is higher than the 30th preset threshold (e.g., more than 3 times every 10 seconds), it is marked as an abnormal scene fluctuation, which may indicate a labeling error; if the transition frequency is within a reasonable range, the test is passed.

[0107] The logic for the action sequence logic check is as follows: analyze whether the temporal sequence of the action tags conforms to logic. If an unreasonable action sequence is detected (such as running directly after sitting down without standing up), it is marked as an action logic error and enters the review queue.

[0108] The logic for checking the temporal existence of objects is as follows: analyze whether the timing of the objects' appearance in the video is reasonable. If an object appears in the middle of the video without any prior introduction process, it is marked as an abrupt appearance; if an object disappears in the middle of the video without leaving the frame or being obscured, it is marked as an abrupt disappearance.

[0109] Uncertainty score calculation: The core of the active closed-loop learning mechanism lies in calculating the uncertainty score of all labeled samples and systematically identifying high-value samples that require manual review.

[0110] The calculation of the uncertainty score takes into account the following factors: The calculation logic for the label boundary distance is as follows: For each predicted label, calculate the distance between its confidence score and the decision boundary. If the confidence score is close to 0.5 (i.e., the decision boundary for binary classification), the uncertainty is high; if the confidence score is close to 0 or 1, the uncertainty is low. The distance calculation formula is: d = min(|p - 0.5| * 2, 1), where p is the confidence score. If d is less than the thirty-first preset threshold, it is marked as high uncertainty.

[0111] The logic for calculating the multi-label competition level is as follows: For multi-label prediction, the degree of competition between each label is analyzed. If there are multiple labels with similar confidence levels (the difference is less than the thirty-second preset threshold), then there is fierce competition between the labels, and the uncertainty is high; if there is a clear preferred label (the difference between it and the second-choice label is greater than the thirty-second preset threshold), then the uncertainty is low.

[0112] The logic for calculating annotation consistency is as follows: it combines the confidence evaluation results of the three granularities (frame level, shot level, and video level). If the confidence of all three granularities is low, the overall uncertainty is high; if the confidence of at least two granularities is high, the overall uncertainty is low.

[0113] The calculation logic for the overall uncertainty is as follows: the three factors mentioned above are weighted and summed to obtain the overall uncertainty score. The judgment logic is as follows: if the overall uncertainty score is higher than the thirty-third preset threshold, the sample enters the priority queue for manual review; if the overall uncertainty score is between the thirty-third and thirty-fourth preset thresholds, the sample enters the general queue; if the overall uncertainty score is lower than the thirty-fourth preset threshold, the sample does not enter the manual review queue.

[0114] Construction of the sampling priority queue: The feedback learning module constructs a sampling priority queue based on uncertainty scores.

[0115] The priority allocation logic is as follows: if the overall uncertainty score of a sample is higher than the thirty-third preset threshold, it is given the highest priority and enters the first position in the manual review queue; if the overall uncertainty score is between the thirty-third and thirty-fifth preset thresholds, it is given a medium priority; if the overall uncertainty score is between the thirty-fifth and thirty-fourth preset thresholds, it is given a lower priority; if the overall uncertainty score is lower than the thirty-fourth preset threshold, it does not enter the manual review queue.

[0116] The system also performs dynamic priority adjustment. The judgment logic is as follows: Calculate the proportion of incorrectly labeled samples in the recent manually reviewed samples. If the error proportion is higher than the thirty-sixth preset threshold, the priority of samples of the same type is increased; if the error proportion is low, the priority adjustment coefficient is decreased.

[0117] Incremental training of the model: During manual review, reviewers not only correct erroneous annotations but also subjectively evaluate the degree of certainty in the annotations. The feedback learning module collects these subjective evaluations to train the uncertainty estimation model, making subsequent uncertainty estimations more accurate.

[0118] The trigger logic for incremental model training is as follows: The system accumulates a number of review samples. If the proportion of correctly corrected samples is higher than the 37th preset threshold and the number of review samples reaches the 38th preset threshold, the complete model fine-tuning process is executed. If the proportion of correctly corrected samples is high but the number of review samples is less than the 38th preset threshold, reaching only the 39th preset threshold, fine-tuning of some network layers is executed. If the proportion of correctly corrected samples is lower than the 37th preset threshold, the cause of the error is analyzed: if it is due to consistency issues caused by unclear annotation specifications, the annotation specifications are updated before fine-tuning is executed; if it is due to model structure mismatch issues, the model structure is adjusted before fine-tuning is executed; if it is due to performance degradation caused by changes in data distribution, domain adaptation training is performed before fine-tuning is executed.

[0119] In some implementations, the system also includes a dynamic threshold adjustment module, which is used to dynamically adjust the threshold parameters of each stage according to the actual situation of the annotation task.

[0120] The working principle of the dynamic threshold adjustment module is as follows: Threshold adjustment based on annotation distribution: The system periodically analyzes the confidence distribution of labeled samples. The judgment logic is as follows: if the proportion of high-confidence samples (confidence greater than 0.8) is higher than 40%, it indicates that the annotation task is relatively simple, and the thresholds at each stage can be appropriately increased to reduce the workload of manual review; if the proportion of high-confidence samples is between 20% and 40%, the current threshold remains unchanged; if the proportion of high-confidence samples is lower than 20%, it indicates that the annotation task is relatively difficult, and the threshold needs to be lowered to increase the coverage of manual review.

[0121] Error rate-based threshold adjustment: The system tracks erroneous annotations discovered during manual review. The judgment logic is as follows: if the error rate of the reviewed samples is higher than the 40th preset threshold, the thresholds at each stage are lowered to strengthen quality control; if the error rate is between the 40th and 41st preset thresholds, the current thresholds remain unchanged; if the error rate is lower than the 41st preset threshold, the thresholds can be appropriately increased to improve annotation efficiency.

[0122] Threshold adjustment based on task type: The system identifies the type characteristics of the current annotation task. The judgment logic is as follows: if the task type is a simple classification task (such as scene classification), the threshold is appropriately increased; if the task type is a complex multi-label task (such as action recognition, attribute annotation), the threshold is decreased; if the task type involves fine-grained classification, the threshold is significantly decreased.

[0123] In some implementations, the system also includes an error mode analysis module for analyzing common error modes during the annotation process to guide continuous optimization of the system.

[0124] The error mode analysis module works as follows: Error type classification: The system automatically classifies errors found during manual review.

[0125] The logic for semantic error classification is as follows: if the error manifests as a category prediction error (such as mislabeling a river as a road), it is classified as a semantic error. Further analysis of the causes of semantic errors reveals that if the error stems from the similarity between categories (such as the visual similarity between rivers and roads), it is recorded as a similarity-class confusion error; if the error stems from a misunderstanding of the annotation specifications, it is recorded as a specification misunderstanding error.

[0126] The logic for classifying timing errors is as follows: if the error manifests as a timing labeling error (such as inaccurate start and end times of an action), it is classified as a timing error. Further analysis of the causes of timing errors reveals that if the error stems from ambiguous boundary determination, it is recorded as a boundary ambiguity error; if the error stems from insufficient understanding of the timing context, it is recorded as a context error.

[0127] The logic for classifying attribute errors is as follows: if the error manifests as an incorrect attribute value (such as mislabeling of color attributes), it is classified as an attribute error. Further analysis of the causes of attribute errors follows: if the error stems from objective difficulties in attribute extraction (such as the influence of lighting on color recognition), it is recorded as an extraction difficulty error; if the error stems from subjective differences in attribute definition, it is recorded as a definition disagreement error.

[0128] Error pattern mining: The system mines common error patterns from error samples.

[0129] The logic for common error analysis is as follows: Collect common features from error samples. If multiple error samples share the same scene type (e.g., errors are concentrated in indoor scenes), then error patterns for that scene type are mined. If error samples share the same action type or object type, pattern mining is also performed.

[0130] The logic for contextual error analysis is as follows: analyze whether the error is related to a specific contextual condition. For example, if the annotation error rate is significantly higher in backlit scenes than in other scenes, then backlighting is a high-risk contextual condition for errors. Under this condition, the system will automatically lower the confidence threshold or increase the probability of manual review.

[0131] Adaptive error prevention: Based on the error pattern analysis results, the system implements an adaptive error prevention strategy.

[0132] The strategy to prevent similarity-based confusion errors is to establish similar category groups in the category library. For categories belonging to the same similarity group, the decision threshold is lowered and the probability of manual review is increased. For example, if rivers and roads are identified as similar categories, the system automatically triggers a more stringent review process when the prediction result is one of these two categories.

[0133] The prevention strategy for high-frequency contextual errors is to establish a mapping table between contextual conditions and error rates. When a contextual condition with a high error rate is encountered, the confidence threshold for that condition is automatically adjusted. For example, if the error rate under backlighting conditions is higher than 30%, the confidence threshold is increased by 0.1 when backlighting conditions are detected.

[0134] The strategy to prevent errors related to difficult attribute extraction is to identify the feature conditions that make attribute extraction difficult. When these feature conditions are encountered, the confidence level of the attribute is discounted or marked in the annotation results as requiring manual verification.

[0135] The complete video content annotation process provided by this invention is as follows: Step 1: The system receives the video data stream to be labeled. The system first performs video format verification and preprocessing.

[0136] Step two involves the feature extraction module preprocessing the video, including format conversion, shot segmentation, and keyframe extraction. Various quality checks are performed during the preprocessing process.

[0137] Step 3: The feature extraction module extracts visual, audio, and text features. Feature quality assessment is performed concurrently during the feature extraction process.

[0138] Step four: The cross-modal adversarial inference module aligns the multimodal features and handles cases of missing modalities. The adaptive modality weighting module calculates the fusion weights for each modality.

[0139] Step 5: The semantic graph construction module builds a hierarchical semantic graph based on the processed features. During the construction process, various judgment logics are executed to establish scene nodes, action nodes, object attribute nodes, and semantic edges between nodes at each level.

[0140] Step six: The graph neural network inference module performs semantic information transfer and feature aggregation on the hierarchical semantic graph. A multi-level attention mechanism calculates intra-level attention, inter-level attention, and temporal attention to update node features.

[0141] Step 7: The annotation output module outputs annotation results based on the updated node features. Label prediction and conflict detection are performed on each type of node.

[0142] Step 8: The quality assessment module performs multi-granularity quality assessments on the annotation results at the frame, shot, and video levels.

[0143] Step nine: Determine if the quality assessment result is lower than the preset threshold. The dynamic threshold adjustment module dynamically adjusts the threshold based on the current annotation distribution and error rate. The judgment logic is as follows: if the overall confidence score is lower than the current dynamic threshold, proceed to step ten; if it is not lower than the current dynamic threshold, proceed to step eleven.

[0144] Step 10: The feedback learning module pushes low-confidence samples to the manual review queue. The priority queue is sorted according to uncertainty scores, and high-value samples are processed first.

[0145] Step 11: Determine if manual review feedback has been received. If feedback is received, proceed to Step 12; otherwise, complete the annotation process and output the final annotation results.

[0146] Step 12: The feedback learning module uses the reviewed labeled data to perform incremental model training. The error pattern analysis module analyzes error patterns and updates the adaptive error prevention strategy.

[0147] Step thirteen: Return to step one and receive the next video segment to be annotated.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A video content annotation method based on big data, characterized in that, Includes the following steps: Obtain multimodal feature data of the video to be labeled; the multimodal feature data includes visual features, audio features and text features of the video; An initial semantic representation of the video is constructed based on the multimodal feature data; the initial semantic representation includes scene nodes, object nodes, action nodes, and attribute nodes; Construct a hierarchical semantic graph; the hierarchical semantic graph includes a top-level scene layer, an intermediate action layer, and a bottom-level object attribute layer; nodes in each layer are connected by semantic edges to form hierarchical semantic relationships; Using graph neural networks, semantic information is transmitted and features are aggregated on the hierarchical semantic graph to generate updated feature representations for each node; The graph neural network includes intra-layer attention mechanism, inter-layer attention mechanism, and temporal attention mechanism; Label prediction is performed based on the updated node feature representation, and the annotation results of the video content are output. The annotation results are subjected to multi-granularity quality assessment, which includes frame-level assessment, shot-level assessment, and video-level assessment. When the quality assessment result is lower than a preset threshold, a manual review process is triggered, and the review result is fed back to the model for incremental training.

2. The video content annotation method according to claim 1, characterized in that: The step of constructing the initial semantic representation of the video based on the multimodal feature data includes: The visual features are classified and detected for scene classification. If the detection confidence is higher than the first preset threshold, a corresponding node is established in the scene layer. If the detection confidence is lower than the first preset threshold but higher than the second preset threshold, it is marked as a scene to be confirmed and enters the manual review queue. If the detection confidence is lower than the second preset threshold, it is determined to be an unknown scene type. Based on the motion information in the visual features, the action category is detected. If the action detection confidence is higher than the third preset threshold, the action node is confirmed to be established. If the action detection confidence is between the third preset threshold and the fourth preset threshold, the audio features are used for auxiliary judgment. The object node is established based on the target detection results in the visual features, and the attribute node is established by extracting object attribute features.

3. The video content annotation method according to claim 1, characterized in that: The rules for establishing semantic edges between nodes in the hierarchical semantic graph include: If the scene node and the action node have a co-occurrence relationship in the spatiotemporal dimension, then a scene-action edge is established; If the action node and the object node have a subject-verb-object relationship, then an action-object edge is established; If the object node and the attribute node have a subordinate relationship, then an object-attribute edge is established.

4. The video content annotation method according to claim 1, characterized in that: The steps for semantic information transmission and feature aggregation using graph neural networks include: Calculate the intra-level attention weights between adjacent nodes in the same layer; if the feature similarity between two nodes is higher than the fifth preset threshold, assign a higher intra-level attention weight; if the feature similarity is lower than the fifth preset threshold but higher than the sixth preset threshold, adjust the weights according to the semantic edge type. Calculate the inter-layer attention weights between nodes at different layers to establish cross-layer information transmission channels; Calculate the temporal attention weights of nodes in different temporal units and perform temporal aggregation of node features; The features of neighboring nodes are aggregated based on attention weights, and the feature representation of the current node is updated.

5. The video content annotation method according to claim 1, characterized in that: The step of predicting labels based on the updated node feature representation includes: For the scene node, if the cosine similarity between its feature vector and the scene category library is higher than the seventh preset threshold, the corresponding category label is output; if the similarity is lower than the eighth preset threshold, it is marked as an unknown category and manual review is triggered. For the action node, if the similarity between the action feature and the best match in the action category library is higher than the ninth preset threshold, then the action label is output. For the object attribute node, if the category of the object node has been determined and the attributes have been completely extracted, then the object label and its attribute label are output.

6. The video content annotation method according to claim 1, characterized in that, It also includes cross-modal adversarial reasoning steps: After acquiring the multimodal feature data, a modality discriminator is constructed to distinguish feature representations from different modalities; Adversarial training enables the feature representations of each modality to achieve distribution alignment in a shared semantic space; When a certain modality is missing, a cross-modal reasoning network is used to infer the semantic representation of the missing modality based on the complete modality.

7. The video content annotation method according to claim 6, characterized in that, It also includes an adaptive modal weighting step: The quality of each modal feature is evaluated; the quality evaluation includes visual quality evaluation of the visual features, audio quality evaluation of the audio features, and text quality evaluation of the text features. The fusion weights of each modality are dynamically adjusted based on the quality assessment results.

8. The video content annotation method according to claim 1, characterized in that: The steps for performing multi-granularity quality assessment on the annotation results include: At the frame level, semantic conflict detection and temporal consistency checks are performed on the annotation results of each frame; if there is a semantic conflict between multiple labels in the same frame, the confidence score of that frame is reduced. At the shot level, the annotation results of each shot are subjected to keyframe annotation consistency rate calculation and object trajectory consistency check; if the annotation consistency rate of keyframes within a shot is higher than the tenth preset threshold, then the shot has a high confidence level. At the video level, the annotation results of the entire video are checked for logical transitions and scene change frequency. If the temporal logic of scene changes is abnormal, the confidence of the entire video is reduced.

9. The video content annotation method according to any one of claims 1 to 8, characterized in that, It also includes the active closed-loop learning step: Calculate the overall uncertainty score for all labeled samples; the uncertainty score takes into account label boundary distance, multi-label competition degree, and label consistency. A sampling priority queue is constructed based on the comprehensive uncertainty score, and samples with high uncertainty are given priority for manual review. Incremental training of the model was performed using the verified labeled data.

10. The video content annotation method according to claim 9, characterized in that: The triggering conditions for incremental training of the model include: The proportion of correctly corrected entries in the reviewed samples was statistically analyzed. If the correct correction ratio is higher than the eleventh preset threshold and the number of review samples reaches the twelfth preset threshold, then the complete model fine-tuning process is executed. If the proportion of correct corrections is lower than the eleventh preset threshold, the cause of the error will be analyzed and the model will be adjusted accordingly before fine-tuning is performed.