A video multi-modal marking method, device, storage medium and program product

Abstract

The present application relates to a kind of video multimodal marking method, equipment, storage medium and program product, by establishing hierarchical label atlas and constructing label-evidence consistency graph to improve video marking accuracy.The method comprises: establishing hierarchical label atlas comprising primary label, secondary label and evidence slot position template;Obtain the video to be marked and auxiliary information and carry out segmentation processing;Utilize multimodal model to obtain primary label candidate set;Obtain secondary label candidate and corresponding evidence information by heterogeneous model;Evidences are mapped to slot position template to construct consistency graph and calculate secondary label candidate consistency score;Finally, determine target label result based on score and label interrelation.System uses graph structure modeling method, comprehensively considers the time, space, semantic association of evidence and label interrelation, solves the problem of misjudgment of adjacent label in vertical field, and improves the accuracy of video content understanding.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video multimodal marking method, device, storage medium and program product. Background Technology

[0002] With the massive growth of short videos and e-commerce videos, automatic understanding and tagging of video content has become a crucial foundational technology for content retrieval, recommendation distribution, and compliance review. In some related technologies, video tagging is typically achieved through multimodal joint analysis of video content and script text. This involves directly using the feature fusion results from image and text models for tag classification, or extracting candidate tag sets through cross-modal retrieval and then combining them with a language model to output the final tags. Some solutions also incorporate knowledge graphs or multi-candidate arbitration mechanisms to help improve the accuracy of tag determination.

[0003] However, in vertical industry applications such as beauty and e-commerce, tagging systems typically have a strict hierarchy and strong proximity. Different fine-grained secondary tags (such as "acne scars" and "blemishes") often have high overlap and intersection in visual presentation, semantic expression, and contextual association. When processing such complex video content, most related technologies perform simple result weighting, voting, or probability filtering at the final model output level. Because the spatiotemporal and semantic relationships between multimodal evidence are severed, the model struggles to accurately distinguish subtle differences in evidence corresponding to highly similar neighboring tags. This leads to issues such as mislabeling of neighboring tags, cross-level tag confusion, and conflicting output tags in fine-grained scenarios within vertical industries, reducing the accuracy of video tagging. Summary of the Invention

[0004] This application provides a video multimodal marking method, device, storage medium, and program product to improve the accuracy of video marking.

[0005] Firstly, this application provides a video multimodal labeling method, which includes: establishing a hierarchical label graph, the hierarchical label graph including first-level labels, second-level labels, inter-label relationships, and evidence slot templates corresponding to each second-level label, the evidence slot templates being used to describe the judgment criteria for the corresponding second-level labels; acquiring the video to be labeled and associated auxiliary information, and performing segmentation and temporal association processing on the video to be labeled and the auxiliary information to obtain a sequence of interconnected video segments and a sequence of auxiliary information segments; inputting the video to be labeled and the auxiliary information into a first multimodal model to obtain a set of first-level label candidates, and determining at least one candidate label branch based on the set of first-level label candidates; inputting the video segment sequence and auxiliary information segment sequence corresponding to each candidate label branch into at least two heterogeneous models respectively to obtain multiple second-level label candidates and evidence information corresponding to the second-level label candidates; mapping the evidence information to the evidence slot templates, and constructing a label-evidence consistency graph by combining the spatiotemporal association, semantic association, and inter-label relationships between the evidence information, and calculating the consistency score of each second-level label candidate; and determining the target label result within the candidate label branch based on the consistency score of each second-level label candidate and the inter-label relationships in the hierarchical label graph.

[0006] In the above embodiments, the system establishes a hierarchical label graph and evidence slot templates as a structured basis for label determination. In the secondary label reasoning stage, a label-evidence consistency graph mechanism is introduced. This maps evidence information output by multiple heterogeneous models to predefined evidence slots, and then performs comprehensive modeling by combining the spatiotemporal and semantic relationships between evidence with the predefined relationships between labels in the label graph. This establishes a structured network of connections between previously fragmented multimodal evidence fragments in the temporal, spatial, and semantic dimensions. This allows the system to perform fine-grained differentiation based on the mutual support between evidence and the mutual exclusion constraints between labels when faced with highly similar neighboring labels, thereby reducing the probability of misjudging neighboring labels and improving the accuracy of video labeling in fine-grained scenarios in vertical industries.

[0007] In conjunction with some embodiments of the first aspect, in some embodiments, a label-evidence consistency graph is constructed by combining the spatiotemporal correlation, semantic correlation, and label-related relationships between evidence information, and the consistency score of each secondary label candidate is calculated. Specifically, this includes: treating each secondary label candidate as a label layer node, and structuring each piece of evidence information mapped to the evidence slot template as an evidence layer node; establishing internal consistency edges between label layer nodes and corresponding evidence layer nodes, the weight of which is determined based on the weighted sum of the temporal intersection-union ratio, spatial intersection-union ratio, and semantic similarity of the corresponding evidence information; establishing conflict edges between label layer nodes with mutually exclusive relationships based on the label-related relationships, and establishing competition edges between different label layer nodes whose spatiotemporal overlap of evidence information is greater than a preset overlap threshold; for each secondary label candidate, subtracting the penalty weights of the connected conflict edges and competition edges from the sum of the weights of all corresponding subordinate internal consistency edges to obtain the initial consistency score of the secondary label candidate.

[0008] In the above embodiments, the system constructs a two-layer graph structure for candidate labels and evidence information. Internal consistency edges are established between label layer nodes and evidence layer nodes to quantify the strength of evidence' support for the labels. Simultaneously, conflict edges are established based on the mutual exclusion relationships between labels, and competing edges are established between different labels for evidence with spatiotemporal overlap. This graph-structured modeling approach transforms the temporal intersection-union ratio, spatial intersection-union ratio, and semantic similarity of evidence into computable edge weights. By introducing penalty mechanisms for conflict and competing edges, the system can simultaneously consider the supporting role of evidence and the constraint relationships between labels during the consistency score calculation process, improving the consistency of label results within the hierarchical system.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, the evidence slot template is configured with modal importance weights for each type of evidence. After obtaining the consistency score of the secondary label candidate, the method further includes: calculating the slot coverage of the corresponding secondary label candidate based on the modal importance weights corresponding to the evidence slots filled with evidence information under each secondary label candidate, wherein the slot coverage is the ratio of the sum of the modal importance weights of the filled slots to the sum of the modal importance weights of all slots; determining whether the slot coverage is lower than a preset minimum coverage threshold; if so, applying a confidence decay penalty to the initial consistency score of the secondary label candidate and updating the decayed score to the consistency score of the secondary label candidate; if not, keeping the consistency score of the secondary label candidate unchanged.

[0010] In the above embodiments, the system calculates the coverage of the evidence slots for each secondary label candidate. The coverage index integrates the modal importance weight ratio of the filled slots and determines whether to apply a confidence decay penalty to the consistency score by comparing it with a preset minimum coverage threshold. Evidence completeness is used as an adjustment factor for label confidence. This means that even if the evidence in other dimensions shows high consistency, the system will still reduce the trust level of the label candidate when key modal evidence is missing. This reduces the risk of mislabeling caused by incomplete evidence structure and increases the dependence of label determination on the completeness of multimodal evidence.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, before determining the target label result within a candidate label branch based on the consistency scores of each secondary label candidate and the relationships between labels in the hierarchical label graph, the method further includes: when the consistency score of a secondary label candidate is lower than a preset threshold, performing preliminary screening in a historical labeled sample library based on the primary label candidate set of the video to be labeled to obtain a candidate sample pool; extracting the multimodal feature vector of the video to be labeled, performing vector similarity retrieval using the multimodal feature vector in the candidate sample pool, and determining a preset number of historical samples with the highest similarity as a preliminary recall set; calculating the graph structure similarity between the label-evidence consistency graph of each historical sample in the preliminary recall set and the label-evidence consistency graph of the video to be labeled; selecting historical samples with graph structure similarity higher than a preset fine screening threshold, and combining them to form a similar sample set within a branch; calibrating the consistency score of each secondary label candidate based on the similar sample set within a branch.

[0012] In the above embodiments, when a low-confidence secondary label candidate is detected, the system performs a coarse screening in the historical sample library using the primary label candidate set, then retrieves the recalled candidate samples using multimodal feature vector similarity, and further calculates the graph structure similarity of the label-evidence consistency graph between the video to be labeled and the historical samples for fine screening. Finally, a set of similar samples within the branch is formed for calibration, transforming historical labeling experience into a reference benchmark for the current judgment. By using graph structure similarity, a high-order feature rather than simple label co-occurrence frequency, for sample screening, the recalled historical samples have higher comparability with the video to be labeled in terms of evidence organization patterns, thus improving the reliability of the calibration process for determining marginal cases.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, the consistency scores of each secondary label candidate are calibrated based on the similar sample set within the branch. Specifically, this includes: statistically analyzing the consistency score distribution characteristics of successful labeled cases for the corresponding secondary label in the similar sample set within the branch, and extracting the target quantile as the target confidence threshold; calculating the statistical median of the weights of each evidence slot in the similar sample set within the branch, and comparing it with the initial weights of the evidence slot template to calculate the calibration offset; using the calibration offset to compensate for the consistency scores of each current secondary label candidate to obtain the candidate calibration score; when the difference between the candidate calibration score and the consistency score before calibration is greater than a preset score change threshold, the consistency score of the corresponding secondary label candidate is updated using the candidate calibration score; the updated consistency score is compared with the target confidence threshold, and if the updated consistency score is lower than the target confidence threshold, the corresponding secondary label candidate is removed.

[0014] In the above embodiments, the system statistically analyzes the consistency score distribution of successful cases corresponding to the same label based on the set of similar samples within the branch, extracts the target quantile as the confidence threshold, and generates a calibration offset by calculating the deviation between the statistical median of the evidence slot weights in historical samples and the current template weights. This offset is then used to compensate for the consistency score of the current label candidate. This adaptive calibration transforms the annotation experience accumulated in the sample library into dynamic scoring benchmarks and weight correction parameters. This allows the system to reduce misjudgments caused by initial slot weight configuration deviations or inconsistent scoring scales when processing videos with patterns similar to historical cases, thus improving the stability of label determination in long-tail scenarios and marginal cases.

[0015] In conjunction with some embodiments of the first aspect, in some embodiments, the video to be labeled and auxiliary information are segmented and temporally correlated to obtain a sequence of interconnected video segments and a sequence of auxiliary information segments. Specifically, this includes: performing initial shot segmentation on the video to be labeled based on the rate of change of the scene to obtain a sequence of basic video segments; extracting multimodal feature vectors of adjacent basic video segments in the sequence of basic video segments and calculating the semantic continuity score between adjacent basic video segments; when the semantic continuity score is higher than a preset merging threshold and the auxiliary information segments corresponding to adjacent basic video segments have semantic referential association, merging the adjacent basic video segments on the time axis to generate an updated sequence of video segments; and re-aligning the auxiliary information based on the time intervals corresponding to the updated video segments and sequences, and outputting the sequence of video segments and the sequence of auxiliary information segments.

[0016] In the above embodiments, based on the initial shot segmentation, the system calculates the semantic continuity score between adjacent video segments and combines it with the referential relationships of corresponding auxiliary information segments to determine whether to perform timeline merging, thereby generating a video segment sequence with higher semantic integrity. This dynamic merging elevates physical segmentation to intelligent segmentation that simultaneously considers visual semantic continuity and textual referential consistency, enabling subsequent multimodal analysis to be performed at a segment granularity that better aligns with content logic, thus improving the accuracy of spatiotemporal alignment between video segments and auxiliary information segments.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, the video segment sequence and auxiliary information segment sequence corresponding to each candidate label branch are respectively input into at least two heterogeneous models to obtain multiple secondary label candidates and evidence information corresponding to each secondary label candidate. Specifically, this includes: obtaining the label density of the candidate label branch and the historical confusion degree of each corresponding secondary label in historical data; selecting at least two heterogeneous models from a preset model pool based on the label density and historical confusion degree, wherein when both the label density and historical confusion degree are higher than a preset complexity threshold, a first heterogeneous model for extracting visual features and a second heterogeneous model for text semantic understanding are selected as at least two heterogeneous models; and inputting each video segment sequence and auxiliary information segment sequence into at least two heterogeneous models for joint inference to obtain multiple secondary label candidates and evidence information corresponding to each secondary label candidate.

[0018] In the above embodiments, the system dynamically selects heterogeneous model combinations from the preset model pool based on the label density of the candidate label branches and the confusion index of the corresponding secondary labels in historical data. In particular, in high-density and high-confusion scenarios, the system simultaneously enables the visual feature extraction model and the text semantic understanding model to perform joint reasoning. This matches the allocation of computing resources with the difficulty of distinguishing the current label space, avoiding the waste of resources caused by excessive reasoning on simple branches. At the same time, on complex branches, the system obtains richer evidence dimensions through the complementarity of multiple models, thereby improving the system's ability to distinguish highly confused label sets while maintaining reasoning efficiency.

[0019] In a second aspect, embodiments of this application provide a video multimodal marking device, which includes one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the video multimodal marking device to perform the method described in the first aspect and any possible implementation thereof.

[0020] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a video multimodal marking device, cause the video multimodal marking device to perform the method described in the first aspect and any possible implementation thereof.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a video multimodal marking device, cause the video multimodal marking device to perform the method described in the first aspect and any possible implementation thereof.

[0022] Understandably, the video multimodal marking device provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. This application constructs a label-evidence consistency graph, aggregating multimodal evidence into a spatiotemporal and semantic association network. By combining support and mutual exclusion constraints for fine-grained classification, it reduces the misclassification rate of nearest neighbor labels and improves the accuracy of fine-grained video labeling in vertical categories.

[0025] 2. This application implements confidence decay penalty by calculating the coverage of evidence slots, incorporates the completeness of multimodal evidence into the confidence adjustment system to reduce the risk of blind mislabeling caused by the absence of some modalities, and enhances the consistency of multi-level output labels.

[0026] 3. This application performs adaptive score calibration of low-confidence labels by comparing the graph structure similarity of historical samples, transforming structured historical experience into a dynamic benchmark, thereby enhancing the stability of edge case labeling output with highly overlapping features. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a video multimodal labeling method in an embodiment of this application;

[0028] Figure 2 This is another flowchart illustrating the video multimodal labeling method in this application embodiment;

[0029] Figure 3 This is a schematic diagram of the physical device structure of a video multimodal marking device in the embodiments of this application. Detailed Implementation

[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0032] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a video multimodal labeling method in an embodiment of this application.

[0033] S101. Establish a hierarchical label map. The hierarchical label map includes first-level labels, second-level labels, relationships between labels, and evidence slot templates corresponding to each second-level label. The evidence slot templates are used to describe the judgment criteria for the corresponding second-level labels.

[0034] The hierarchical tag map refers to a multi-level knowledge structure used to organize video content tags, including first-level tags (such as coarse-grained categories like "skincare" and "makeup"), second-level tags (such as fine-grained categories like "acne scar repair" and "blemish fading"), and relationships between tags (mutual exclusion, inclusion, co-occurrence, etc.). Evidence slot templates represent the structured evidence framework required for determining specific second-level tags. Each slot corresponds to a type of evidence (such as visual feature slots, text description slots, and temporal behavior slots), and a modal importance weight is configured for that slot. For example, the template for the tag "acne scar repair" could include "facial red marks - visual slot (weight 0.4)," "mention of repair products - text slot (weight 0.3)," and "application action - behavior slot (weight 0.3)." Relationships between tags represent semantic constraints. Mutual exclusion means that two tags should not appear simultaneously (such as "dry skin" and "oily skin"). Inclusion refers to the subordinate logic of hierarchical tags, and co-occurrence represents the statistical probability of tag combinations in historical data.

[0035] This step is triggered during system initialization or when the tagging system is updated. Specifically, the system generates the atlas by combining manual rule configuration with historical sample statistics: First, domain experts define the tag hierarchy and basic mutual exclusion relationships based on business knowledge, organizing them into a tree structure or storing them in a graph database; then, the system traverses the historical labeled sample library, statistically analyzes the types of evidence and their spatiotemporal distribution characteristics in successfully labeled cases for each secondary tag, sorts them in descending order of modal contribution to generate evidence slot templates for each tag, and initializes the modal importance weight of the slots to the normalized frequency of occurrence of the corresponding evidence in historical cases. For example, if the tag "acne scar repair" has 400 instances of visual evidence of red facial marks, 300 instances of text mentioning repair products, and 300 instances of application actions in 1000 historical samples, then the weights of the three types of slots are initialized to 0.4, 0.3, and 0.3, respectively. The system stores the tag graph in the form of an adjacency list, which supports quick querying of the set of second-level tags under the first-level tags and their mutual exclusion constraints. At the same time, it associates each second-level tag with its own unique evidence slot template JSON structure, which facilitates the structured filling of subsequent evidence mapping.

[0036] S102. Obtain the video to be labeled and the associated auxiliary information, and perform segmentation and time association processing on the video to be labeled and the auxiliary information to obtain a sequence of interrelated video segments and a sequence of auxiliary information segments.

[0037] Among them, auxiliary information refers to multimodal supplementary data associated with video content on the timeline, including speech text transcribed by ASR (Automatic Speech Recognition), on-screen text extracted by OCR (Optical Character Recognition), subtitle text, product titles, and user comments, etc. Each piece of auxiliary information carries a start and end timestamp to identify its effective interval on the video timeline; the video segment sequence represents an ordered set of segments after the original video is segmented according to semantic integrity, and each segment contains start and end frame numbers, duration, and corresponding visual content; the auxiliary information segment sequence is used to represent the subset of auxiliary information associated with each video segment after time alignment processing, maintaining a one-to-one correspondence with the video segment sequence.

[0038] Specifically, the system uses a video application shot boundary detection algorithm (such as histogram difference-based image change rate calculation) to initially segment the video at the peak of the image change rate to obtain a basic segment sequence. For two adjacent basic segments, the system extracts the visual feature vectors of the last and first three frames (using a pre-trained image encoder), calculates the cosine similarity as a semantic continuity score, and checks whether the auxiliary information corresponding to the two segments contains pronoun references (such as "it" or "this") or topic continuation (determined by sentence vector similarity > 0.7). When the semantic continuity score is higher than the preset merging threshold of 0.8 and the auxiliary information has referential associations, the two segments are merged into a larger semantic unit on the timeline to avoid artificially splitting content with the same expressive intent.

[0039] The merging operation is iterated until no adjacent segments meet the conditions. Finally, the system redistributes the auxiliary information according to the timestamp based on the updated video segment time interval: if the time interval of a certain ASR text intersects with that of a certain video segment, then the text is assigned to the auxiliary information set of the corresponding segment.

[0040] In some preferred embodiments, the above steps specifically include:

[0041] The initial shot segmentation of the video to be labeled is performed based on the rate of change of the image to obtain a basic video segment sequence;

[0042] Extract multimodal feature vectors from adjacent basic video segments in the basic video segment sequence, and calculate the semantic continuity score between adjacent basic video segments;

[0043] When the semantic continuity score is higher than the preset merging threshold, and the auxiliary information segments corresponding to adjacent basic video segments have semantic referential relationships, the adjacent basic video segments are merged on the time axis to generate an updated video segment sequence.

[0044] Based on the updated video clips and their corresponding time intervals, the auxiliary information is realigned, and the video clip sequence and the auxiliary information clip sequence are output.

[0045] Among them, the basic video segment sequence refers to the preliminary segment set obtained by segmenting the video at the peak of the difference in gray-level histogram between consecutive frames through the image change rate detection algorithm. Each basic segment corresponds to a relatively independent shot unit. The semantic continuity score is used to represent the degree of connection between adjacent basic segments at the visual semantic level. It is obtained by extracting the visual feature vectors of the last and first 3 frames of the segment (using ResNet-50 pre-trained on ImageNet to extract 2048-dimensional features) and calculating the cosine similarity. The value range is [0, 1]. The larger the value, the stronger the semantic continuity. The referential association refers to the pronoun referential relationship (such as "it" or "this one" pointing to the previous entity) or topic continuation relationship (determined by sentence vector similarity > 0.7) between adjacent auxiliary information segments at the text level, indicating that the two texts constitute a coherent expression in semantic logic. The preset merging threshold is generally set to 0.8, which comprehensively considers the semantic integrity requirements of shot switching in short video scenarios and the balance of computational efficiency.

[0046] Specifically, the system iterates through the basic segment sequence in chronological order. For each pair of adjacent segments A and B, it extracts the last 3 frames of A and the first 3 frames of B and encodes them as feature vectors. It then calculates the cosine similarity between each pair of these 6 vectors and takes the maximum value as the semantic continuity score. For example, in a skincare tutorial, segment A shows the product's appearance and ends at 10 seconds, while segment B starts showing the application action from 10.2 seconds. Although there is a camera transition, the visual subjects (product and hand) are continuous, and the calculated similarity is 0.83 > 0.8.

[0047] The system further examines the corresponding auxiliary information: the ASR text of segment A is "This serum has a light texture," and segment B is "It absorbs very quickly." The system detects the pronoun "it" referring to the preceding product, indicating a referential relationship. After satisfying both conditions, A and B are merged into a single segment [0s-15s] on the timeline, avoiding the artificial separation of "product introduction + usage demonstration" into two independent units during subsequent multimodal analysis, which share the same expressive intent.

[0048] S103. Input the video to be labeled and auxiliary information into the first multimodal model to obtain a first-level label candidate set, and determine at least one candidate label branch based on the first-level label candidate set.

[0049] The first multimodal model refers to a pre-trained visual-language fusion model (such as CLIP, BLIP, etc.) used for coarse-grained classification, which can simultaneously process video frame sequences and text input and output label probability distributions; the first-level label candidate set represents the ordered set of first-level labels whose output probabilities exceed a preset threshold (such as 0.3) and their confidence scores; the candidate label branch is used to represent the second-level label search space that expands downward along the hierarchical label graph starting from a certain first-level label, and one candidate label branch corresponds to a subtree structure in the graph.

[0050] This step is triggered after video segmentation and auxiliary information alignment are completed, quickly converging the search space. Specifically, the keyframe sequence of the video to be labeled (sampled at fixed intervals or representative frames of each segment are extracted) and all auxiliary information text are concatenated and input into the first multimodal model. This model outputs the predicted probability vectors of all first-level labels. The system retains labels with a probability value greater than 0.3 to form a candidate set of first-level labels and sorts them in descending order of probability. For each first-level label in the set, the system queries all its subordinate second-level label nodes in the hierarchical label graph. Each such subtree constitutes a candidate label branch.

[0051] For example, if the primary tag candidate set is {"Skincare": 0.8, "Makeup": 0.5}, the system generates two candidate tag branches: Branch A contains secondary tags under "Skincare" such as "Acne Scar Repair", "Spot Fading", and "Pore Minimizing", while Branch B contains secondary tags under "Makeup" such as "Eye Makeup Tutorials" and "Lip Makeup Recommendations". When the probability of all primary tags is below 0.3, the system uses a safety net strategy to select the Top-1 tags to generate branches and marks them as low-confidence for manual review. This two-stage design utilizes the coarse-grained discrimination capability of primary tags to quickly narrow the search scope, limiting the global secondary tag space (which may contain hundreds of tags) to a few candidate branches (each branch typically contains 5-20 secondary tags), reducing the computational complexity of subsequent fine-grained inference, while retaining the ability to process multiple branches in parallel to handle situations where the video may involve multiple themes.

[0052] S104. Input the video segment sequence and auxiliary information segment sequence corresponding to each candidate label branch into at least two heterogeneous models to obtain multiple secondary label candidates and the evidence information corresponding to the secondary label candidates.

[0053] Heterogeneous models refer to multiple models that differ in architecture, training data, or feature extraction dimensions. For example, visual feature extraction models based on convolutional neural networks (such as ResNet and EfficientNet) focus on extracting visual features such as image texture, color, and object shape; text semantic understanding models based on Transformers (such as BERT and RoBERTa) focus on extracting semantic relationships and contextual information in text; and behavior recognition models based on temporal convolutional networks focus on extracting action sequence features. Secondary label candidates represent the fine-grained labels and their initial confidence scores output by heterogeneous models within a certain candidate label branch. Evidence information is used to represent the multimodal content fragments that the model locates during the inference process that support the determination of a specific label. Each piece of evidence information includes the evidence type (visual / textual / behavioral), the time interval in the video, the spatial location in the image (bounding box coordinates), the description of the evidence content, and the strength of its association with the label.

[0054] This step is performed separately for each candidate label branch, enabling richer model combinations in high-confusion scenarios. Specifically, the system calculates the label density of the branch (the ratio of the number of secondary labels within the branch to the total number of primary labels) and the historical confusion rate (the normalized mean of the number of misclassifications between label pairs within the branch in the historical sample library). When both are higher than the preset complexity threshold (density > 0.15 and confusion rate > 0.3), the system selects the first heterogeneous model (visual feature extractor) and the second heterogeneous model (text semantic understander) from the preset model pool to form an inference combination; if the branch is simple (e.g., containing only 3 secondary labels and with a low historical misclassification rate), only a lightweight single fusion model is selected to save computational resources.

[0055] During inference, the system inputs the frame sequences of each video segment into the first heterogeneous model to obtain the visual confidence score of each segment on each secondary label and the corresponding keyframe position (located through attention mechanism or gradient backpropagation) as visual evidence. In parallel, auxiliary information segments are input into the second heterogeneous model to obtain the semantic matching score of the text on each secondary label and the keyword segments that trigger the judgment as textual evidence. For example, for the label "acne scar repair," the visual model might output visual evidence such as "a red mark exists on the left side of the face in the frame from seconds 12-18, with a confidence score of 0.7," while the textual model outputs textual evidence such as "the phrase 'fades acne scars' in the ASR text 'This essence can fade acne scars' at second 15, with a semantic matching score of 0.8." The system then aggregates the outputs of all heterogeneous models to generate a candidate option and its evidence list for each secondary label within a branch.

[0056] In some preferred embodiments, the above steps specifically include:

[0057] Obtain the label density of the candidate label branches and the historical confusion of each corresponding secondary label in historical data;

[0058] Based on label density and historical confusion, at least two heterogeneous models are selected from a preset model pool. When both label density and historical confusion are higher than a preset complexity threshold, a first heterogeneous model for extracting visual features and a second heterogeneous model for text semantic understanding are selected as at least two heterogeneous models.

[0059] Each video segment sequence and auxiliary information segment sequence are input into at least two heterogeneous models for joint inference to obtain multiple secondary label candidates and the evidence information corresponding to each secondary label candidate.

[0060] Among them, label density represents the proportion of the number of second-level labels within a candidate label branch to the theoretical maximum number of subordinate labels of that first-level label, used to measure the crowding of the label space within a branch; historical confusion refers to the frequency of misclassification between any two second-level labels within a branch in the historical labeled sample library, calculated as the sum of the number of misclassifications of all label pairs within the branch divided by the total number of label pairs and then divided by the total number of historical samples. This indicator quantifies the difficulty of distinguishing labels within a branch; the model pool is used to represent a set of various inference models pre-configured by the system, including lightweight fusion models (for simple scenarios), visual feature extraction models, text semantic understanding models, behavior recognition models, etc., with each model labeled with its preferred modality type and computational resource consumption.

[0061] This step is executed immediately after the candidate label branch is determined, and inference resources are adaptively configured according to task complexity. Specifically, the system first queries the hierarchical label graph to obtain the list of second-level labels within the branch and calculates the label density; it then calculates the confusion matrix of the labels within the branch in the historical sample database and extracts the sum of the off-diagonal elements to calculate the historical confusion degree.

[0062] When the label density is higher than 0.15 (e.g., the "skincare" branch contains 15 secondary labels while the theoretical maximum is 100) and the historical confusion is higher than 0.3 (e.g., the misclassification rate between "acne scar repair" and "blemish fading" in historical samples reaches 30%), the system determines it as a high-complexity scenario. It selects a first heterogeneous model (e.g., ResNet-101 visual extractor, which focuses on capturing fine-grained visual features such as facial texture and blemish morphology) and a second heterogeneous model (e.g., RoBERTa text encoder, which focuses on understanding the subtle semantic differences such as "repair", "fade", and "improve") from the model pool for joint inference; if the label density is <0.1 and the confusion is <0.2, only a single lightweight CLIP model is used to quickly output the results.

[0063] During joint inference, two models process video segments and auxiliary information segments in parallel. The first model outputs the visual confidence vector of each secondary label and the corresponding image region location, while the second model outputs the semantic matching vector and trigger keywords. The system aligns the two sets of outputs by timestamp and merges them into a unified list of secondary label candidates, with each candidate carrying evidence information from different modalities. This dynamic strategy links inference cost to task difficulty: simple branches are quickly passed using lightweight models, while highly confusing branches utilize specialized models to deeply mine multimodal evidence, thereby improving the ability to distinguish easily confused fine-grained labels while ensuring overall throughput efficiency.

[0064] S105. Map the evidence information to the evidence slot template, combine the spatiotemporal relationship, semantic relationship and tag relationship between the evidence information to construct the tag-evidence consistency graph, and calculate the consistency score of each secondary tag candidate.

[0065] The label-evidence consistency graph refers to constructing a two-layer graph structure of secondary label candidates and evidence information. Label layer nodes represent each secondary label candidate, and evidence layer nodes represent structured evidence information. Nodes are connected by different types of edges to model multi-dimensional relationships. Internal consistency edges are used to represent the support relationship between label layer nodes and their subordinate evidence layer nodes. The edge weights are calculated based on the weighted sum of the temporal intersection-union ratio (intersection of the evidence time interval and the time interval of the label-related segments), spatial intersection-union ratio (IoU value of the evidence spatial location and the label-related image area), and semantic similarity (cosine similarity between the evidence content vector and the label description vector). Conflict edges represent negative connections established between label layer nodes based on mutual exclusion constraints in the relationship between labels. Competition edges are used to represent the competition relationship established when evidence from different labels highly overlaps in time and space.

[0066] This step is performed after obtaining the secondary label candidates and evidence information for all candidate label branches. Graph structure modeling is used to quantify the comprehensive support of evidence for the label. Specifically, the system first instantiates each secondary label candidate as a label layer node, and structures each piece of evidence information into evidence layer nodes according to its type, spatiotemporal location, and content description. Then, an internal consistency edge is established between each label node and its associated evidence node. The edge weight is calculated as: w = 0.3 × temporal IoU + 0.3 × spatial IoU + 0.4 × semantic similarity. Temporal IoU is calculated using the evidence timestamp and the timestamp of the label-related segment. Spatial IoU is calculated using the evidence image location box and the label-related region. The label-related region is a reference box obtained by a pre-set general object detector performing prior detection on the current video image based on the semantic concept of the secondary label candidate. Semantic similarity is obtained by calculating the cosine distance after encoding the evidence text description and label name into vectors. The weight coefficients are configured based on business experience.

[0067] For example, between the "acne scar repair" tag node and a visual evidence node of "red mark on the left side of the face for 12-18 seconds," if the temporal IoU = 0.8, spatial IoU = 0.6, and semantic similarity = 0.75, then the edge weight = 0.3 × 0.8 + 0.3 × 0.6 + 0.4 × 0.75 = 0.72. The system further queries the hierarchical tag graph, establishing conflict edges between tag nodes marked as mutually exclusive (such as "dry skin" and "oily skin") and assigning a fixed penalty weight of -0.5. For evidence nodes under different tags, if their spatiotemporal overlap (temporal IoU × spatial IoU) is greater than the preset threshold of 0.6, then a competing edge is established between the corresponding tag nodes and assigned a penalty weight of -0.3, indicating that the two tags are competing for the same evidence. Finally, an initial consistency score is calculated for each secondary tag candidate: Score = Σ (internal consistency edge weight) - Σ (conflict edge weight) - Σ (competitive edge weight).

[0068] This graph-structured modeling establishes a structured network of associations among multimodal evidence fragments in three dimensions: time, space, and semantics. This enables the system to consider not only the supporting strength of a single piece of evidence when evaluating the confidence of a label, but also to perceive the mutual corroboration between evidence and the constraint conflicts between labels. Thus, when faced with visually similar neighboring labels such as "acne scars" and "blemishes," it can accurately distinguish them by comparing subtle differences in the spatiotemporal distribution and semantic description of their evidence, thereby reducing the probability of misjudging neighboring labels.

[0069] In some preferred embodiments, after the above steps map the evidence information to the evidence slot template, the specific steps include: taking each secondary label candidate as a label layer node, and structuring each piece of evidence information mapped to the evidence slot template as an evidence layer node;

[0070] An internal consistency edge is established between the label layer node and the corresponding evidence layer node. The weight of the internal consistency edge is determined based on the weighted sum of the temporal intersection-union ratio, spatial intersection-union ratio and semantic similarity of the corresponding evidence information.

[0071] Based on the relationship between labels, conflict edges are established between label layer nodes that have mutual exclusion relationships, and competition edges are established between different label layer nodes whose spatiotemporal overlap of evidence information is greater than a preset overlap threshold.

[0072] For each secondary label candidate, the initial consistency score of the secondary label candidate is obtained by subtracting the penalty weights of the connected conflicting and competing edges from the sum of the weights of all its corresponding internal consistent edges.

[0073] The label layer nodes represent upper-level nodes in the consistency graph that represent candidates for secondary labels. Each node carries the label name, its branch, and the current cumulative confidence score. The evidence layer nodes represent evidence information nodes after being structured by the evidence slot template. Node attributes include evidence type (visual / textual / behavioral), time interval [start, end], spatial coordinates (x, y, w, h), evidence content description text, and source model identifier. Temporal IoU is the intersection length of two time intervals divided by the union length, used to quantify the degree of overlap of evidence in the time dimension. Spatial IoU is the intersection area of ​​two rectangles divided by the union area, used to quantify the degree of overlap of evidence in the image space. Semantic similarity is obtained by encoding the evidence content text and label name into vectors and then calculating the cosine similarity, with a value range of [0, 1]. The larger the value, the closer the semantics. The preset overlap threshold is used to control the sensitivity of establishing competitive edges. Setting it to 0.6 means that a competitive relationship is considered to exist only when the spatiotemporal overlap of evidence with different labels exceeds 60%.

[0074] This step is performed after mapping the evidence information to the evidence slot template to calculate the overall confidence score for each secondary label candidate. Specifically, the system iterates through each secondary label candidate and creates a label layer node for each candidate; at the same time, it iterates through all evidence information and structures the evidence into evidence layer nodes according to the slot type defined in the evidence slot template (such as "visual-facial texture", "text-efficacy description", "behavior-applying action"). The nodes store the time interval of the evidence (such as [12.5s, 18.3s]), the image position (such as the bounding box [120, 80, 200, 150]), the content description (such as "red mark on the left side of the face"), and the source model (such as "ResNet visual model").

[0075] Subsequently, the system establishes internal consistency edges between the label layer nodes and their associated evidence layer nodes: First, it calculates the temporal IoU. For example, if the video segment corresponding to the label "acne scar repair" has a time interval of [10s, 20s], and a certain visual evidence interval is [12.5s, 18.3s], then the intersection length = 18.3 - 12.5 = 5.8s, the union length = 20 - 10 = 10s, and the temporal IoU = 5.8 / 10 = 0.58; then, it calculates the spatial IoU. The system calls a pre-set general object detector (e.g., based on prior knowledge to identify objects in the image). The baseline bounding box of the label-related region (the "face / skin" area related to "acne scar repair") is obtained. If the evidence location box is [120, 80, 200, 150] and the label-related region is [100, 70, 220, 160], the spatial IoU is calculated as 0.65 by the intersection and union of rectangles. Finally, the evidence description "red mark" and the label name "acne scar repair" are encoded into vectors and the cosine similarity is calculated as 0.72. Substituting into the formula, the edge weight is 0.3×0.58+0.3×0.65+0.4×0.72=0.657.

[0076] The system further queried the hierarchical label graph and found that "acne scar repair" and "blemish fading" were marked as easily confused relationships in the graph (although not completely mutually exclusive, they had a high historical level of confusion). A conflicting edge was established between the two label nodes and a weight of -0.5 was assigned. At the same time, the spatiotemporal overlap of visual evidence under each of the two labels was detected (temporal IoU=0.7, spatial IoU=0.85, product=0.595<0.6, no competing edge was triggered). If the evidence overlap of another pair of labels, "pore shrinkage" and "skin brightening", reached 0.68>0.6, a competing edge was established between the corresponding label nodes and a weight of -0.3 was assigned.

[0077] Finally, the initial consistency score is summed for each label node: Initial consistency score = Sum of internal consistency weights of all incoming edges - Weights of all conflicting edges - Weights of all competing edges. For example, "acne scar repair" has 3 pieces of evidence supporting it, with edge weights of 0.657, 0.543, and 0.721 respectively. There is 1 conflicting edge -0.5, and no competing edges, so the score = 0.657 + 0.543 + 0.721 - 0.5 = 1.421. Internal consistency edges quantify the multi-dimensional support strength of evidence, while conflicting and competing edges introduce mutual exclusion constraints and resource competition mechanisms between labels. This allows the system to consider both evidence sufficiency (positive accumulation) and label rationality (negative penalty) in the consistency score calculation. When two neighboring labels are visually difficult to distinguish, the system can break the balance by comparing the semantic differences or subtle shifts in the spatiotemporal distribution of their textual evidence to achieve precise differentiation of fine-grained labels.

[0078] In other preferred embodiments, the evidence slot template is configured with modal importance weights for each evidence type. After obtaining the consistency scores of the secondary label candidates, the method further includes:

[0079] Based on the modal importance weight of the evidence slot filled with evidence information under each secondary label candidate, the slot coverage of the corresponding secondary label candidate is calculated. The slot coverage is the ratio of the sum of the modal importance weights of the filled slots to the sum of the modal importance weights of all slots.

[0080] Determine whether the slot coverage is lower than the preset minimum coverage threshold;

[0081] If so, apply a confidence decay penalty to the initial consistency score of the secondary label candidate, and update the decayed score to the consistency score of the secondary label candidate.

[0082] If not, the consistency score of the secondary label candidates remains unchanged.

[0083] Modal importance weight refers to the pre-configured weight value for each type of evidence slot in the evidence slot template, reflecting the importance of that type of evidence to the label determination. For example, in the "acne scar repair" label template, the "visual - red mark" slot has a weight of 0.4, the "text - repair efficacy" slot has a weight of 0.3, and the "behavioral - application action" slot has a weight of 0.3, with a sum of 1.0. Slot coverage is used to represent the ratio of the sum of modal importance weights corresponding to the slots filled with evidence information under the current secondary label candidate to the sum of all slot weights, used to quantify the completeness of the evidence structure. The minimum coverage threshold represents the lower limit of slot coverage required by the system. Setting it to 0.5 means that at least 50% of the slots need to be filled to be considered sufficient evidence. Confidence decay penalty is a reduction factor applied to the initial consistency score when the slot coverage is insufficient. It is usually designed as a linear decay function penalty = max(0, (minimum coverage threshold - actual coverage) × attenuation coefficient), the attenuation coefficient is set according to the business's sensitivity to the completeness of evidence (e.g., 1.5 means that 15% of the score is deducted for every 10% below the threshold).

[0084] This step is performed after calculating the initial consistency score of each secondary label candidate to correct the confidence of candidates with incomplete evidence structures. Specifically, the system iterates through each secondary label candidate, queries its corresponding evidence slot template to obtain all defined slots and weights. For example, the "acne scar repair" template contains 3 slots {visual - red mark: 0.4, text - repair efficacy: 0.3, behavior - application action: 0.3}, with a total weight of 1.0. The system checks the evidence layer nodes under the current candidate to determine which slots have been filled: if there is an evidence node of type "visual - red mark", the visual slot is filled; if there is text evidence containing keywords such as "repair" and "fade", the text slot is filled. Assuming that a video only detects visual and text evidence, and behavioral evidence is missing, the sum of the weights of the filled slots = 0.4 + 0.3 = 0.7, and the slot coverage = 0.7 / 1.0 = 0.7.

[0085] The system compares the coverage score with a preset minimum coverage threshold of 0.5. If the coverage score is ≥ 0.5, the evidence is deemed sufficient, and the initial consistency score remains unchanged. If the coverage score is < 0.5 (e.g., if only visual slots are filled, coverage = 0.4), the attenuation penalty is calculated as (0.5 - 0.4) × 1.5 = 0.15. The initial consistency score is then multiplied by (1 - 0.15) = 0.85 to obtain the updated consistency score. For example, if a candidate has an initial score of 1.421 and a coverage score of 0.4, the updated score would be 1.421 × 0.85 = 1.208.

[0086] S106. Based on the consistency scores of each secondary label candidate and the relationships between labels in the hierarchical label graph, determine the target label result within the candidate label branch.

[0087] The target label result refers to the final label set output by the system after comprehensive evaluation. It can be a single secondary label (such as "acne scar repair"), a primary-secondary label path (such as "skincare-acne scar repair"), or a combination of multiple non-mutually exclusive labels (such as {"skincare-acne scar repair", "skincare-pore shrinking"}). Each label carries a final confidence score, a summary of key evidence, and its temporal location in the video. The relationship between labels is used in this step to perform mutual exclusion constraint filtering and co-occurrence rationality verification to ensure that the output label set is semantically and logically consistent.

[0088] This step is performed after all secondary label candidates have completed consistency score calculations and slot coverage corrections, and is the final decision-making stage of the labeling process. Specifically, for each candidate label branch, the system sorts the secondary label candidates within the branch in descending order of consistency score, first extracting the candidate with the highest score as the preliminary result for that branch; then, it queries the label relationships in the hierarchical label graph to check whether there is a mutual exclusion relationship between the current candidate and the selected candidates in other branches. If so, it compares the scores of the two and retains the one with the higher score, while removing the one with the lower score. For example, if "dry skin" (score 1.35) and "oily skin" (score 1.28) are selected at the same time, the system retains "dry skin" and removes "oily skin" based on the mutual exclusion constraint.

[0089] When multiple candidate scores within a branch are close (difference < 0.1), the system activates a tie-break rule: prioritizing candidates with higher slot coverage; if coverage is the same, selecting candidates with more diverse evidence sources (more evidence from different heterogeneous models); if still indistinguishable, retaining all tied candidates and marking them as requiring manual review. Furthermore, the system uses a label co-occurrence probability matrix (obtained from historical sample statistics) to verify the rationality of multi-label outputs: if the historical co-occurrence probability of two non-mutually exclusive labels is less than 0.05 (e.g., "acne scar repair" and "lip makeup recommendation" rarely appear simultaneously), the system applies a penalty factor of 0.2 to the sum of their consistency scores, reducing the probability of unreasonable combinations. Finally, a JSON structure of the target label result is generated, including the label name, confidence level, first-level branch, key evidence summary (e.g., "12-18 seconds of facial red marks + 15 seconds of text 'fade acne scars'"), evidence keyframe timestamps, and slot coverage, for downstream business use or manual review.

[0090] The above embodiments have described the basic process of video labeling based on label-evidence consistency maps. To handle edge cases where feature overlap causes the confidence level of secondary labels to fall below a preset threshold, this application further provides, based on the above, the following... Figure 2 The optimized embodiment shown focuses on illustrating the specific process of introducing historical key similar samples for comparison verification and score adaptive calibration.

[0091] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the video multimodal labeling method in this application embodiment.

[0092] S201. When the consistency score of a candidate secondary label is lower than a preset threshold, a preliminary screening is performed in the historical labeled sample library based on the candidate set of primary labels of the video to be labeled to obtain a candidate sample pool.

[0093] The preset threshold is a boundary value used to judge whether the consistency score of the secondary label candidates is reliable enough. It is usually set according to the score distribution of historical successful annotation cases (such as taking the 75th percentile). Candidates below this threshold are marked as low confidence and need to be calibrated. The historical annotated sample library is used to represent the annotated video database accumulated by the system that has completed manual review or automatic verification. Each sample record includes video features, primary labels, secondary labels, evidence slot filling status, label-evidence consistency graph structure and final annotation results. The candidate sample pool represents a subset of potentially relevant samples initially screened from the historical sample library based on the primary label candidate set, which is a pre-selected set for subsequent fine retrieval.

[0094] Specifically, the system first scans the consistency scores of all secondary label candidates within the current candidate label branch. If at least one candidate's score is found to be below a preset threshold (e.g., 1.2), the historical sample retrieval process is initiated. The system extracts the primary label candidate set of the video to be labeled (e.g., {"skincare": 0.8, "makeup": 0.5}), and performs initial screening in the historical labeled sample library: selecting all samples whose primary label fields intersect with the current candidate set (i.e., the primary label of the historical samples belongs to either {"skincare" or "makeup"}), forming a candidate sample pool.

[0095] For example, if the sample database contains 100,000 records, of which 30,000 are labeled "skincare" and 20,000 are labeled "makeup," then the candidate sample pool will be approximately 50,000 records (with some overlap). This coarse-grained screening utilizes the coarse-grained filtering capability of the first-level labels to quickly narrow down the search scope, avoiding the huge overhead of calculating high-dimensional feature similarity across the entire database. The system builds a first-level label index in the candidate sample pool to accelerate the query and prepares the dataset for subsequent vector similarity retrieval.

[0096] S202. Extract the multimodal feature vector of the video to be labeled, and use the multimodal feature vector to perform vector similarity retrieval in the candidate sample pool to determine the preset number of historical samples with the highest similarity as the initial recall set.

[0097] Among them, multimodal feature vectors refer to high-dimensional vector representations generated by fusing visual, text, and audio information. They are usually obtained by joint encoding of video keyframe sequences and auxiliary information text by a pre-trained multimodal encoder (such as CLIP). The dimension is generally 512 or 768. The closer the vectors are in semantic space, the more similar the content is. Vector similarity retrieval is used to represent the efficient nearest neighbor search operation performed in vector databases (such as Faiss and Milvus). By calculating the cosine similarity or Euclidean distance between the query vector and the vectors in the database, the top-K most similar samples are returned. The preset number K represents the number of historical samples initially recalled. It is usually set to 50-100 to achieve a balance between recall rate and fine screening computation. The preliminary recall set is an ordered list of the K most relevant historical samples selected based on multimodal feature similarity, which provides candidate input for subsequent graph structure similarity fine screening.

[0098] This step is performed immediately after obtaining the candidate sample pool to further filter out historical samples with similar content. Specifically, the system first performs multimodal feature extraction on the video to be labeled: the keyframe sequence of the video (representative frames are extracted from each segment) is visually encoded using a pre-trained CLIP model, and all auxiliary information text is text-encoded. The visual vector and text vector are then fused into a 512-dimensional multimodal feature vector through a weighted average (the weights are dynamically adjusted according to the proportion of visual content and the richness of text information).

[0099] Subsequently, a vector index is built in the candidate sample pool (if the sample pool is large, an approximate nearest neighbor algorithm such as HNSW can be used). The feature vector of the video to be labeled is used as the query vector, and Top-K similarity retrieval is performed. The cosine similarity between the query vector and the feature vector of each sample in the candidate sample pool is calculated, and the top K=50 samples are returned in descending order of similarity. For example, if the video to be labeled is a skincare tutorial explaining acne scar repair, its feature vector is close to the vector distance of other acne scar repair videos in the historical database in the semantic space. The Top-50 samples recalled by the system are likely to be videos on the same topic, and the similarity scores may be distributed in the range of 0.75-0.92. These recalled samples constitute the preliminary recall set, which is significantly smaller than the candidate sample pool (50,000 entries), but the content relevance is significantly improved.

[0100] S203. Calculate the graph structure similarity between the label-evidence consistency graph of each historical sample in the preliminary recall set and the label-evidence consistency graph of the video to be labeled.

[0101] Among them, graph structure similarity refers to the degree of similarity between two label-evidence consistency graphs in terms of topological structure and node attributes. The calculation methods include graph edit distance, graph kernel function, or vector similarity after graph neural network embedding. This index not only compares the overlap of label sets, but also compares the evidence organization pattern, edge connection relationship and weight distribution, and can capture higher-order labeling pattern similarity. The label-evidence consistency graph is used as the comparison object here. The graph structure of historical samples is directly loaded from the sample library records, and the graph structure of the video to be labeled has been completed in step S105.

[0102] This step is performed after obtaining the initial recall set to refine the recalled samples. Specifically, the system loads the constructed label-evidence consistency graph (containing label layer nodes, evidence layer nodes, internal consistency edges, conflict edges, competing edges, and the weights of each edge) of the video to be labeled. Then, it traverses each historical sample in the initial recall set and reads the consistency graph structure corresponding to that sample from the sample library (the historical annotation process is also constructed and stored according to step S105). The system uses a graph kernel function method to calculate the similarity between the two graphs: first, it extracts the subgraph patterns of the two graphs (such as the connection methods and edge weight intervals of three-node subgraphs), counts the frequency of each subgraph pattern in the two graphs, and calculates the inner product of the frequency vectors as the graph kernel similarity; or it uses a graph neural network (GNN) method, inputting the two graphs into a pre-trained graph encoder to generate graph-level embedding vectors, and calculates the vector cosine similarity.

[0103] For example, if the consistency graph of the video to be labeled contains a "acne scar repair" tag node connected to 3 evidence nodes (visual, textual, and behavioral) and has a conflicting edge with "blemish fading," and a historical sample's graph also exhibits a similar pattern (also 3 supporting evidence nodes + 1 conflicting edge, with similar evidence types and edge weight distributions), then the graph structure similarity between the two graphs may reach 0.85. However, another historical sample, although also labeled "acne scar repair," has only 2 evidence nodes and no conflicting edges, so its graph structure similarity may only be 0.62. The system records the graph structure similarity score between each sample in the initial recall set and the video to be labeled, preparing for the next step of fine screening.

[0104] S204. Filter out historical samples with graph structure similarity higher than the preset screening threshold, and combine them to form a set of similar samples within the branch.

[0105] The preset screening threshold is a graph structure similarity threshold used to determine whether historical samples are sufficiently similar to the video to be labeled in terms of annotation patterns. It is usually set to 0.7-0.8 to ensure that the selected samples are highly consistent with the current case in terms of evidence organization and label constraints. The intra-branch similar sample set represents the set of high-quality historical samples retained after graph structure similarity screening within the current candidate label branch. These samples will serve as the empirical benchmark for subsequent calibration operations.

[0106] This step is performed after calculating the graph structure similarity between each historical sample in the initial recall set and the video to be labeled, completing the fine screening of historical samples. Specifically, the system iterates through the 50 historical samples in the initial recall set and their corresponding graph structure similarity scores, filtering out samples with scores higher than the preset fine screening threshold of 0.75. For example, among the 50 recalled samples, there may be 8 samples with a graph structure similarity exceeding 0.75, and these 8 samples are retained; the remaining 42 samples, although similar to the video to be labeled at the multimodal feature vector level (recalled in S202), have structural differences in evidence organization patterns and label relationship constraints, and are excluded.

[0107] The system combines the eight selected samples to form a set of similar samples within a branch, and records the final secondary label, consistency score, evidence slot filling status, and graph structure data for each sample. If the screening result is empty (i.e., no sample graph structure similarity exceeds the threshold), the system adopts a degradation strategy: lowering the fine screening threshold to 0.65 and re-screening; if it is still empty, the current branch is marked as having no reliable historical reference and the calibration step is skipped, maintaining the original consistency score for final decision-making or manual review.

[0108] S205. Calibrate the consistency score of each secondary label candidate based on the set of similar samples within the branch.

[0109] The intra-branch similar sample set refers to the set of high-quality historically labeled samples retained within the current candidate label branch after initial screening using multimodal features and fine screening using graph structure similarity.

[0110] This step is triggered after the graph structure similarity screening of historical samples is completed and a similar sample set within the branch is successfully constructed. It is mainly applied to complex labeling scenarios such as marginal cases or highly overlapping features. Specifically, the system compares the label-evidence consistency graph of the current video with the similar sample set within the branch. Because in vertically fine-grained scenarios (such as "acne scars" and "blemishes" in beauty videos), the initially preset slot weights often cannot cover all long-tail variants. The system extracts the score distribution and weight preferences of successfully labeled cases in the similar sample set as a dynamic benchmark for the current score. To further improve the system's scalability, the calibration objects can be extended to parameters such as slot weights, conflict penalty coefficients, label output thresholds, and candidate branch ranking weights, in addition to consistency scores, and a comprehensive calibration log recording and backtracking mechanism is established. This adaptive calibration based on graph structure similar samples transforms static rules into dynamic empirical benchmarks, effectively enhancing the stability and accuracy of marginal case labeling output.

[0111] In some preferred embodiments, the above steps specifically include: statistically analyzing the consistency score distribution characteristics of successfully labeled cases corresponding to secondary labels in the similar sample set within the branch, extracting the target quantile as the target confidence threshold; calculating the statistical median of the weights of each evidence slot in the similar sample set within the branch, comparing it with the initial weights of the evidence slot template, and calculating the calibration offset.

[0112] Among them, successful labeled cases refer to the label data in the historical sample library that has been manually verified or verified as correct by real online business; the target quantile refers to a specific statistical position (such as the 5th percentile) extracted from the consistency score distribution of successful cases, which is used to represent the minimum score threshold for the acceptance of this type of label; the target confidence threshold is the dynamic judgment passing line determined by the target quantile; the statistical median is used to represent the median value of the weight of a specific evidence slot in the similar sample set; the calibration offset is the difference between the statistical median and the initial weight of the current evidence slot template.

[0113] This step is executed immediately after extracting the set of similar samples within the branch. It is used to quantify the scoring benchmark and dynamic weight adjustment parameters from historical experience. Specifically, the system first traverses the set of similar samples, extracts all successful labeled cases for the corresponding secondary label (e.g., "acne scar repair"), plots a histogram of their consistency score distribution, and takes the 5th percentile (e.g., 1.15 points) as the target confidence threshold. Then, it calculates the median weight of the visual slots that actually play a role in these samples (assumed to be 0.5), and subtracts it from the initial template weight (e.g., 0.4) to obtain the calibration offset (+0.1). If the number of similar samples is insufficient (e.g., less than 5), the system adopts a conservative strategy, updating only the confidence threshold without updating the weight offset.

[0114] The consistency scores of each candidate secondary label are compensated using the calibration offset to obtain the candidate calibration score. When the difference between the candidate calibration score and the consistency score before calibration is greater than the preset score change threshold, the consistency score of the corresponding candidate secondary label is updated using the candidate calibration score. The updated consistency score is compared with the target confidence threshold. If the updated consistency score is lower than the target confidence threshold, the corresponding candidate secondary label is removed.

[0115] Among them, the candidate calibration score refers to the temporary score obtained by recalculating the score of each slot of the current secondary label candidate after weighting and compensating with the calibration offset; the score change threshold refers to the preset protection limit (such as 0.05) used to prevent excessive score oscillation, and the calibration is only recognized as valid when the score change exceeds this value.

[0116] This step is performed after calculating the calibration offset and the target confidence threshold, and it's the decision-making stage for determining the final retention or rejection of secondary label candidates. Specifically, the system adds the calibration offset (e.g., visual slot + 0.1) to the slot score corresponding to the current candidate and re-accumulates it to obtain the candidate's calibration score. Assuming the score before calibration is 1.10 and the candidate's calibration score after compensation is 1.22, the difference of 0.12 is greater than the preset score change threshold of 0.05, so the system updates the candidate's consistency score to 1.22. Subsequently, the system compares the updated score of 1.22 with the target confidence threshold (e.g., 1.15). Because 1.22 is greater than 1.15, the candidate is retained; if the updated score is only 1.12, which is lower than 1.15, it is rejected. To ensure business security, rejected candidates can be retained in the manual review queue.

[0117] The video multimodal marking device in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of a video multimodal marking device in the embodiments of this application.

[0118] It should be noted that, Figure 3 The structure of the video multimodal marking device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0119] like Figure 3As shown, the video multimodal marking device includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage section 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0120] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0121] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0122] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0124] Specifically, the video multimodal marking device of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the video multimodal marking method provided in the above embodiment.

[0125] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the video multimodal marking device described in the above embodiments; or it may exist independently and not assembled into the video multimodal marking device. The storage medium carries one or more computer programs, which, when executed by a processor of the video multimodal marking device, cause the video multimodal marking device to implement the video multimodal marking method provided in the above embodiments.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0127] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A video multimodal labeling method, characterized in that, The method includes: A hierarchical label map is established, which includes primary labels, secondary labels, relationships between labels, and evidence slot templates corresponding to each secondary label. The evidence slot templates are used to describe the judgment criteria for the corresponding secondary labels. The video to be labeled and its associated auxiliary information are obtained, and the video to be labeled and the auxiliary information are segmented and time-related to obtain a sequence of interrelated video segments and a sequence of auxiliary information segments. The video to be labeled and the auxiliary information are input into the first multimodal model to obtain a first-level label candidate set, and at least one candidate label branch is determined based on the first-level label candidate set. Each candidate label branch is input into at least two heterogeneous models to obtain multiple secondary label candidates and evidence information corresponding to the secondary label candidates. The evidence information is mapped to the evidence slot template. The spatiotemporal correlation, semantic correlation and the relationship between the evidence information are combined with the relationship between the tags to construct a tag-evidence consistency graph and calculate the consistency score of each secondary tag candidate. Based on the consistency scores of each secondary label candidate and the relationships between labels in the hierarchical label graph, the target label result is determined within the candidate label branch.

2. The method according to claim 1, characterized in that, The process of constructing a label-evidence consistency graph by combining the spatiotemporal and semantic relationships between the evidence information and the relationships between the labels, and calculating the consistency score of each secondary label candidate, specifically includes: Each secondary label candidate is used as a label layer node, and each piece of evidence information mapped to the evidence slot template is structured as an evidence layer node; An internal consistency edge is established between the label layer node and the corresponding evidence layer node. The weight of the internal consistency edge is determined based on the weighted sum of the temporal intersection-union ratio, spatial intersection-union ratio, and semantic similarity of the corresponding evidence information. Based on the relationship between the tags, conflict edges are established between tag layer nodes that have mutual exclusion relationships, and competition edges are established between different tag layer nodes whose spatiotemporal overlap of evidence information is greater than a preset overlap threshold. For each secondary label candidate, the initial consistency score of the secondary label candidate is obtained by subtracting the penalty weights of the connected conflicting and competing edges from the sum of the weights of all corresponding internal consistent edges.

3. The method according to claim 2, characterized in that, The evidence slot template is configured with modal importance weights for each evidence type. After the step of obtaining the consistency score of the secondary label candidates, the method further includes: Based on the modal importance weight corresponding to the evidence slot filled with evidence information under each secondary label candidate, the slot coverage of the corresponding secondary label candidate is calculated. The slot coverage is the ratio of the sum of the modal importance weights of the filled slots to the sum of the modal importance weights of all slots. Determine whether the slot coverage is lower than a preset minimum coverage threshold; If so, apply a confidence decay penalty to the initial consistency score of the secondary label candidate, and update the decayed score to the consistency score of the secondary label candidate. If not, then the consistency score of the secondary label candidate remains unchanged.

4. The method according to claim 1, characterized in that, Before the step of determining the target label result within the candidate label branch based on the consistency scores of each secondary label candidate and the inter-label relationships in the hierarchical label graph, the method further includes: When the consistency score of a secondary label candidate is lower than a preset threshold, a preliminary screening is performed in the historical labeled sample library based on the primary label candidate set of the video to be labeled to obtain a candidate sample pool. Extract the multimodal feature vector of the video to be labeled, and use the multimodal feature vector to perform vector similarity retrieval in the candidate sample pool to determine a preset number of historical samples with the highest similarity as the initial recall set; Calculate the graph structure similarity between the label-evidence consistency graph of each historical sample in the preliminary recall set and the label-evidence consistency graph of the video to be labeled; Historical samples with a graph structure similarity higher than a preset screening threshold are selected and combined to form the set of similar samples within the branch. The consistency score of each secondary label candidate is calibrated based on the set of similar samples within the branch.

5. The method according to claim 4, characterized in that, The calibration of the consistency score of each secondary label candidate based on the set of similar samples within the branch specifically includes: The consistency score distribution characteristics of successful labeling cases corresponding to secondary labels in the similar sample set within the branch are statistically analyzed, and the target quantile is extracted as the target confidence threshold. Calculate the statistical median of the weights of each evidence slot in the similar sample set within the branch, and compare it with the initial weights of the evidence slot template to calculate the calibration offset; The consistency score of each candidate secondary label is compensated using the calibration offset to obtain the candidate calibration score; When the difference between the candidate calibration score and the consistency score before calibration is greater than a preset score change threshold, the consistency score of the corresponding secondary label candidate is updated using the candidate calibration score. The updated consistency score is compared with the target confidence threshold. If the updated consistency score is lower than the target confidence threshold, the corresponding secondary label candidate is removed.

6. The method according to claim 1, characterized in that, The step of segmenting and temporally associating the video to be labeled and the auxiliary information to obtain a sequence of interrelated video segments and a sequence of auxiliary information segments specifically includes: The video to be labeled is initially segmented based on the rate of change of the image to obtain a basic video segment sequence. Extract the multimodal feature vectors of adjacent basic video segments from the basic video segment sequence, and calculate the semantic continuity score between adjacent basic video segments; When the semantic continuity score is higher than the preset merging threshold, and the auxiliary information segments corresponding to the adjacent basic video segments have semantic referential association, the adjacent basic video segments are merged on the time axis to generate an updated video segment sequence. Based on the updated video clips and the corresponding time intervals, the auxiliary information is realigned, and the video clip sequence and the auxiliary information clip sequence are output.

7. The method according to claim 6, characterized in that, The step of inputting the video segment sequence and auxiliary information segment sequence corresponding to each candidate label branch into at least two heterogeneous models to obtain multiple secondary label candidates and the evidence information corresponding to the secondary label candidates specifically includes: Obtain the label density of the candidate label branch and the historical confusion degree of each of the corresponding secondary labels in historical data; Based on the label density and the historical confusion, at least two heterogeneous models are selected from a preset model pool. When both the label density and the historical confusion are higher than a preset complexity threshold, a first heterogeneous model for extracting visual features and a second heterogeneous model for text semantic understanding are selected as the at least two heterogeneous models. Each video segment sequence and each auxiliary information segment sequence are input into the at least two heterogeneous models for joint reasoning to obtain multiple secondary label candidates and evidence information corresponding to each secondary label candidate.

8. A video multimodal marking device, characterized in that, The video multimodal marking device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the video multimodal marking device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the video multimodal marking device, the video multimodal marking device performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the video multimodal marking device, the video multimodal marking device performs the method as described in any one of claims 1-7.

Images