Lightweight semantic ontology-based type constraint iterative positive and negative sample construction method

Abstract

The present application relates to the technical field of data processing, and more particularly to a type constraint iterative positive and negative sample construction method based on a lightweight semantic ontology. Video data and action metadata are obtained to generate structured action descriptions and perform format and semantic quality inspection. A slot word table is counted to construct a semantic ontology containing multiple relationship types. Under the constraint of the semantic ontology, the original slot value is replaced with a same-layer easily confused or mutually exclusive node to generate a candidate negative sample description. The model error sample is determined as a positive and negative sample through a two-option judgment of a visual language model, and the next round of sampling probability is iteratively adjusted according to the wrong answer rate until the stop condition is reached. Finally, context review is performed to remove semantically identical or visually indistinguishable entries to form a video, positive description and negative description triple. The method effectively improves the precision and efficiency of positive and negative sample construction and enhances the discrimination ability of the video description model.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for constructing positive and negative samples based on type constraints using a lightweight semantic ontology. Background Technology

[0002] In the current field of video action understanding, model training often relies on large-scale manually annotated image-text pairs, where the construction of positive and negative descriptions is particularly crucial. Existing conventional approaches often employ rule-based or template-based automatic generation methods, such as randomly replacing core verbs, objects, or scene nouns in action descriptions to construct semantically distinct negative samples. Another common approach utilizes pre-trained language models or knowledge graphs, attempting to generate semantically plausible negative samples through synonym, antonym, or hyponym relationships. These methods are widely used in data augmentation, aiming to improve the model's ability to discriminate action details.

[0003] Random or simple synonym substitutions often generate negative samples with low discrimination. For example, replacing "running" with "walking" makes it difficult to form a clear visual boundary, preventing the model from effectively learning key action features. While substitutions based on pre-trained language models consider semantic relationships, they lack constraints on the specific action context and visual consistency in the video, easily producing visually invalid or semantically redundant samples. For example, replacing "throw" with "spare" in a "throwing a ball" scene results in a minimal or even indistinguishable visual difference. These low-quality negative samples interfere with model convergence and fail to truly enhance the ability to discriminate action types, object attributes, or interaction details.

[0004] Existing methods typically generate a large number of negative samples at once and then mix or use them directly, without dynamically adjusting the replacement strategy based on the model's recognition difficulty. The differences in importance between different action types and slots are ignored, resulting in a lack of targeted training for easily confused actions or rare slots, thus limiting overall generalization performance. Therefore, existing techniques struggle to consistently produce high-quality, discriminative positive and negative samples, hindering the improvement of accuracy in video action understanding models. Summary of the Invention

[0005] This invention provides a method for constructing positive and negative samples based on type constraints of a lightweight semantic ontology, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a method for constructing positive and negative samples based on type constraints using a lightweight semantic ontology, comprising:

[0007] Acquire video data and action metadata, and use a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots labeled. Slot values ​​are derived from video footage observation. Perform two layers of automatic quality checks on slot descriptions: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked.

[0008] A slot vocabulary is formed by statistically analyzing the frequency of slot values ​​in each slot. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused intra-level relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot value with easily confused or mutually exclusive nodes within the same slot. A visual language model is used to perform a binary choice between the video, the original description, and the candidate negative description. Candidate negative samples that are incorrectly selected by the model are determined as positive and negative samples. The error rate of each slot in this round is calculated, and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met.

[0009] Contextual review is performed on positive and negative samples, and items whose original values ​​and replaced values ​​are semantically identical or visually indistinguishable in the specific action context are removed, forming a triplet of video, positive description, and negative description.

[0010] Generate structured action descriptions with multiple preset slots labeled, including:

[0011] The multiple preset slots include eleven semantic dimensions: instrument, contact point, contact method, body posture and alignment state, movement trajectory, movement name, force exertion point, force exertion method, anatomical left and right sides, gender, and observer's perspective.

[0012] After extracting visual features frame by frame from the video, the visual language model combines the action category information in the action metadata to map the visual features of each frame to the slot value field of the corresponding semantic dimension. The slot value is assigned based on the visual language model’s direct visual observation of the video content rather than reasoning and deduction from the metadata text tags.

[0013] Semantic independence verification is performed on the slot values ​​of each slot within the same action description. Slot by slot, it is checked whether there is a situation where a slot value of a certain semantic dimension contains information belonging to other semantic dimensions. For the detected semantically intersecting slot values, dimension backtracking and re-labeling are performed until each slot value remains semantically pure within its own semantic dimension and does not overlap with the information of other slots.

[0014] Slots for which the visual language model cannot obtain sufficient visual information to fill in the slot value in the video are marked as slots to be confirmed. These slots, along with the corresponding keyframe intervals in the video, are submitted to the manual annotation interface, where the annotators determine and fill in the slot values ​​based on the actual video footage.

[0015] The slot description undergoes two layers of automatic quality checks: format validation and semantic review.

[0016] The first-level format validation iterates through all the slot value fields of the marked slots, detects illegal key names, non-Chinese slot values, dimension cross-referencing errors where slot values ​​are inserted into adjacent semantic dimensions, and missing items with empty slot values. Samples that do not conform to the format specifications are marked with error type and error slot position and then rejected to the queue to be repaired.

[0017] The second layer of semantic review involves the language model reading the slot descriptions that have passed the first layer of verification one by one, evaluating whether each slot value is consistent with the visual content that can actually be observed in the video, and rating the discernibility of each slot value as to whether it can be directly determined from the visual features of the video. Slot values ​​with a consistency or discernibility level lower than the preset judgment standard are marked as semantic deviations and the deviation type is recorded.

[0018] Samples that fail the first or second layer of quality inspection are resubmitted to the first layer of quality inspection entry point after the manual annotator directly corrects the slot values ​​marked as semantic deviation or format error based on the video footage, and then enters the re-inspection cycle.

[0019] After this round of quality inspection is completed, the error type distribution, error slot frequency and revision operation records of all quality inspection samples will be completely saved as the quality inspection history. This history will be transferred to the next round of quality inspection as an automatic comparison reference for similar error patterns. Error patterns that have appeared frequently in the quality inspection history will be given priority for detection and interception in the next round of quality inspection.

[0020] A semantic ontology is constructed based on a slot-based vocabulary, including synonym relationships, hyponymous relationships, antonyms, easily confused homonymous relationships, and mutually exclusive relationships.

[0021] For each slot value node in the slot vocabulary, two rounds of language model calls are performed to generate and verify the data. In the first round, the data node is generated based on the context distribution of the slot value node in the dataset, resulting in eight types of ontology attributes: English name, Chinese definition, synonym set, hypernym set, hyponym set, antonym set, easily confused same-level node set, and mutually exclusive node set. In the second round, the generated results from the first round are used as input to perform semantic verification on each of the eight types of attributes and correct any relationship deviations.

[0022] Easily confused same-layer relationships are defined as a type of node relationship where two different slot value nodes under the same semantic dimension have similarity or substitutability in semantic expression, but the actual visual content presented in their corresponding video frames has clear and identifiable differences.

[0023] Mutually exclusive relationships are defined as a type of node relationship in which two different slot value nodes under the same semantic dimension are semantically opposed to each other and their corresponding visual presentation features in the video frame are also clearly distinguishable.

[0024] Visual discernibility is checked on the set of relations generated in the first round of calls. The visual corresponding areas of the nodes at both ends of each relation in the actual video frame are traversed. Visually indistinguishable relation node pairs are removed, and only semantic relation pairs with observable differences in the video visual dimension are retained as the components of the final ontology.

[0025] Under semantic ontology constraints, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused or mutually exclusive nodes within the same slot, including:

[0026] Based on the weighted sampling probability of each slot in this round, the target slot to be replaced is determined. The set of easily confused nodes and the set of mutually exclusive nodes corresponding to the original slot value are queried from the semantic ontology using the original slot value of the target slot as the retrieval key. The two sets are then merged into a candidate replacement node pool.

[0027] The scope of the replacement operation is strictly limited to the slot value field of the single semantic dimension of the target slot. The slot values ​​of the other ten semantic dimensions in the structured action description remain completely consistent with the original description without any changes. This ensures that the generated candidate negative sample description and the original description only have controlled slot value differences in the single semantic dimension of the target slot.

[0028] Based on the semantic distance between each candidate node and the original slot value in the semantic ontology, probability sampling is performed from the candidate replacement node pool. The closer the semantic distance, the higher the probability of the candidate node being sampled. The selected replacement node is written to the position of the target slot value to replace the original slot value, generating a candidate negative sample description that is semantically different from the original description only in the target slot.

[0029] Perform the same format validation process as the original slot description on the candidate negative sample description to ensure that the replacement operation does not introduce illegal key names, non-Chinese values, or new format-level errors due to dimension crossover.

[0030] Replacing the original slot value with a confusing node or a mutually exclusive node within the same slot under semantic ontology constraints also includes:

[0031] Identify associated slots from the semantic ontology that are semantically coupled with the target replacement slot. The semantic coupling relationship is defined as a dependency relationship between two slot values ​​of different semantic dimensions that are semantically mutually constrained and cannot be changed independently. When the slot value of one of the slots changes, the validity of the original slot value of the associated slot needs to be re-evaluated in the updated action semantic context.

[0032] After the replacement operation of the target slot value is completed, the list of all semantic coupling rules triggered by the change of the target slot value to the replacement node in the semantic ontology is traversed. For each coupling rule, the slot value of the associated slot is synchronously adjusted according to the specific semantic constraint content of the coupling rule, so that the adjusted slot value of the associated slot and the semantic connotation expressed by the replacement node maintain overall consistency and logical self-consistency at the action semantic level.

[0033] The target slot replacement operation and the associated slot synchronization adjustment operation jointly generate a multi-dimensional semantically consistent composite negative sample description. Compared with the original description, the composite negative sample description has controlled semantic differences in the target slot and each associated slot after synchronization adjustment, but the overall action semantic description still maintains internal semantic closure and logical coherence.

[0034] Perform contextual review on positive and negative samples, removing entries where the original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, including:

[0035] Traverse the original slot value and the replaced slot value of each record in the positive and negative sample database, retrieve the relationship type between the two in the semantic ontology, and if the two are synonymous node pairs and are verified to point to the same visual content or the same observable fact state in the specific action context presented in the corresponding video, they are determined to be semantically identical, and the entry is removed from the positive and negative sample database.

[0036] Multiple non-continuous keyframes are extracted from the video to form a frame sampling set. The visual representation areas corresponding to the original slot value and the replaced slot value are located in each frame and compared frame by frame. If no visual representation difference caused by slot value replacement can be observed in all keyframes of the frame sampling set, it is determined to be visually indistinguishable and the entry is removed from the positive and negative sample library.

[0037] Using a language model, sentence-level semantic information of the context of the replacement slot value in each positive and negative sample record is analyzed, and conclusions and reasoning texts for determining semantically identical and visually indistinguishable values ​​are output respectively.

[0038] For questionable entries whose confidence level is lower than a preset threshold by the language model, they are automatically marked as manually reviewed entries and the original video, original description and the replaced description are displayed for comparison. The annotator then makes a final decision on whether to retain or remove the entry based on the actual video footage.

[0039] A second aspect of this invention provides a type-constrained iterative positive and negative sample construction system based on a lightweight semantic ontology, comprising:

[0040] The description generation unit is used to acquire video data and action metadata. It uses a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots. The slot values ​​are derived from video footage observation. The slot descriptions undergo two layers of automatic quality checks: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked.

[0041] The ontology construction unit is used to count the frequency of slot values ​​under each slot to form a slot vocabulary. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused same-layer relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused nodes or mutually exclusive nodes in the same slot.

[0042] The positive and negative sample unit is used to perform a binary choice between video, original description and candidate negative description using a visual language model. The candidate negative sample that the model selects incorrectly is determined as a positive or negative sample. The error rate of each slot in this round is calculated and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met.

[0043] The context review unit is used to perform context review on positive and negative samples, and remove entries whose original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, forming a triplet of video, positive description and negative description.

[0044] A third aspect of the present invention provides an electronic device, comprising:

[0045] processor;

[0046] Memory used to store processor-executable instructions;

[0047] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0048] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0049] This method automates the high-quality construction process for both positive and negative samples, significantly reducing manual intervention. A visual language model combines video footage and metadata to generate multi-slot structured action descriptions, which undergo dual quality checks—format verification and semantic review—to ensure the accuracy and reliability of the slot information. A semantic ontology constructed by statistically analyzing slot value frequencies systematically encompasses synonyms, hyponyms, antonyms, easily confused elements at the same level, and mutually exclusive relationships. This provides precise semantic constraints for negative sample generation, avoiding semantic contradictions or irrelevant noise introduced by random replacements and improving sample effectiveness.

[0050] Under semantic ontology constraints, candidate negative samples are generated by replacing the original slot values ​​with easily confused or mutually exclusive nodes. The visual language model uses a binary choice judgment to automatically filter out samples incorrectly selected by the model as positive and negative samples, significantly reducing the cost of manual screening. During the iteration process, the sampling probability of the next round is adjusted by weighting the error rate of each slot, so that the slots that are easily confused by the model have more opportunities to generate difficult samples. The distribution of the difficulty of negative samples is adaptively optimized, continuously approaching the actual discrimination bottleneck, and significantly enhancing the challenge and diversity of the final negative sample set.

[0051] The context-based review mechanism rigorously removes entries where the original and replaced values ​​are semantically identical or visually indistinguishable in the specific action context, ensuring that each pair of negative samples has clear semantic distinction and preventing invalid samples from contaminating the training set. The resulting video, positive description, and negative description triplet has extremely low noise and can be directly used to train the fine-grained action understanding ability of the visual language model, significantly improving the model's action recognition accuracy in complex scenes, while effectively reducing the maintenance costs of subsequent manual correction and data cleaning. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the method for constructing positive and negative samples based on type constraints using a lightweight semantic ontology. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Figure 1 This is a flowchart illustrating the type-constrained iterative positive and negative sample construction method based on a lightweight semantic ontology according to an embodiment of the present invention, including:

[0056] Acquire video data and action metadata, and use a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots labeled. Slot values ​​are derived from video footage observation. Perform two layers of automatic quality checks on slot descriptions: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked.

[0057] The frequency of slot values ​​under each slot is counted to form a slot vocabulary. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused same-layer relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused nodes or mutually exclusive nodes in the same slot.

[0058] The visual language model is used to make a binary choice between video, original description and candidate negative description. The candidate negative samples that the model selects incorrectly are determined as positive and negative samples. The error rate of each slot in this round is calculated and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met.

[0059] Contextual review is performed on positive and negative samples, and items whose original values ​​and replaced values ​​are semantically identical or visually indistinguishable in the specific action context are removed, forming a triplet of video, positive description, and negative description.

[0060] Generate structured action descriptions with multiple preset slots labeled, including:

[0061] The multiple preset slots include eleven semantic dimensions: instrument, contact point, contact method, body posture and alignment state, movement trajectory, movement name, force exertion point, force exertion method, anatomical left and right sides, gender, and observer's perspective.

[0062] After extracting visual features frame by frame from the video, the visual language model combines the action category information in the action metadata to map the visual features of each frame to the slot value field of the corresponding semantic dimension. The slot value is assigned based on the visual language model’s direct visual observation of the video content rather than reasoning and deduction from the metadata text tags.

[0063] Semantic independence verification is performed on the slot values ​​of each slot within the same action description. Slot by slot, it is checked whether there is a situation where a slot value of a certain semantic dimension contains information belonging to other semantic dimensions. For the detected semantically intersecting slot values, dimension backtracking and re-labeling are performed until each slot value remains semantically pure within its own semantic dimension and does not overlap with the information of other slots.

[0064] Slots for which the visual language model cannot obtain sufficient visual information to fill in the slot value in the video are marked as slots to be confirmed. These slots, along with the corresponding keyframe intervals in the video, are submitted to the manual annotation interface, where the annotators determine and fill in the slot values ​​based on the actual video footage.

[0065] In generating structured motion descriptions, the design of pre-defined slots is fundamental to the entire annotation system. Addressing the semantic complexity of human motion videos, slots are divided into eleven semantic dimensions: instrument, contact point, contact method, body posture and alignment, motion trajectory, motion name, force exertion point, force exertion method, anatomical left and right sides, gender, and observer's perspective. These eleven dimensions orthogonally decompose the same motion from different angles, covering the motion's physical attributes (instrument, contact point, contact method), kinematic attributes (motion trajectory, motion name, force exertion point, force exertion method), anatomical attributes (anatomical left and right sides, body posture and alignment), and observer's perspective attributes (gender, observer's perspective). This multi-dimensional decomposition allows each structured description to express the semantic content of the motion in a fine-grained and comparable form, providing a clear space for slot replacement operations in the subsequent construction of positive and negative samples.

[0066] When processing video, the visual language model first extracts visual features frame by frame, obtaining a high-dimensional visual representation of each frame. Based on this, and combining the action category information provided in the action metadata, the visual features of each frame are mapped to the corresponding semantic dimension slot value field. It is crucial to emphasize that the slot values ​​must be assigned based on the visual language model's direct visual observation of the video content, rather than inference from the metadata text labels. For example, the "contact area" slot should be filled based on the contact area between the human body part and the equipment or environment actually observed by the model in the video frame, rather than semantic association based on the action name recorded in the metadata. The core purpose of this requirement is to ensure a high degree of consistency between the annotation results and the video content, avoiding deviations from the true visual information due to prior bias in the metadata labels, thereby ensuring the semantic alignment quality between the positive description in the subsequent triples and the video content.

[0067] After completing the initial slot value filling, semantic independence verification needs to be performed on the slot values ​​of each slot within the same action description. The goal of semantic independence verification is to detect whether there are any cases where information belonging to other semantic dimensions is mixed into the slot values ​​of a certain semantic dimension. Taking a specific scenario as an example, if the filled value of the "force exertion part" slot is "right quadriceps muscle exerts force to push forward", where "right" belongs to the anatomical left and right side dimension, and "push forward" belongs to the action trajectory dimension, neither of these two parts of information belongs to the force exertion part dimension itself, so this slot value has a semantic overlap problem. For the detected semantically overlapping slot values, a dimension backtracking operation is performed to separate the cross-dimensional information from the current slot value and backfill it into the corresponding correct dimension slots. Then, the current slot is re-labeled so that it only retains the semantic content belonging to its own dimension. The above detection and re-labeling process is iteratively executed until all slot values ​​remain semantically pure within their respective semantic dimensions and do not overlap with the information of other slots. The introduction of semantic independence verification fundamentally guarantees the orthogonality among the eleven semantic dimensions, enabling subsequent slot replacement operations under semantic ontology constraints to be performed precisely within a single dimension, without the semantic boundaries of the replacement operations becoming blurred due to semantic pollution of slot values.

[0068] For situations where the visual language model cannot obtain sufficient discriminative information from the video footage to complete the slot value filling, the corresponding slot is marked as a slot awaiting confirmation. Typical scenarios triggering this marking include: insufficient video resolution leading to indistinguishable details, key parts being occluded during action execution, poor lighting conditions resulting in loss of color or contour information, and excessively fast action speed preventing a single frame from capturing the complete motion trajectory. When marking a slot awaiting confirmation, the keyframe interval most relevant to the slot's judgment is extracted simultaneously from the video. The keyframe interval is determined based on the model's assessment of the visual information content of each frame, prioritizing the continuous frame segments containing the richest visual cues related to the slot. The slot awaiting confirmation and its corresponding keyframe interval are submitted together to the manual annotation interface, where annotators manually determine and fill in the slot value based on the actual video footage. The manual annotation interface supports frame-level browsing, slow-motion playback, and local area zoom-in functions to assist annotators in making accurate judgments when visual information is limited. Manually entered slot values ​​must also undergo semantic independence verification to ensure that the manual annotation results are consistent with the model's automatic annotation results in terms of format specifications and semantic purity requirements.

[0069] It's worth noting that the logic for filling the "Observer's Perspective" slot differs somewhat from other slots. This slot describes the spatial position of the camera relative to the actor performing the action during video recording, such as a frontal view, side view, or top-down view. Its slot value is derived from a direct visual judgment of the video composition and shooting angle, rather than a description of the action itself. The "Anatomy Left and Right Sides" slot is also unique. Its assignment requires distinguishing between left and right in an anatomical sense (i.e., left and right relative to the midline of the actor's body) and left and right in a spatial sense (i.e., left and right relative to the observer's perspective). These two often have a mirror relationship in videos with non-frontal perspectives. The annotation guidelines explicitly require that the assignment of values ​​for the "Anatomy Left and Right Sides" slots uniformly adopt anatomical standards, that is, using the actor himself as the reference frame, rather than the direction of the image, thereby ensuring the semantic consistency of the slot value is not affected by the shooting angle.

[0070] The "gender" slot is filled based on the physical characteristics of the person performing the action in the video frame. The visual language model makes the judgment based on visual observation. If the physical characteristics in the frame are insufficient to support the judgment, it is also marked as a slot to be confirmed and handed over to manual annotation. This approach is consistent with the processing logic of other slots with insufficient visual information, reflecting the consistent implementation of the principle of prioritizing visual observation throughout the entire annotation process.

[0071] Through the above-mentioned multi-dimensional slot design, visual observation-first assignment specification, semantic independence verification mechanism, and human-computer collaborative unconfirmed slot processing flow, the final generated structured action description meets the quality requirements for subsequent positive and negative sample construction in terms of semantic dimension integrity, visual reliability of slot values, and cross-dimensional semantic independence, providing a high-quality annotation foundation for negative sample generation based on slot replacement.

[0072] The slot description undergoes two layers of automatic quality checks: format validation and semantic review.

[0073] The first-level format validation iterates through all the slot value fields of the marked slots, detects illegal key names, non-Chinese slot values, dimension cross-referencing errors where slot values ​​are inserted into adjacent semantic dimensions, and missing items with empty slot values. Samples that do not conform to the format specifications are marked with error type and error slot position and then rejected to the queue to be repaired.

[0074] The second layer of semantic review involves the language model reading the slot descriptions that have passed the first layer of verification one by one, evaluating whether each slot value is consistent with the visual content that can actually be observed in the video, and rating the discernibility of each slot value as to whether it can be directly determined from the visual features of the video. Slot values ​​with a consistency or discernibility level lower than the preset judgment standard are marked as semantic deviations and the deviation type is recorded.

[0075] Samples that fail the first or second layer of quality inspection are resubmitted to the first layer of quality inspection entry point after the manual annotator directly corrects the slot values ​​marked as semantic deviation or format error based on the video footage, and then enters the re-inspection cycle.

[0076] After this round of quality inspection is completed, the error type distribution, error slot frequency and revision operation records of all quality inspection samples will be completely saved as the quality inspection history. This history will be transferred to the next round of quality inspection as an automatic comparison reference for similar error patterns. Error patterns that have appeared frequently in the quality inspection history will be given priority for detection and interception in the next round of quality inspection.

[0077] Performing two layers of automated quality checks—format validation and semantic review—on slot descriptions is the core step in ensuring the quality of structured action description annotations. These two layers of checks are independent yet sequentially linked; the former performs mechanical checks on structural compliance, while the latter conducts in-depth semantic verification, together forming a complete quality barrier covering both format and content.

[0078] The first layer of format validation iterates through all labeled slot value fields, checking each one for four typical errors. Illegal key name errors refer to slot field names that are not within the set of legal key names defined in the preset slot terminology. For example, if an annotator mistakenly writes "Main Action" instead of "Action Subject" in the slot, or uses a custom abbreviation, these are considered illegal key names, directly marked, and rejected. Non-Chinese slot value errors refer to slot value fields containing English characters, numbers, or mixed language expressions. Since the entire annotation system uses Chinese as the base language, any non-Chinese content is considered formatting abnormal and requires a forced rollback to the annotation stage for re-entry. Dimension cross-reference errors refer to slot value content that actually belongs to an adjacent semantic dimension. For example, "Quickly wave your arm" which should be in the "Action Method" slot is mistakenly entered into the "Action Result" slot, or the content of the "Use Tool" slot is mixed into the "Action Object" slot. Such errors can cause cross-dimensional confusion in the subsequent semantic ontology construction stage, and therefore need to be identified through slot value matching with the slot semantic template. Missing item errors refer to slot value fields being empty or containing only blank characters. Under the current annotation specifications, all preset slots are required fields, and any empty value is considered an annotation omission. For samples that detect any of the above types of errors, their error type label and the corresponding error slot position index are recorded. The sample is then rejected and placed in the repair queue, pausing its entry into the second-level verification.

[0079] The second layer of semantic review involves the language model reading each slot description that has passed the first layer of format validation. Combining this with the corresponding video content, two independent evaluations are performed on each slot value. The first evaluation focuses on consistency, checking whether the content described by the slot value matches the visually observable information in the video. The language model uses video keyframes or video description summaries as visual references, comparing the slot values ​​with the video content to determine if there are discrepancies. For example, if a person in the video actually uses their left hand to perform an action, but the annotator filled in "right hand" in the "action / limb" slot, this is marked as a consistency deviation. The second evaluation focuses on discernibility, checking whether the attribute described by the slot value can be directly determined from the visual features of the video, rather than relying on the annotator's subjective inference or off-screen audio. Content clearly presented in the video receives a higher discernibility rating; content that is difficult to judge directly due to occlusion, motion blur, or limited viewpoint receives a lower discernibility rating. Slot values ​​with a consistency or distinguishability level lower than the preset judgment standard are marked as semantic deviations, and the deviation types are recorded, including two categories: consistency deviation and distinguishability deviation, so that they can be clearly and specifically corrected during subsequent manual revisions.

[0080] Samples that fail the first or second level of quality control are transferred to the manual annotation revision process. Manual annotators directly revise the slot values ​​marked as semantically biased or formatted incorrectly based on the video footage. The revision is strictly limited to the marked slots; unmarked slot values ​​remain unchanged to avoid introducing new errors. After revision, the sample is resubmitted to the first level of quality control entry point, entering a re-inspection cycle. The re-inspection cycle is completely identical to the initial quality control process, undergoing both format verification and semantic review again. If errors still exist, the sample is rejected again and returned to the repair queue until it passes both levels of quality control. This mechanism ensures that every sample that ultimately enters the subsequent processes undergoes complete two-layer verification, with no exceptions where a sample passes only a single layer of inspection.

[0081] After the completion of this round of quality inspection, the error type distribution of all quality inspection samples, the error frequency statistics for each slot, and all revision operation records of manual annotators will be completely saved as a structured quality inspection history. The quality inspection history is organized by slot as the primary key. Each record includes the sample identifier, error type label, error slot location, slot value before revision, slot value after revision, and the round number of this quality inspection, forming a traceable and complete revision chain. When the next round of quality inspection is initiated, the above quality inspection history will be passed into the quality inspection process as an automatic comparison reference, prioritizing the detection and interception of error patterns that frequently appear in the history. Specifically, if the error frequency of a certain slot in the history exceeds a preset threshold, a stricter rule matching will be applied to the slot value in the next round of format verification, and a lower judgment standard will be adopted for the consistency and distinguishability evaluation of the slot in the semantic review stage. This allows for early interception of similar problems in error-prone locations, reducing the workload of manual revision and improving the overall stability of annotation quality.

[0082] As quality inspection rounds accumulate, the error pattern library in the quality inspection history becomes increasingly rich, and the coverage of automatic comparison references continues to expand. For specific error combinations that repeatedly appear in multiple quality inspection rounds, such as the simultaneous occurrence of dimension cross-referencing errors in a certain slot and missing item errors in adjacent slots, these can be treated as associated error patterns for overall detection in subsequent rounds. Once one of these errors is detected, proactive verification of associated slots is triggered, further improving the efficiency of error detection in the format verification stage. The semantic review level also benefits from the accumulation of historical records. For slots with frequently occurring semantic deviations in the past, the language model can prioritize referring to the revision results of historical deviation samples when evaluating each item, using them as auxiliary basis for similar judgments, thereby improving the stability and consistency of the judgment. This entire two-layer quality inspection mechanism, through the layered design of format constraints and semantic verification, combined with continuous optimization driven by manual revision and historical records, forms a closed-loop annotation quality control system, providing highly reliable structured action description inputs for subsequent semantic ontology construction and positive and negative sample generation.

[0083] A semantic ontology is constructed based on a slot-based vocabulary, including synonym relationships, hyponymous relationships, antonyms, easily confused homonymous relationships, and mutually exclusive relationships.

[0084] For each slot value node in the slot vocabulary, two rounds of language model calls are performed to generate and verify the data. In the first round, the data node is generated based on the context distribution of the slot value node in the dataset, resulting in eight types of ontology attributes: English name, Chinese definition, synonym set, hypernym set, hyponym set, antonym set, easily confused same-level node set, and mutually exclusive node set. In the second round, the generated results from the first round are used as input to perform semantic verification on each of the eight types of attributes and correct any relationship deviations.

[0085] Easily confused same-layer relationships are defined as a type of node relationship where two different slot value nodes under the same semantic dimension have similarity or substitutability in semantic expression, but the actual visual content presented in their corresponding video frames has clear and identifiable differences.

[0086] Mutually exclusive relationships are defined as a type of node relationship in which two different slot value nodes under the same semantic dimension are semantically opposed to each other and their corresponding visual presentation features in the video frame are also clearly distinguishable.

[0087] Visual discernibility is checked on the set of relations generated in the first round of calls. The visual corresponding areas of the nodes at both ends of each relation in the actual video frame are traversed. Visually indistinguishable relation node pairs are removed, and only semantic relation pairs with observable differences in the video visual dimension are retained as the components of the final ontology.

[0088] After obtaining the slot vocabulary, a structured semantic ontology needs to be constructed for each slot value node in the vocabulary so that meaningful slot value replacement can be performed under semantic constraints in the subsequent positive and negative sample generation stage. The construction of the semantic ontology adopts a two-round language model invocation mechanism of "generation-verification", which sequentially completes the two stages of attribute generation and relation correction for each slot value node.

[0089] For each slot value node in the vocabulary, the first round of language model calls uses the context distribution of that node in the dataset as the core input. Specifically, the surrounding slot description text, action scene description, and video metadata when the slot value node appears in all labeled samples are fed into the language model. The model is required to generate eight types of ontology attributes, including English name, Chinese definition, synonym set, hypernym set, hyponym set, antonym set, easily confused sibling node set, and mutually exclusive node set. The system includes: an English name for standardized node representation to avoid ambiguity due to language differences; a Chinese definition to clarify the semantic boundaries of the slot value within the action context; a synonym set for other semantically equivalent and interchangeable slot value representations; a hypernym set for more abstract parent nodes; a hyponym set for more concrete sub-nodes; an antonym set for nodes semantically opposite to the node; and sets for easily confused sibling nodes and mutually exclusive nodes, which require consideration of the specific video action scene and referencing the visual characteristics of the slot value in the actual video footage during generation. The prompts in the first call must explicitly specify a structured JSON output format, with one field corresponding to each attribute type, for subsequent programmatic processing.

[0090] The second round of language model calls uses the eight attribute categories generated in the first round as complete input, performing semantic verification on each attribute category. The core objective of the verification is to identify potential relational biases that may have occurred in the first round of generation, such as mislabeling hypernyms as synonyms or misclassifying easily confused intra-level relationships as antonyms. The second round of calls requires the model to provide a decision on whether to retain, modify, or delete each node in each attribute category, and to provide reasons for modifying or deleting entries. For the synonym set, the verification focuses on confirming whether candidate synonyms are truly interchangeable with the original node in the context of the action, avoiding the misclassification of words that are only superficially similar but semantically different into the synonym set. For the hypernym and hyponym sets, the verification focuses on the correctness of the direction of the hierarchical relationship. For the antonym set, the verification focuses on whether the opposition relationship holds true in the semantic dimension of the slot. For easily confused intra-level node sets and mutually exclusive node sets, the second round of calls also needs to combine visual feature descriptions from the video footage for auxiliary judgment to ensure that the visual discernibility conditions of the relationship definition are met.

[0091] In this method, easily confused co-level relationships have specific technical connotations and need to be strictly distinguished from synonymous and hierarchical relationships. Easily confused co-level relationships are defined as follows: under the same semantic dimension, two different slot value nodes have similarity or substitutability at the semantic expression level; that is, from the perspective of textual description, they are quite similar and easily confused. However, in their respective corresponding video frames, the actual visual content presented has clear and identifiable differences. Taking action type slots as an example, "jogging" and "brisk walking" are semantically similar, both describing human movement behavior and having overlapping speed ranges, making them easily confused. However, in the video frames, their gait characteristics and the state of their feet off the ground show observable visual differences, thus constituting an easily confused co-level relationship. This type of relationship is the most challenging source of substitution in constructing positive and negative samples because the model needs to rely on visual information rather than textual information to correctly distinguish them.

[0092] Mutually exclusive relationships are also defined within the same semantic dimension, but their core characteristics lie in the semantic opposition and the clear distinction in visual presentation. A mutually exclusive relationship requires that the two endpoints not only be semantically opposed but also have clearly distinguishable differences in their corresponding visual areas within the video frame. Taking the action direction slot as an example, "clockwise" and "counterclockwise" are semantically opposed, and their rotation directions are also clearly visually distinguishable in the video frame, thus constituting a mutually exclusive relationship. The difference between mutually exclusive and antonymous relationships is that antonymous relationships emphasize semantic opposition, while mutually exclusive relationships, in addition to semantic opposition, emphasize the constraint of visual distinguishability, ensuring that the relationship can serve as a valid basis for positive and negative sample substitution.

[0093] After two rounds of language model calls, a visual distinguishability review is performed on all relation sets generated in the first round and corrected in the second round. This review step traverses the nodes at both ends of each relation, locates the corresponding visual regions of these two nodes in the actual video frame, and compares the visual content. The visual distinguishability review can be assisted by a visual language model: image pairs containing typical video frames of the two nodes are input into the visual language model, and the model is asked whether it can distinguish the content described by the two nodes based solely on visual information, and gives a judgment of distinguishability or indistinguishability. For relation node pairs judged to be visually indistinguishable, regardless of their semantic relation type, they are removed and not included in the final ontology. The purpose of this removal mechanism is to ensure that all semantic relation pairs retained in the ontology have observable differences in the video visual dimension, thereby ensuring that the candidate negative samples generated based on the ontology have visual distinguishability, making the positive and negative sample selection task meaningful.

[0094] The relationship pairs retained after visual discernibility review, together with the other six types of attributes of the corresponding nodes, constitute the final semantic ontology. This ontology is organized in the form of a graph structure, with nodes representing slot values ​​and edges representing the five types of semantic relationships mentioned above. In the subsequent positive and negative sample generation stage, the slot value replacement of candidate negative samples is strictly performed under the constraints of the ontology graph structure. Replacement is only allowed along edges that are easily confused with each other or mutually exclusive relationships. This ensures that the generated candidate negative samples are semantically deceptive while visually distinguishable from the original slot values, providing effective test samples for the subsequent visual language model binary choice task.

[0095] Under semantic ontology constraints, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused or mutually exclusive nodes within the same slot, including:

[0096] Based on the weighted sampling probability of each slot in this round, the target slot to be replaced is determined. The set of easily confused nodes and the set of mutually exclusive nodes corresponding to the original slot value are queried from the semantic ontology using the original slot value of the target slot as the retrieval key. The two sets are then merged into a candidate replacement node pool.

[0097] The scope of the replacement operation is strictly limited to the slot value field of the single semantic dimension of the target slot. The slot values ​​of the other ten semantic dimensions in the structured action description remain completely consistent with the original description without any changes. This ensures that the generated candidate negative sample description and the original description only have controlled slot value differences in the single semantic dimension of the target slot.

[0098] Based on the semantic distance between each candidate node and the original slot value in the semantic ontology, probability sampling is performed from the candidate replacement node pool. The closer the semantic distance, the higher the probability of the candidate node being sampled. The selected replacement node is written to the position of the target slot value to replace the original slot value, generating a candidate negative sample description that is semantically different from the original description only in the target slot.

[0099] Perform the same format validation process as the original slot description on the candidate negative sample description to ensure that the replacement operation does not introduce illegal key names, non-Chinese values, or new format-level errors due to dimension crossover.

[0100] When determining the target slot to be replaced, a weighted random sampling is performed on all slots based on the sampling probability distribution calculated by weighting the error rates of each slot after the current iteration. The sampling result is the target slot for constructing candidate negative samples in this round. Slots with higher error rates have a greater weight in the sampling probability distribution, which means that the visual language model has a harder time distinguishing positive and negative samples in this semantic dimension. The iteration process will tilt resources towards these slots, thus focusing the mining of positive and negative samples on the semantic dimension where the model is truly weak. After determining the target slot, the slot value recorded in the original description corresponding to the current video is used as the search key to perform a node query in the semantic ontology, extracting the set of easily confused nodes at the same level and the set of mutually exclusive nodes corresponding to the original slot value. Easily confused nodes at the same level are nodes that share some semantic features with the original slot value at the same level of the semantic ontology but have distinguishable differences. Mutually exclusive nodes are nodes that are clearly opposed to or cannot coexist with the original slot value in the semantic space. The union of the two sets forms a candidate replacement node pool, which serves as the source space for subsequent probability sampling.

[0101] After the candidate replacement node pool is established, the scope of the replacement operation needs to be strictly defined. The structured action description consists of slot fields with multiple semantic dimensions. Each slot corresponds to an independent semantic dimension, such as action subject, action category, action object, spatial location, action method, and duration. The replacement operation is strictly limited to the slot value field of the single semantic dimension of the target slot. The values ​​of all other slots remain completely consistent with the original description, without any modification. The core significance of this constraint is to ensure that the source of difference between the candidate negative sample description and the original description is singular and controllable—the only semantic difference between the two exists only in the semantic dimension corresponding to the target slot; the descriptive information of other dimensions is completely identical to both. If multiple slots are allowed to be replaced simultaneously, the visual language model will face interference from multi-dimensional differences when performing binary choice judgments, making it impossible to accurately determine which semantic dimension the model is confused about, and the semantic attribution of positive and negative samples will also lose accuracy. Through a controlled design of single-slot, single-replacement, each candidate negative sample description can be precisely traced to the dimension of its semantic deviation from the original description, providing a reliable attribution basis for subsequent error rate statistics and slot sampling probability updates.

[0102] When selecting a specific replacement node from the candidate replacement node pool, a probabilistic sampling mechanism based on semantic distance is employed. The semantic ontology organizes each slot value node and its relationships into a graph structure, and the semantic distance between nodes can be quantified using path lengths on the graph or a pre-calculated distance matrix. Let the original slot value node be... The candidate replacement node pool The candidate nodes are , and The semantic distance in the semantic ontology graph is The probability that the candidate node will be sampled is... Inversely proportional to semantic distance, specifically, ,in To control the steepness of the probability distribution, a temperature parameter is used to sum and iterate through all candidate nodes in the candidate replacement node pool. Candidate nodes with closer semantic distances have a higher probability of being sampled. The motivation for this design is that replacement nodes semantically similar to the original slot value are more visually difficult to distinguish from the original value. Therefore, the generated candidate negative sample descriptions exert stronger confusion pressure on the visual language model and are more likely to become true positive and negative samples. While nodes with greater semantic distances also retain a certain sampling probability to avoid complete exclusion, their selection frequency is lower, maintaining the overall difficulty level of candidate negative samples.

[0103] After probability sampling is completed, the selected replacement node is written to the slot value field of the target slot, replacing the original slot value. The contents of the remaining slot fields remain unchanged, thus generating a candidate negative sample description that differs from the original description only in the target slot. After generation, the candidate negative sample description immediately undergoes the same format validation process as the original slot description. Format validation covers multiple levels: checking whether all key names in the description are valid key names for the preset slots, disallowing illegal key names introduced by the replacement operation; checking whether all slot values ​​are Chinese characters, excluding non-Chinese characters or mixed-encoding values ​​introduced during the node writing process; and checking whether each slot value is written to a slot field that matches its semantic dimension, preventing dimension misalignment errors, i.e., a value of one semantic dimension is incorrectly written to a slot field of another semantic dimension. Dimension misalignment errors are a type of hidden format problem. Their harm lies in the fact that even if a single slot value is valid, if it appears under an incorrect semantic dimension, the overall semantic structure of the description will be distorted. Subsequent binary selection by the visual language model will be based on a structurally flawed candidate negative sample, rendering the screening results worthless. Therefore, format validation must cover the legality checks of dimension attribution, rather than being limited to surface verification at the key name and character level.

[0104] Candidate negative sample descriptions that pass format validation will proceed to the subsequent visual language model's binary selection stage. Descriptions that fail format validation are discarded, and sampling is re-performed from the candidate replacement node pool to generate new candidate negative sample descriptions, until the generated candidate descriptions pass format validation. This closed-loop guarantee mechanism ensures that every candidate negative sample description entering the judgment stage meets the legality requirements at the structural level, fundamentally preventing noise samples caused by format defects from being mixed into the positive and negative sample set. The entire candidate negative sample description generation process—from target slot determination, candidate replacement node pool construction, semantic distance weighted probability sampling, controlled replacement of single slots, to format validation closed loop—constitutes a complete and internally consistent generation mechanism, providing a quality-controlled source of candidates for subsequent positive and negative sample screening.

[0105] Replacing the original slot value with a confusing node or a mutually exclusive node within the same slot under semantic ontology constraints also includes:

[0106] Identify associated slots from the semantic ontology that are semantically coupled with the target replacement slot. The semantic coupling relationship is defined as a dependency relationship between two slot values ​​of different semantic dimensions that are semantically mutually constrained and cannot be changed independently. When the slot value of one of the slots changes, the validity of the original slot value of the associated slot needs to be re-evaluated in the updated action semantic context.

[0107] After the replacement operation of the target slot value is completed, the list of all semantic coupling rules triggered by the change of the target slot value to the replacement node in the semantic ontology is traversed. For each coupling rule, the slot value of the associated slot is synchronously adjusted according to the specific semantic constraint content of the coupling rule, so that the adjusted slot value of the associated slot and the semantic connotation expressed by the replacement node maintain overall consistency and logical self-consistency at the action semantic level.

[0108] The target slot replacement operation and the associated slot synchronization adjustment operation jointly generate a multi-dimensional semantically consistent composite negative sample description. Compared with the original description, the composite negative sample description has controlled semantic differences in the target slot and each associated slot after synchronization adjustment, but the overall action semantic description still maintains internal semantic closure and logical coherence.

[0109] In the process of constructing positive and negative samples, the replacement operation of a single slot can sometimes cause internal contradictions in the overall action description. This can lead to negative samples being easily identified as incorrect descriptions by visual language models due to semantic inconsistency, thus losing their core value as "difficult" negative samples. To solve this problem, it is necessary to introduce a mechanism for identifying and synchronously adjusting the semantic coupling relationship between slots within the framework of semantic ontology. This would enable the replacement operation to collaboratively generate semantically consistent composite negative sample descriptions across multiple slot dimensions.

[0110] In semantic ontology, different slots are not independent of each other, but rather exist in a special kind of dependency relationship, namely semantic coupling. Taking action description as an example, there is semantic coupling between the "holding method" slot and the "using tool" slot: when the value of the "using tool" slot is changed from "chopsticks" to "fork", the original value of the "holding method" slot, "clamping", is no longer valid in the new action semantic context and should be adjusted to a slot value that is consistent with the use of a fork, such as "stab" or "pick". Similarly, there may also be a coupling relationship between the "body part" slot and the "action range" slot, and there may also be mutual constraints between the "walking speed" slot and the "gait type" slot. The essence of semantic coupling is that there is a mutually restrictive dependency relationship between the slot values ​​of two different semantic dimensions. Once the value of one slot changes, the original value of the other slot needs to be re-evaluated for validity in the updated semantic context, and the original value cannot be simply retained.

[0111] During the semantic ontology construction phase, the identification of coupling relationships between slots is achieved through the following methods: Statistical analysis is performed on the co-occurrence patterns of each slot value in the slot vocabulary to identify frequently co-occurring or mutually exclusive slot value combinations in real-world labeled data. Statistically significant co-occurrence constraints are then used as candidate coupling rules and entered into the semantic ontology. Simultaneously, these candidate coupling rules are manually reviewed using domain knowledge to eliminate pseudo-coupling relationships caused by sample bias, ensuring that the coupling rules ultimately entered into the ontology have a genuine semantic basis. Each coupling rule is stored in the ontology as a directed relational edge. The starting point of the edge is the triggering slot and its specific slot value (or slot value range), and the ending point is the affected associated slot. The edge includes a set of legal slot values ​​or an adjustment constraint description for the associated slot under the triggering condition.

[0112] When generating candidate negative samples, after completing the target slot replacement operation, immediately traverse the entire list of semantic coupling rules triggered by the change event "target slot value becomes replacement node" in the semantic ontology. Let the original slot value of the target slot be... The semantics of the replaced node are The set of associated slots that are coupled with the target slot is denoted as . , of which Each associated slot is denoted as .for Each associated slot Query from semantic ontology The coupling rules triggered by this change The rule clearly describes in In the semantic context of the expressed action, The set of valid slot values .like Current slot value Already belong to If no adjustment is needed; if Then from China based on and Prioritize the selection of target values ​​based on semantic similarity. Complete the process for associated slots Synchronous adjustment.

[0113] The priority strategy for synchronous adjustment is reflected in: In the middle, priority is given to selecting with The legal value with the smallest semantic distance at the same level (i.e., same-level relation nodes) in the semantic ontology is used to ensure that the adjustment range is minimized, so that the difference between the composite negative sample description and the original description remains within a controlled range, and the negative sample does not become obvious due to excessive deviation from the original action semantics caused by the adjustment of the associated slot. If no node at the same level exists, the nearest valid value covered by the parent node is selected. The entire synchronization adjustment process is triggered by the target slot replacement operation, and each associated slot is processed sequentially according to the coupling rule list to form a complete multi-slot collaborative change operation.

[0114] It's worth noting that adjusting associated slots can itself trigger a new layer of coupling rules, i.e., a propagation chain with coupling relationships. To avoid infinite recursion, an upper limit is set on the coupling propagation depth in actual implementations. When the propagation depth exceeds When this happens, propagation stops, the original value of the associated slot in the current layer is retained, and the entry is marked as requiring manual review. In most practical action description scenarios, the coupling propagation depth does not exceed two layers, therefore... Setting it to 2 is usually sufficient to meet the requirements.

[0115] After replacing the target slot and simultaneously adjusting the associated slots, the generated composite negative sample description structurally satisfies the following characteristics: Compared with the original description, the target slot and all synchronously adjusted associated slots exhibit controlled semantic differences, while the remaining slots that did not trigger coupling rules retain their original values; within the description, the semantic constraints between the values ​​of each slot conform to the legality rules in the semantic ontology, and the overall action description maintains semantic closure and logical coherence, without internal contradictions. This multi-dimensional semantic consistency makes the composite negative sample description appear on the surface as "another reasonable but incorrect action description," rather than an obviously self-contradictory description, thus significantly enhancing the deceptiveness of the negative sample to the visual language model, making it more difficult for the model to easily exclude it.

[0116] In the subsequent visual language model's binary choice process, the composite negative sample description is presented alongside the original positive sample description in a holistic form. The model needs to determine which description is more accurate with the aid of video footage. Because the composite negative sample description differs from the original description in multiple slots, and the values ​​in each slot are internally consistent, the model cannot quickly eliminate negative samples based solely on the semantic irrationality of a single slot. It must comprehensively judge multiple visual cues in the video footage. This places higher demands on the model's fine-grained visual semantic understanding capabilities and also makes the selected positive and negative samples more valuable for training.

[0117] During the context review phase, for composite negative sample descriptions, in addition to performing the original semantic substance similarity test on the replacement value of the target slot, it is also necessary to examine each synchronously adjusted associated slot individually to see if the slot value before and after the adjustment constitutes a visually distinguishable semantic difference in the specific action context. If the value before and after the adjustment of an associated slot cannot form an observable visual difference in the video frame, the adjustment of the associated slot will not be included in the effective difference dimension, but it will not affect the retention judgment of the entire composite negative sample description. The retention judgment is still mainly based on the visual difference produced by the replacement of the target slot.

[0118] Perform contextual review on positive and negative samples, removing entries where the original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, including:

[0119] Traverse the original slot value and the replaced slot value of each record in the positive and negative sample database, retrieve the relationship type between the two in the semantic ontology, and if the two are synonymous node pairs and are verified to point to the same visual content or the same observable fact state in the specific action context presented in the corresponding video, they are determined to be semantically identical, and the entry is removed from the positive and negative sample database.

[0120] Multiple non-continuous keyframes are extracted from the video to form a frame sampling set. The visual representation areas corresponding to the original slot value and the replaced slot value are located in each frame and compared frame by frame. If no visual representation difference caused by slot value replacement can be observed in all keyframes of the frame sampling set, it is determined to be visually indistinguishable and the entry is removed from the positive and negative sample library.

[0121] Using a language model, sentence-level semantic information of the context of the replacement slot value in each positive and negative sample record is analyzed, and conclusions and reasoning texts for determining semantically identical and visually indistinguishable values ​​are output respectively.

[0122] For questionable entries whose confidence level is lower than a preset threshold by the language model, they are automatically marked as manually reviewed entries and the original video, original description and the replaced description are displayed for comparison. The annotator then makes a final decision on whether to retain or remove the entry based on the actual video footage.

[0123] After iterative filtering, the positive and negative sample database cannot be directly used as training data because it may still contain a special type of noise entries: the original slot value and the replaced slot value actually point to the same visual content in a specific action context, or they are simply indistinguishable at the video frame level. If such entries are included in the training set, they will transmit incorrect contrast signals to the model, thereby impairing the model's ability to perceive fine-grained visual differences. Therefore, it is necessary to perform contextual review on the positive and negative sample database to systematically identify and remove such entries.

[0124] The first dimension of contextual review is determining semantically equivalent relationships. Each record in the positive and negative sample database is traversed, extracting the original and replaced slot values. The relationship type between these two values ​​is then retrieved from the constructed semantic ontology graph. If the search results indicate that they are synonymous node pairs, further verification is performed by considering the specific video and action context corresponding to the entry. Synonymous node pairs are labeled as near-synonyms or equivalents at the semantic ontology level, but the semantic boundaries of two words may diverge in different action scenarios. For example, "grasp" and "hold" are synonyms in a general context, but may correspond to distinguishable hand gestures in certain fine-grained operation scenarios. Therefore, determining synonymy solely based on ontology relationship type is insufficient; verification must also consider the specific action context presented in the video. The verification method involves observing whether the action elements described by the original and replaced slot values ​​in the corresponding frame sequence of the video point to the same observable visual content or the same observable factual state in the video. If the verification result is positive, the entry is determined to be semantically identical and removed from the positive and negative sample database. The reason for removal and related evidence are recorded.

[0125] The second dimension of contextual review is the determination of visual indistinguishability. For entries not eliminated by the semantically identical rule, a further visual distinguishability test is performed at the video level. Specifically, multiple non-contiguous keyframes are extracted from the corresponding video to form a frame sampling set. The purpose of the non-contiguous sampling strategy is to cover different stages of the action process, avoiding the omission of key action transition moments due to minimal changes between consecutive frames, and also avoiding misjudgments caused by accidental similarities at a fixed point in time within the video. In each frame of the frame sampling set, the visual representation areas corresponding to the original slot value and the replaced slot value are located, and a frame-by-frame comparison is performed within these areas. The location of the visual representation area depends on the slot type; for example, the action slot corresponds to the human joint movement area, the tool slot corresponds to the handheld object area, and the position slot corresponds to the spatial relationship area. If no visual difference resulting from slot value replacement can be observed in any of the keyframes of the frame sampling set, the entry is determined to be visually indistinguishable and is removed from the positive and negative sample database. It should be emphasized that the condition for visual indistinguishability is "no difference in all keyframes". If there is no difference only in some frames but observable difference in other frames, the elimination condition is not met and the entry is retained.

[0126] Both of the aforementioned dimensions of review are automated in practice using language models. The language model receives positive and negative sample records one by one. Input information includes: the original slot value, the replaced slot value, complete structured action description text, and keyframe images extracted from the video. Supported by contextual information, the language model outputs independent conclusions for semantically identical and visually indistinguishable judgments, along with textual reasoning. This reasoning text requires the model to explicitly explain the basis for its judgment, such as indicating that both describe the same hand movement trajectory in the video, or that the replaced tool and the original tool visually present the same form and usage across all keyframes. The purpose of requiring the model to output reasoning is twofold: firstly, to provide a reference for subsequent manual review; and secondly, to improve the accuracy of the model's judgments through a chain-like reasoning process, reducing random misjudgments caused by the model directly outputting conclusions.

[0127] Language models inherently exhibit uncertainty when handling boundary cases; therefore, a confidence assessment mechanism is introduced into the automated decision-making process. The language model outputs a confidence score for each decision. Entries with confidence scores below a preset threshold are automatically marked for manual review, and the system does not automatically make a decision to retain or remove them. The manual review interface presents the annotator with three sets of comparison information: the original video (supporting playback and frame-by-frame viewing), the original structured description text, and the replaced structured description text, along with the reasoning text output by the language model for reference. Based on the actual video footage and the reasoning provided by the language model, the annotator makes a final decision to retain or remove the entry and records the decision. The annotator's final decision has the highest priority, overriding the language model's automatic decision.

[0128] After three stages of processing—semantically identical elimination, visually indistinguishable elimination, and manual review—the remaining entries in the positive and negative sample database all meet the following conditions: the original slot value and the replaced slot value have a semantically distinguishable difference, and this difference has an observable visual correspondence at the video frame level. Based on these entries, the video, positive description (including the structured action description of the original slot value), and negative description (including the structured action description of the replaced slot value) are organized into triples to form the final training dataset. The triple structure ensures that each positive and negative sample pair is anchored to the same video content, forcing the model to rely on a fine understanding of the video frame to distinguish between positive and negative descriptions during training, rather than relying on the surface statistical features of the descriptive text. This effectively improves the model's fine-grained visual-language alignment ability in action understanding tasks.

[0129] A second aspect of this invention provides a type-constrained iterative positive and negative sample construction system based on a lightweight semantic ontology, comprising:

[0130] The description generation unit is used to acquire video data and action metadata. It uses a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots. The slot values ​​are derived from video footage observation. The slot descriptions undergo two layers of automatic quality checks: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked.

[0131] The ontology construction unit is used to count the frequency of slot values ​​under each slot to form a slot vocabulary. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused same-layer relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused nodes or mutually exclusive nodes in the same slot.

[0132] The positive and negative sample unit is used to perform a binary choice between video, original description and candidate negative description using a visual language model. The candidate negative sample that the model selects incorrectly is determined as a positive or negative sample. The error rate of each slot in this round is calculated and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met.

[0133] The context review unit is used to perform context review on positive and negative samples, and remove entries whose original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, forming a triplet of video, positive description and negative description.

[0134] A third aspect of the present invention provides an electronic device, comprising:

[0135] processor;

[0136] Memory used to store processor-executable instructions;

[0137] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0138] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0139] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing positive and negative samples based on type constraints using lightweight semantic ontology, characterized in that, include: Acquire video data and action metadata, and use a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots labeled. Slot values ​​are derived from video footage observation. Perform two layers of automatic quality checks on slot descriptions: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked. A slot vocabulary is formed by statistically analyzing the frequency of slot values ​​in each slot. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused intra-level relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot value with easily confused or mutually exclusive nodes within the same slot. A visual language model is used to perform a binary choice between the video, the original description, and the candidate negative description. Candidate negative samples that are incorrectly selected by the model are determined as positive and negative samples. The error rate of each slot in this round is calculated, and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met. Contextual review is performed on positive and negative samples, and items whose original values ​​and replaced values ​​are semantically identical or visually indistinguishable in the specific action context are removed, forming a triplet of video, positive description, and negative description.

2. The method according to claim 1, characterized in that, Generate structured action descriptions with multiple preset slots labeled, including: The multiple preset slots include eleven semantic dimensions: instrument, contact point, contact method, body posture and alignment state, movement trajectory, movement name, force exertion point, force exertion method, anatomical left and right sides, gender, and observer's perspective. After extracting visual features frame by frame from the video, the visual language model combines the action category information in the action metadata to map the visual features of each frame to the slot value field of the corresponding semantic dimension. The slot value is assigned based on the visual language model’s direct visual observation of the video content rather than reasoning and deduction from the metadata text tags. Semantic independence verification is performed on the slot values ​​of each slot within the same action description. Slot by slot, it is checked whether there is a situation where a slot value of a certain semantic dimension contains information belonging to other semantic dimensions. For the detected semantically intersecting slot values, dimension backtracking and re-labeling are performed until each slot value remains semantically pure within its own semantic dimension and does not overlap with the information of other slots. Slots for which the visual language model cannot obtain sufficient visual information to fill in the slot value in the video are marked as slots to be confirmed. These slots, along with the corresponding keyframe intervals in the video, are submitted to the manual annotation interface, where the annotators determine and fill in the slot values ​​based on the actual video footage.

3. The method according to claim 2, characterized in that, The slot description undergoes two layers of automatic quality checks: format validation and semantic review. The first-level format validation iterates through all the slot value fields of the marked slots, detects illegal key names, non-Chinese slot values, dimension cross-referencing errors where slot values ​​are inserted into adjacent semantic dimensions, and missing items with empty slot values. Samples that do not conform to the format specifications are marked with error type and error slot position and then rejected to the queue to be repaired. The second layer of semantic review involves the language model reading the slot descriptions that have passed the first layer of verification one by one, evaluating whether each slot value is consistent with the visual content that can actually be observed in the video, and rating the discernibility of each slot value as to whether it can be directly determined from the visual features of the video. Slot values ​​with a consistency or discernibility level lower than the preset judgment standard are marked as semantic deviations and the deviation type is recorded. Samples that fail the first or second layer of quality inspection are resubmitted to the first layer of quality inspection entry point after the manual annotator directly corrects the slot values ​​marked as semantic deviation or format error based on the video footage, and then enters the re-inspection cycle. After this round of quality inspection is completed, the error type distribution, error slot frequency and revision operation records of all quality inspection samples will be completely saved as the quality inspection history. This history will be transferred to the next round of quality inspection as an automatic comparison reference for similar error patterns. Error patterns that have appeared frequently in the quality inspection history will be given priority for detection and interception in the next round of quality inspection.

4. The method according to claim 1, characterized in that, A semantic ontology is constructed based on a slot-based vocabulary, including synonym relationships, hyponymous relationships, antonyms, easily confused homonymous relationships, and mutually exclusive relationships. For each slot value node in the slot vocabulary, two rounds of language model calls are performed to generate and verify the data. In the first round, the data node is generated based on the context distribution of the slot value node in the dataset, resulting in eight types of ontology attributes: English name, Chinese definition, synonym set, hypernym set, hyponym set, antonym set, easily confused same-level node set, and mutually exclusive node set. In the second round, the generated results from the first round are used as input to perform semantic verification on each of the eight types of attributes and correct any relationship deviations. Easily confused same-layer relationships are defined as a type of node relationship where two different slot value nodes under the same semantic dimension have similarity or substitutability in semantic expression, but the actual visual content presented in their corresponding video frames has clear and identifiable differences. Mutually exclusive relationships are defined as a type of node relationship in which two different slot value nodes under the same semantic dimension are semantically opposed to each other and their corresponding visual presentation features in the video frame are also clearly distinguishable. Visual discernibility is checked on the set of relations generated in the first round of calls. The visual corresponding areas of the nodes at both ends of each relation in the actual video frame are traversed. Visually indistinguishable relation node pairs are removed, and only semantic relation pairs with observable differences in the video visual dimension are retained as the components of the final ontology.

5. The method according to claim 1, characterized in that, Under semantic ontology constraints, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused or mutually exclusive nodes within the same slot, including: Based on the weighted sampling probability of each slot in this round, the target slot to be replaced is determined. The set of easily confused nodes and the set of mutually exclusive nodes corresponding to the original slot value are queried from the semantic ontology using the original slot value of the target slot as the retrieval key. The two sets are then merged into a candidate replacement node pool. The scope of the replacement operation is strictly limited to the slot value field of the single semantic dimension of the target slot. The slot values ​​of the other ten semantic dimensions in the structured action description remain completely consistent with the original description without any changes. This ensures that the generated candidate negative sample description and the original description only have controlled slot value differences in the single semantic dimension of the target slot. Based on the semantic distance between each candidate node and the original slot value in the semantic ontology, probability sampling is performed from the candidate replacement node pool. The closer the semantic distance, the higher the probability of the candidate node being sampled. The selected replacement node is written to the position of the target slot value to replace the original slot value, generating a candidate negative sample description that is semantically different from the original description only in the target slot. Perform the same format validation process as the original slot description on the candidate negative sample description to ensure that the replacement operation does not introduce illegal key names, non-Chinese values, or new format-level errors due to dimension crossover.

6. The method according to claim 5, characterized in that, Replacing the original slot value with a confusing node or a mutually exclusive node within the same slot under semantic ontology constraints also includes: Identify associated slots from the semantic ontology that are semantically coupled with the target replacement slot. The semantic coupling relationship is defined as a dependency relationship between two slot values ​​of different semantic dimensions that are semantically mutually constrained and cannot be changed independently. When the slot value of one of the slots changes, the validity of the original slot value of the associated slot needs to be re-evaluated in the updated action semantic context. After the replacement operation of the target slot value is completed, the list of all semantic coupling rules triggered by the change of the target slot value to the replacement node in the semantic ontology is traversed. For each coupling rule, the slot value of the associated slot is synchronously adjusted according to the specific semantic constraint content of the coupling rule, so that the adjusted slot value of the associated slot and the semantic connotation expressed by the replacement node maintain overall consistency and logical self-consistency at the action semantic level. The target slot replacement operation and the associated slot synchronization adjustment operation jointly generate a multi-dimensional semantically consistent composite negative sample description. Compared with the original description, the composite negative sample description has controlled semantic differences in the target slot and each associated slot after synchronization adjustment, but the overall action semantic description still maintains internal semantic closure and logical coherence.

7. The method according to claim 1, characterized in that, Perform contextual review on positive and negative samples, removing entries where the original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, including: Traverse the original slot value and the replaced slot value of each record in the positive and negative sample database, retrieve the relationship type between the two in the semantic ontology, and if the two are synonymous node pairs and are verified to point to the same visual content or the same observable fact state in the specific action context presented in the corresponding video, they are determined to be semantically identical, and the entry is removed from the positive and negative sample database. Multiple non-continuous keyframes are extracted from the video to form a frame sampling set. The visual representation areas corresponding to the original slot value and the replaced slot value are located in each frame and compared frame by frame. If no visual representation difference caused by slot value replacement can be observed in all keyframes of the frame sampling set, it is determined to be visually indistinguishable and the entry is removed from the positive and negative sample library. Using a language model, sentence-level semantic information of the context of the replacement slot value in each positive and negative sample record is analyzed, and conclusions and reasoning texts for determining semantically identical and visually indistinguishable values ​​are output respectively. For questionable entries whose confidence level is lower than a preset threshold by the language model, they are automatically marked as manually reviewed entries and the original video, original description and the replaced description are displayed for comparison. The annotator then makes a final decision on whether to retain or remove the entry based on the actual video footage.

8. A type-constrained iterative positive and negative sample construction system based on lightweight semantic ontology, used to implement the method as described in any one of claims 1-7, characterized in that, include: The description generation unit is used to acquire video data and action metadata. It uses a visual language model to combine video footage and metadata to generate structured action descriptions with multiple preset slots. The slot values ​​are derived from video footage observation. The slot descriptions undergo two layers of automatic quality checks: format verification and semantic review. Samples that fail the checks are manually revised and then re-checked. The ontology construction unit is used to count the frequency of slot values ​​under each slot to form a slot vocabulary. Based on the slot vocabulary, a semantic ontology is constructed that includes synonyms, hierarchical relationships, antonyms, easily confused same-layer relationships, and mutually exclusive relationships. Under the constraints of the semantic ontology, candidate negative sample descriptions are generated by replacing the original slot values ​​with easily confused nodes or mutually exclusive nodes in the same slot. The positive and negative sample unit is used to perform a binary choice between video, original description and candidate negative description using a visual language model. The candidate negative sample that the model selects incorrectly is determined as a positive or negative sample. The error rate of each slot in this round is calculated and the sampling probability of slot replacement in the next round is determined by weighting the error rate. The generation and screening of candidate negative samples are iteratively executed until the preset stopping condition is met. The context review unit is used to perform context review on positive and negative samples, and remove entries whose original value and the replaced value are semantically identical or visually indistinguishable in the specific action context, forming a triplet of video, positive description and negative description.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images