Multi-modal data-based intelligent VLA (Virtual Local Area) data pre-labeling method and device

Abstract

The invention relates to a multi-modal data-based intelligent VLA data pre-labeling method and device, and the method comprises the steps: obtaining a multi-modal data stream containing visual modal data, language modal data and action modal data, and carrying out the time sequence segmentation of the multi-modal data stream according to a time sequence, and obtaining a plurality of time period data units; for each time period data unit, identifying a current task execution state based on the visual modal data and the language modal data, and generating at least one candidate labeling hierarchy set corresponding to the state according to the task execution state; performing multi-modal semantic representation on each level label in the candidate label level set, and constructing a multi-modal consistency relation graph containing visual nodes, language nodes and action nodes; and executing a consistency propagation operation in the multi-modal consistency relation graph to obtain a multi-modal consistency constraint relation among the nodes. According to the scheme, the stability and reliability of multi-modal labeling in a complex scene can be improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method and apparatus for pre-annotating embodied intelligence VLA data based on multimodal data. Background Technology

[0002] With the development of artificial intelligence technology, multimodal data based on vision, language, and action are widely used in fields such as embodied intelligence, robot control, autonomous driving, and human-computer interaction. In these application scenarios, it is usually necessary to construct a VLA dataset containing visual information, language descriptions, and corresponding action behaviors for training or validating multimodal intelligent models.

[0003] In practical applications, VLA data typically originates from multimodal data streams collected during the operation of real systems. These data are large in scale, span long periods, and exhibit significant differences in collection frequency and representation among different modalities. Therefore, efficient and accurate annotation of multimodal data has become a crucial factor limiting the efficiency of VLA dataset construction.

[0004] Existing methods for labeling VLA data mainly include manual labeling, semi-automatic labeling, and model-based inference-based automatic labeling. While manual labeling offers high accuracy, it suffers from high cost, low efficiency, and difficulty in scaling. Semi-automatic labeling typically relies on manual correction of automatically generated results, still requiring significant human intervention. In contrast, model-based inference-based automatic labeling usually generates corresponding labeling results directly by extracting and matching features from multimodal data.

[0005] However, existing automatic or semi-automatic annotation methods typically treat multimodal consistency as a static result when processing multimodal data. They lack the ability to model the changes in multimodal semantic relationships as the processing progresses, making it difficult to accurately reflect the stability and reliability of multimodal annotation in complex scenarios. Summary of the Invention

[0006] Based on this, the present invention provides an embodied intelligent VLA data pre-annotation method and apparatus based on multimodal data, which can improve the stability and reliability of multimodal annotation in complex scenarios.

[0007] This invention provides a method for embodied intelligent VLA data pre-annotation based on multimodal data, comprising:

[0008] A multimodal data stream containing visual modal data, language modal data, and action modal data is acquired, and the multimodal data stream is segmented temporally according to the time order to obtain multiple time period data units;

[0009] For each time period data unit, the current task execution state is identified based on the visual modality data and the language modality data, and at least one set of candidate annotation levels corresponding to the task execution state is generated according to the task execution state.

[0010] Multimodal semantic representation is performed on each level of the candidate annotation hierarchy set, and a multimodal consistency relationship graph containing visual nodes, language nodes, and action nodes is constructed.

[0011] A consistency propagation operation is performed in the multimodal consistency relationship graph to obtain the multimodal consistency constraint relationship between each node and to detect conflicting node combinations.

[0012] For the detected combinations of conflicting nodes, based on the corresponding candidate annotation level, a counterfactual multimodal sample associated with the candidate annotation is constructed;

[0013] The counterfactual multimodal samples are introduced into the multimodal consistency graph, and multiple rounds of consistency backtracking verification are performed to determine the consistency stability of the candidate labels under different modal combinations.

[0014] Based on the consistency and stability results, annotation lifecycle state transition control is performed on the candidate annotations to determine the current annotation state of the candidate annotations;

[0015] When the annotation status of the candidate annotations meets the preset freezing conditions, the candidate annotations are determined as the pre-annotation results of VLA data and output.

[0016] In one embodiment, obtaining multiple time-segment data units includes:

[0017] Time synchronization and alignment processing is performed on the multimodal data stream composed of visual modal data, language modal data, and action modal data;

[0018] Following the temporal order of the multimodal data stream, candidate time period boundaries are determined segment by segment based on the action change information in the action modality data;

[0019] By combining scene change information in the visual modality data and semantic boundary information in the language modality data, the candidate time period boundary is corrected;

[0020] Based on the corrected time period boundaries, the multimodal data stream is divided into multiple time period data units.

[0021] In one embodiment, the step of performing multimodal semantic representation on each level of the candidate annotation hierarchy set and constructing a multimodal consistency graph containing visual nodes, language nodes, and action nodes includes:

[0022] For each level of annotation in the candidate annotation level set, a corresponding modality sub-consistency graph is constructed. The modality sub-consistency graph contains visual semantic units, linguistic semantic units, or action semantic units associated with the annotation at that level.

[0023] Based on the modal sub-consistency graph, a cross-modal consistency mapping graph is constructed according to the semantic mapping relationship between different modal semantic units to describe the correspondence between different modal sub-consistency graphs;

[0024] The modal sub-consistency graph is combined with the cross-modal consistency mapping graph to form a graph set structure of multimodal consistency relationship graph, wherein the graph set structure includes at least a primary consistency graph and multiple auxiliary consistency graphs;

[0025] An initial consistency status identifier is assigned to each graph node in the graph set structure. The consistency status identifier is used to characterize the consistency confidence level of the node under the current candidate annotation conditions.

[0026] Based on the hierarchical type and task execution status of the candidate annotations, the node states in the graph set structure are grouped and initialized so that nodes at different levels or in different states have different consistency state starting conditions.

[0027] In one embodiment, each graph node in the graph set structure of the multimodal consistency graph is categorized according to the semantic unit type it represents:

[0028] Visual nodes represent image features or object recognition results extracted from visual modalities;

[0029] Language nodes represent lexical or semantic information extracted from language modalities;

[0030] Action nodes represent motion or behavioral information extracted from action modalities.

[0031] In one embodiment, performing a consistency propagation operation in the multimodal consistency graph to obtain multimodal consistency constraints between nodes and detecting conflicting node combinations includes:

[0032] In the graph set structure of the multimodal consistency relationship graph, multiple consistency propagation stages are executed sequentially according to a preset propagation stage order, and each propagation stage corresponds to a different consistency propagation goal;

[0033] In each propagation phase, intra-layer consistency propagation and cross-layer consistency propagation are performed according to the consistency constraint type. Figure 1 Consistency propagation, where intra-layer consistency propagation is used to update the node state within the same modality sub-consistency graph, and cross-layer consistency propagation... Figure 1Consistency propagation is used to pass consistency impact information between the primary consistency graph and the secondary consistency graph;

[0034] During the consistency propagation process, the propagation path information corresponding to the changes in node state is recorded to form the node state evolution trajectory;

[0035] After completing at least one full propagation cycle, the evolution trajectory of the node state is analyzed to identify whether there are conflict evolution patterns in which the node state exhibits periodic oscillations, continuous degradation, or branching during multiple propagation stages.

[0036] When the conflict evolution pattern is detected, the corresponding node combination is determined to be a conflict node combination.

[0037] In one embodiment, the pre-annotation method satisfies one or more of the following:

[0038] The intra-layer consistency propagation is used to update the consistency state of nodes within the same modality, and the cross-modal consistency propagation is used to pass consistency between nodes in different modalities. The propagation is executed in a preset priority order, and the propagation priority between different modalities is dynamically adjusted according to the initial state of the nodes.

[0039] During the propagation process, the initial consistency state of the nodes and the historical propagation results are used to determine whether the propagation conditions are met.

[0040] When the consistency propagation condition is met, the next propagation phase continues; when the condition is not met, propagation is paused, waiting for more samples or condition updates.

[0041] Based on the changes in the consistency state of graph nodes, it is determined whether a node has reached a conflict threshold; wherein, the conflict threshold is determined based on the historical changes in the consistency state of the node, the correlation between modalities, and the credibility of the candidate labels.

[0042] In one embodiment, constructing counterfactual multimodal samples associated with the candidate labels based on the corresponding candidate label level for the detected conflict node combinations includes:

[0043] Based on the combination of conflicting nodes, candidate labels with a high degree of correlation with the conflict are determined;

[0044] For the identified candidate labels, at least one modality of data is replaced, perturbed, or masked;

[0045] Construct counterfactual multimodal samples based on the processed modal data.

[0046] In one embodiment, determining the consistency stability of the candidate labels under different modality combinations includes:

[0047] The counterfactual multimodal samples are incorporated into the multimodal consistency graph;

[0048] Perform multi-round consistency backtracking verification on the multimodal consistency graph after introducing counterfactual multimodal samples;

[0049] The consistency and stability of candidate labels are evaluated based on the results of multiple rounds of backtracking verification.

[0050] In one embodiment, the step of performing annotation lifecycle state transition control on the candidate annotations based on the consistency stability result to determine the current annotation state of the candidate annotations includes:

[0051] Based on the consistency stability results, the confidence level of the candidate labels is calculated;

[0052] Based on the matching results between the trust level and multiple preset trust level ranges or thresholds, the candidate annotation is controlled to perform state transitions between multiple preset annotation lifecycle states.

[0053] This invention provides an embodied intelligent VLA data pre-annotation device based on multimodal data, comprising:

[0054] The acquisition unit is used to acquire a multimodal data stream containing visual modal data, language modal data, and action modal data, and to perform time-series segmentation of the multimodal data stream according to the time order to obtain multiple time-segment data units;

[0055] The generation unit is used to identify the current task execution state based on the visual modal data and the language modal data for each time period data unit, and generate at least one set of candidate annotation levels corresponding to the task execution state.

[0056] The construction unit is used to perform multimodal semantic representation on each level of the candidate annotation hierarchy set and construct a multimodal consistency relationship graph containing visual nodes, language nodes and action nodes.

[0057] The detection unit is used to perform a consistency propagation operation in the multimodal consistency relationship graph to obtain the multimodal consistency constraint relationship between each node and to detect conflicting node combinations.

[0058] The construction unit is used to construct counterfactual multimodal samples associated with the candidate labels based on the corresponding candidate label level for the detected combination of conflicting nodes.

[0059] The processing unit is configured to introduce the counterfactual multimodal samples into the multimodal consistency graph, perform multi-round consistency backtracking verification to determine the consistency stability of the candidate labels under different modal combinations, and, based on the consistency stability results, perform label lifecycle state transition control on the candidate labels to determine the current labeling state of the candidate labels;

[0060] The output unit is used to determine the candidate annotation as the pre-annotation result of VLA data and output it when the annotation status of the candidate annotation meets the preset freezing conditions.

[0061] The above approach involves temporal segmentation of multimodal data streams (visual, linguistic, and action modalities) and joint identification of task execution status at the time-segment level to generate a candidate annotation hierarchy set. This ensures that the multimodal annotation process has clear task semantic constraints. Simultaneously, by constructing a multimodal consistency graph and performing consistency propagation and conflict detection, counterfactual multimodal samples are introduced for multi-round consistency backtracking verification. This systematically evaluates the consistency stability of candidate annotations under different modal combinations. Combined with an annotation lifecycle state transition control mechanism, highly consistent and stable VLA data pre-annotation results are output when preset freezing conditions are met. This effectively reduces multimodal annotation ambiguity and conflict risks, improves the accuracy, robustness, and automation of multimodal data annotation, and is suitable for constructing visual-linguistic-action multimodal data in complex task scenarios. Attached Figure Description

[0062] Figure 1 A flowchart illustrating an embodied intelligent VLA data pre-annotation method based on multimodal data, provided in an embodiment of the present invention;

[0063] Figure 2 A flowchart for determining multiple time period data units is provided in an embodiment of the present invention;

[0064] Figure 3 A flowchart for constructing a multimodal consistency graph is provided as an embodiment of the present invention;

[0065] Figure 4 A flowchart for determining multimodal consistency constraint relationships is provided in an embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the structure of an embodied intelligent VLA data pre-labeling device based on multimodal data, provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the specific details described below are only a part of the embodiments of the present invention, and the present invention can be implemented in many other embodiments different from those described herein. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0068] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0070] As mentioned in the background section, existing annotation schemes typically treat multimodal consistency as a static result, lacking the ability to model the changes in multimodal semantic relationships as the processing progresses, and thus failing to accurately reflect the stability and reliability of multimodal annotation in complex scenarios.

[0071] For example, existing methods often employ single-step consistency judgments or simple similarity calculations to evaluate the relationships between visual, linguistic, and action modalities. These methods typically treat multimodal consistency as a static result, lacking the ability to model how multimodal semantic relationships change during processing, and thus failing to accurately reflect the stability and reliability of multimodal annotation in complex scenarios.

[0072] Furthermore, existing multimodal annotation methods often fail to adequately consider the relationships between different semantic levels. For example, high-level semantic annotation, action sequence annotation, and action parameter annotation are often processed independently, lacking a unified structured representation. This makes it difficult to effectively establish consistency constraints between different levels of annotation, thereby further affecting the overall reliability of the annotation results.

[0073] Therefore, existing solutions generally suffer from insufficient consistency modeling capabilities, simplistic conflict handling mechanisms, and a lack of systematic description of the consistency evolution process in the pre-annotation of multimodal VLA data, making it difficult to meet the demand for high-quality VLA data annotation in complex application scenarios.

[0074] Based on this, this invention provides a VLA data pre-annotation method driven by multimodal consistency for complex task scenarios. By performing temporal segmentation and task state recognition on multimodal data streams containing visual, linguistic, and action modalities, it achieves precise alignment of multimodal information at both the temporal and semantic levels, reducing annotation ambiguity caused by task stage confusion or modal asynchrony from the source. Furthermore, it dynamically generates candidate annotation hierarchy sets for different task execution states and models them using multimodal semantic representation and consistency relationship graphs, enabling explicit expression and joint reasoning of semantic constraints between visual, linguistic, and action modalities, thereby effectively improving the overall consistency level of candidate annotations. Further, by performing consistency propagation and conflict detection in the multimodal consistency relationship graph and combining counterfactual multimodal samples for multi-round consistency backtracking verification, it achieves a systematic evaluation of the stability of candidate annotations under different modal combinations and assumptions, avoiding the risk of mislabeling caused by single judgments or localized information. Based on the consistency and stability results, the annotation lifecycle state transition control is implemented for candidate annotations, and highly consistent and stable VLA data pre-annotation results are output when the preset freezing conditions are met, thereby significantly reducing semantic conflicts and uncertainties in the multimodal annotation process.

[0075] In other words, this invention elevates the traditional multimodal annotation method that relies on human experience or static rules to an automated decision-making process based on multimodal information fusion, consistent reasoning, and dynamic lifecycle control. While ensuring annotation accuracy and robustness, it significantly improves the automation level and engineering applicability of multimodal data annotation, and is particularly suitable for the construction of visual-language-action multimodal data and the generation of high-quality datasets in complex task execution scenarios.

[0076] To enable those skilled in the art to better understand and implement this solution, the following detailed description of the specific solution, principles, advantages, and effects of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0077] See Figure 1 , Figure 1 A flowchart of an embodied intelligent VLA data pre-annotation method based on multimodal data provided in this embodiment of the invention is shown below. Figure 1 As shown, the following can be executed:

[0078] S101: Acquire a multimodal data stream containing visual modal data, language modal data, and action modal data, and perform time-series segmentation on the multimodal data stream according to the time order to obtain multiple time-segment data units.

[0079] Specifically, during task execution, multimodal data from different perception and interaction channels can be collected simultaneously. Visual modal data may include image data, video frame data, or target detection results; language modal data may include speech signals, text commands, or semantic parsing results; and motion modal data may include robotic arm control signals, human motion trajectories, or sequences of interactive events.

[0080] Since the multimodal data mentioned above has a natural continuity in the time dimension, the multimodal data stream can be processed by time-series segmentation according to the preset time granularity or event triggering rules, so as to obtain multiple independent time-series data units arranged in time order.

[0081] It should be noted that data units in different time periods may correspond to different task stages, subtasks, or operation segments, and this embodiment does not impose any restrictions on this.

[0082] In one embodiment, see Figure 2 The illustrated embodiment of the present invention provides a flowchart for determining multiple time-segment data units, such as... Figure 2 As shown, the confirmation steps include:

[0083] S201 performs time synchronization and alignment processing on the multimodal data stream composed of visual modal data, language modal data, and action modal data.

[0084] Specifically, in actual data acquisition or operation, visual modal data (such as video frame sequences), language modal data (such as speech signals or text transcription information), and motion modal data (such as human posture, mechanical motion parameters, or motion data collected by sensors) usually come from different sources, and their sampling frequency, timestamp accuracy, and data length may all differ.

[0085] Therefore, before performing joint analysis on multimodal data, it is necessary to perform time synchronization and alignment of each modality to construct a unified multimodal data stream.

[0086] In this embodiment, time synchronization and alignment processing may include: adding or correcting a unified timestamp to each modal data; performing interpolation, downsampling or resampling operations on modal data with different sampling frequencies; and performing time offset correction on modal data with time drift.

[0087] By using the above method, it is possible to ensure that there is a one-to-one correspondence between different modal data at the same point in time or within the same time interval, thereby forming a multimodal data stream arranged along the time axis.

[0088] It should be noted that the above time synchronization and alignment methods are only illustrative examples, and the embodiments of the present invention are not limited thereto, as long as the consistency of multimodal data in the time dimension can be achieved.

[0089] S202, along the time sequence of the multimodal data stream, based on the action change information in the action modality data, determine the boundaries of candidate time periods segment by segment.

[0090] Specifically, in the multimodal data stream after time synchronization is completed, action modal data can usually directly reflect the occurrence and changes of behavior or operation, such as the start of action, the end of action, or the switching of action state.

[0091] Therefore, in this embodiment, the action modal data can be analyzed along the time sequence of the multimodal data stream to detect action change points, thereby initially determining the candidate boundaries of the time period.

[0092] As a non-limiting example, action change information may include: the magnitude of the action exceeds a preset threshold; the action type changes; the action state changes from static to dynamic or from dynamic to static.

[0093] When any of the above-mentioned action changes are detected, the corresponding time point can be marked as a candidate time period boundary. By analyzing the action modal data segment by segment, multiple candidate time period boundaries can be obtained, thereby initially dividing the continuous multimodal data stream into several candidate time periods.

[0094] S203, combining scene change information in visual modality data and semantic boundary information in language modality data, corrects the boundaries of candidate time periods.

[0095] Specifically, candidate time period boundaries determined solely based on action modality data may be inaccurate or overly coarse. To improve the accuracy and semantic consistency of time period segmentation, this embodiment further introduces visual modality data and language modality data to refine the candidate time period boundaries.

[0096] The scene change information in the visual modal data may include, but is not limited to: significant changes in the content of the image; the appearance or disappearance of key targets or subjects; and abrupt changes in the distribution of visual features.

[0097] Semantic boundary information in language modality data may include, but is not limited to: the end of a sentence or turn; the switching of semantic topics; and semantic delimiters corresponding to speech pauses or text punctuation marks.

[0098] In this embodiment, the candidate time period boundary determined by the action modality data can be matched and fused with the change points in the visual modality and the language modality. When the change information in the three modalities is consistent or highly correlated in time, the corresponding candidate time period boundary is confirmed; when there is a deviation, the position of the candidate time period boundary is moved forward or backward to correct it.

[0099] S204 divides the multimodal data stream into multiple time-segment data units based on the corrected time-segment boundaries.

[0100] Specifically, after refining the candidate time period boundaries, a set of finalized time period boundaries can be obtained. Based on these time period boundaries, the multimodal data stream is segmented along the time axis to obtain multiple time period data units.

[0101] Each time period data unit contains visual modal data, language modal data, and action modal data within the corresponding time range, and the data of each modality are consistent in time and semantics.

[0102] By using the above-mentioned division method, not only can the correlation between multimodal data be preserved, but a structured data foundation can also be provided for subsequent multimodal understanding, behavior analysis, event recognition or model training.

[0103] It should be noted that the number and length of time period data units can be adjusted according to specific application scenarios, and this embodiment of the invention does not impose any restrictions on this.

[0104] Using the above example, this method achieves refined time segmentation of multimodal data streams by synchronizing and aligning visual, linguistic, and action modal data in time, determining candidate time segment boundaries based on action change information in the action modal data, and then refining these boundaries by combining scene change information from the visual modal data and semantic boundary information from the linguistic modal data. This method effectively improves the accuracy of time segment boundary determination and multimodal semantic consistency, avoiding segmentation errors caused by relying solely on a single modality. It also enhances adaptability to the variability and uncertainty of multimodal data in complex real-world scenarios, thus providing a structured and high-quality data foundation for subsequent multimodal understanding, behavior analysis, event recognition, and model training.

[0105] S102, for each time period data unit, identify the current task execution state based on visual modal data and language modal data, and generate at least one candidate annotation level set corresponding to the task execution state.

[0106] Specifically, for any data unit in any time period, the task execution status of the current time period can be comprehensively determined by performing scene understanding, target recognition, or behavior analysis on the visual modal data, and combining the instruction content, semantic intent, or contextual information in the language modal data.

[0107] After determining the task execution status, at least one set of candidate annotation levels that matches the task execution status can be generated based on a predefined annotation system or hierarchical structure.

[0108] As a non-limiting example, when the task execution state is identified as the "grabbing phase", the candidate annotation level set may include annotation options at different levels such as target selection, grabbing action, force feedback adjustment, etc.

[0109] It should be noted that the same task execution state can correspond to multiple candidate annotation level sets to reflect the different annotation granularity and semantic abstraction levels.

[0110] In this way, within each time period data unit, the task execution status is jointly identified based on visual modal data and linguistic modal data, and a candidate annotation level set corresponding to the task execution status is generated. This introduces task semantic constraints in the early stage of annotation, reduces annotation ambiguity, and improves the matching degree between multi-level annotations and actual task status.

[0111] S103, perform multimodal semantic representation on each level of annotation in the candidate annotation level set, and construct a multimodal consistency relationship graph containing visual nodes, language nodes and action nodes.

[0112] Specifically, for each level of annotation in the candidate annotation hierarchy set, its semantic feature representation in the visual modality, language modality, and action modality can be extracted, and the semantic features can be mapped to the corresponding nodes.

[0113] Based on this, a multimodal consistency graph can be constructed, in which nodes of different modalities are connected through semantic similarity, temporal correlation or causal correlation, thereby forming a cross-modal structured representation.

[0114] It should be noted that the multimodal consistency diagram can not only reflect the relationships within a single modality, but also the collaborative constraint relationships between different modalities, providing a foundation for subsequent consistency analysis.

[0115] See Figure 3 The flowchart shown in the embodiment of the present invention provides a method for constructing a multimodal consistency graph, as follows: Figure 3 As shown, it may include:

[0116] S301, for each level of annotation in the candidate annotation level set, construct a corresponding modal sub-consistency graph. The modal sub-consistency graph contains visual semantic units, linguistic semantic units, or action semantic units associated with that level of annotation.

[0117] Specifically, the candidate annotation level set can include annotations at multiple different semantic abstraction levels, such as object-level annotations, attribute-level annotations, relation-level annotations, or event-level annotations. For each level of annotation, semantic units associated with the semantics of that level of annotation can be extracted from the multimodal data, and a corresponding modality sub-consistency graph can be constructed accordingly.

[0118] In this embodiment, the modal sub-consistency graph is used to describe the semantic consistency relationship within the same modality or the same semantic level.

[0119] As a non-limiting example, for the visual modality, target regions, key points, or visual feature vectors in an image can be used as visual semantic units; for the language modality, words, phrases, or syntactic structures in text can be used as language semantic units; and for the action modality, action segments or motion features in a time series can be used as action semantic units.

[0120] It should be noted that the modal sub-consistency graphs corresponding to different levels of annotation can differ in the number of nodes, node types, and graph structure. For example, the modal sub-consistency graph corresponding to lower-level annotations can have more nodes to characterize fine-grained semantics, while the modal sub-consistency graph corresponding to higher-level annotations can have fewer nodes to characterize abstract semantics. This embodiment of the invention does not impose any limitations on this, as long as a corresponding modal sub-consistency graph can be constructed for each level of annotation.

[0121] S302, Based on the modal sub-consistency graph, a cross-modal consistency mapping graph is constructed according to the semantic mapping relationship between different modal semantic units, which is used to describe the correspondence between different modal sub-consistency graphs.

[0122] Specifically, semantic units across different modalities typically exhibit mappable or alignable relationships, such as the referential relationship between visual targets and linguistic descriptions, or the semantic correspondence between action semantics and textual instructions. Based on these semantic mapping relationships, cross-modal consistency mapping graphs can be constructed to describe the connection structure between different modal sub-consistency graphs.

[0123] In this embodiment, the nodes of the cross-modal consistency mapping graph can originate from semantic units in different modal sub-consistency graphs, and its edges are used to represent the correspondence, similarity, or constraint relationships between cross-modal semantic units. As a non-limiting example, an edge in the cross-modal consistency mapping graph can indicate that the similarity between a certain visual semantic unit and a certain linguistic semantic unit in the semantic space is higher than a preset threshold.

[0124] It should be noted that the cross-modal consistency mapping graph is not limited to covering nodes in all modal sub-consistency graphs. In some embodiments, cross-modal mapping relationships are established only for semantic units with high semantic relevance to reduce computational complexity and improve the effectiveness of consistency modeling.

[0125] S303, combine the modal sub-consistency graph with the cross-modal consistency mapping graph to form a graph set structure of multimodal consistency relationship graph, wherein the graph set structure includes at least a primary consistency graph and multiple auxiliary consistency graphs.

[0126] Specifically, by combining multiple modal sub-consistency graphs and cross-modal consistency mapping graphs, a graph set structure can be formed to comprehensively characterize multimodal consistency relationships. The graph set structure can simultaneously reflect the consistency relationships within a single modality and the consistency constraints between cross-modalities.

[0127] In this embodiment, the graph containing cross-modal mapping relationships can be used as the main consistency graph to characterize the core consistency structure among multiple modalities; while the modal sub-consistency graphs corresponding to each level can be used as auxiliary consistency graphs to supplement the description of local consistency features under different levels or different modalities.

[0128] It should be noted that the primary consistency graph and the secondary consistency graph can be associated through shared nodes, mapping edges, or index relationships. This embodiment of the invention does not limit the specific organizational form of the graph set structure, as long as it can express multimodal consistency relationships.

[0129] In one embodiment, each graph node in the graph set structure of the multimodal consistency graph is categorized according to the semantic unit type it represents:

[0130] A visual node represents the image features or object recognition results extracted from a visual modality.

[0131] Language nodes represent lexical or semantic information extracted from a language modality.

[0132] Action nodes represent motion or behavioral information extracted from action modalities.

[0133] Specifically, the multimodal consistency graph is used to perform unified semantic modeling on data from different modalities, so as to express the correspondence between "what was seen," "what was said," and "what was done" in the same graph structure. To achieve the above modeling, this application abstracts graph nodes as basic elements that carry semantic units, and classifies nodes according to the source modality and semantic attributes of the semantic units. This allows for the use of differentiated feature representation methods and constraint rules for different types of nodes when constructing cross-modal association edges, calculating consistency scores, or performing inference.

[0134] In some embodiments, a visual node corresponds to a semantic unit of a visual modality, which may include, but is not limited to, image frames, video clips, or multi-view images. The node features of a visual node may be composed of feature vectors output by a visual encoder, which may be the backbone network of a convolutional neural network, a visual Transformer, or an object detection network. Simultaneously, visual nodes may also be associated with structured information such as object categories, bounding boxes, keypoints, segmentation masks, color and texture attributes, and their confidence scores.

[0135] In some embodiments, a language node corresponds to a semantic unit of a language modality, which may include, but is not limited to, text, transcribed text after speech recognition, or instruction statements. A language node can represent semantic fragments such as words, phrases, entities, event descriptions, or relation descriptions. The node features of a language node can be composed of embedding vectors output by a language encoder, which can be a Transformer-type pre-trained model or other sequence modeling networks. Language nodes may also include information such as part-of-speech tags, dependency relations, named entity types, semantic role annotation results, and context window indexes.

[0136] In some embodiments, an action node corresponds to a semantic unit of an action modality, which may include, but is not limited to, human action sequences, mechanical component motion trajectories, equipment workflow logs, control command sequences, or behavioral information reflected by sensor timing data. The node features of an action node may be composed of feature representations output by an action recognition network, a timing modeling network, or a state machine parsing module; action nodes may also include attributes such as action start and end times, duration, participating objects, action direction / velocity, key moment states, and action confidence.

[0137] In some embodiments, based on the above node type classification, the multimodal consistency graph can further construct cross-modal association edges, such as "reference / description" relationship edges between visual nodes and language nodes, "action / influence" relationship edges between action nodes and visual nodes, and "command / explanation" relationship edges between action nodes and language nodes. By pre-defining node types and configuring corresponding feature extraction and constraint rules for different types of nodes, the stability of cross-modal alignment can be improved, and the inconsistency risk caused by semantic ambiguity or modal noise can be reduced, thereby providing structured support for subsequent consistency judgment, anomaly detection, event tracking, or multimodal retrieval.

[0138] S304 assigns an initial consistency status identifier to each graph node in the graph set structure. The consistency status identifier is used to characterize the consistency confidence level of the node under the current candidate annotation conditions.

[0139] Specifically, after constructing the multimodal consistency graph, an initial consistency status identifier can be assigned to each node in the graph set structure. The consistency status identifier can be used to reflect the credibility or consistency confidence level of the node under the current candidate annotation conditions.

[0140] In this embodiment, the consistency status identifier can be represented in numerical form, vector form, or discrete status label form. As a non-limiting example, the consistency status identifier can be a continuous value between 0 and 1, used to represent the consistency strength between the node and the candidate label.

[0141] It should be noted that the allocation method of the initial consistency state identifier can be determined based on prior knowledge, statistical features or the output results of the pre-trained model, and the embodiments of the present invention do not impose any restrictions on this.

[0142] S305, based on the hierarchical type and task execution status of the candidate annotations, group and initialize the node states in the graph set structure so that nodes in different levels or different states have different consistency state starting conditions.

[0143] Specifically, different levels of annotation differ in semantic abstraction and task importance, and the states of different nodes may also differ during task execution. Therefore, nodes in the graph set structure can be grouped according to the level type of the candidate annotations and the current task execution state, and different consistency state initiation conditions can be set for nodes in different groups.

[0144] In this embodiment, as a non-limiting example, high-level semantic nodes can be initialized with a higher consistency confidence level, while low-level or noisy semantic nodes can be initialized with a lower consistency confidence level; or, depending on the different stages of task execution, verified nodes and nodes to be verified can be assigned different initial state identifiers.

[0145] It should be noted that by initializing node states in groups, the convergence speed and stability of subsequent consistency inference or optimization processes can be improved, enabling the multimodal consistency graph to more accurately reflect the authenticity and reliability of candidate labels.

[0146] This invention performs multimodal semantic representation on the annotations at each level in the candidate annotation hierarchy set and constructs a multimodal consistency graph. This enables visual, linguistic, and action modalities to be associated and modeled within a unified graph structure, thereby explicitly characterizing the consistency constraints between different modalities and improving the overall consistency and interpretability of the multimodal annotation results.

[0147] S104. Perform a consistency propagation operation in the multimodal consistency graph to obtain the multimodal consistency constraint relationships between nodes and detect conflicting node combinations.

[0148] Specifically, consistency propagation or constraint propagation algorithms can be executed on multimodal consistency graphs, enabling semantic information and consistency constraints between nodes to be diffused and updated within the graph structure.

[0149] By analyzing the node states after propagation, it is possible to identify node combinations that have semantic, logical, or temporal conflicts between different modalities or different levels of annotation.

[0150] As a non-restrictive example, when a visual node indicates "action completed" while the action node is still in the "in execution" state, it can be determined that there is a consistency conflict in the combination of nodes.

[0151] See Figure 4 The flowchart shown in the embodiment of the present invention provides a method for determining multimodal consistency constraints, as follows: Figure 4 As shown, it may include:

[0152] S401, in the graph set structure of the multimodal consistency relationship graph, multiple consistency propagation stages are executed sequentially according to the preset propagation stage order, and each propagation stage corresponds to a different consistency propagation goal.

[0153] Specifically, a multimodal consistency graph can be composed of multiple modal sub-consistency graphs, and the modalities can include, but are not limited to, text modalities, image modalities, speech modalities, or structured data modalities. To achieve consistency constraints and collaborative optimization among multimodal information, the consistency propagation process is divided into multiple sequentially executed consistency propagation stages.

[0154] In this embodiment, each propagation stage corresponds to a preset consistency propagation goal, such as a propagation stage to strengthen semantic consistency, a propagation stage to constrain structural consistency, or a consistency correction propagation stage to eliminate modal bias. By executing the stages sequentially according to the preset order, consistency information can gradually accumulate and converge in the multimodal relationship graph, thereby avoiding the instability caused by one-time propagation.

[0155] It should be noted that the number of propagation stages and their corresponding propagation targets can be configured according to the actual application scenario. This embodiment of the invention does not limit this, as long as multi-stage consistent propagation can be achieved.

[0156] S402, in each propagation phase, perform intra-layer consistency propagation and cross-layer consistency propagation according to the consistency constraint type. Figure 1 Consistency propagation, where intra-layer consistency propagation is used to update the node state within the same modality sub-consistency graph, and cross-layer consistency propagation... Figure 1 Consistency propagation is used to pass consistency impact information between the primary consistency graph and the secondary consistency graph.

[0157] Specifically, within each propagation phase, the node state update method is differentiated based on the consistency constraint type corresponding to the current propagation phase. For nodes belonging to the same modal sub-consistency graph, the state is updated within that modal sub-consistency graph through the intra-layer consistency propagation mechanism to ensure that the consistency constraints between nodes within the same modality are satisfied.

[0158] At the same time, in order to achieve consistent coordination between different modalities, cross-modal operations are also performed. Figure 1 Sexually transmitted infections. (Cross-border) Figure 1 Consistency propagation is used to pass consistency impact information between the primary consistency graph and at least one secondary consistency graph, so that changes in the state of a node in one modality can have a constraint or correction effect on associated nodes in other modalities.

[0159] By propagating consistency within layers and across layers Figure 1 By combining consistent propagation, we can achieve consistent and coordinated updates among multiple modalities while maintaining the stability of a single modality.

[0160] In some embodiments, intra-layer consistency propagation is used to update the consistency state of nodes within the same modality, and cross-modal consistency propagation is used to pass consistency between nodes in different modalities. The propagation is executed in a preset priority order, and the propagation priority between different modalities is dynamically adjusted according to the initial state of the nodes.

[0161] S403 records the propagation path information corresponding to the changes in node state during the consistency propagation process, so as to form the node state evolution trajectory.

[0162] Specifically, consistency propagation within the execution layer and across... Figure 1 During the propagation process, the state changes of each node at different propagation stages and steps are recorded. Simultaneously, path information such as the source node, propagation stage, and propagation direction that triggers the state changes is recorded, thereby constructing the state evolution trajectory of the nodes as they evolve through the propagation process.

[0163] In this embodiment, the node state evolution trajectory can be used to characterize the dynamic change process of the node in the multi-stage consensus propagation, providing basic data support for subsequent analysis of the stability or conflict of node behavior.

[0164] During the propagation process, the initial consistency state of the nodes and historical propagation results are used to determine whether the propagation conditions are met. If the consistency propagation conditions are met, the next propagation stage continues; if not, propagation is paused, awaiting more samples or updated conditions.

[0165] S404 After completing at least one full propagation cycle, the evolution trajectory of the node state is analyzed to identify whether there is a conflict evolution pattern in which the node state exhibits periodic oscillations, continuous degradation, or branching in multiple propagation stages.

[0166] Specifically, a complete propagation cycle can include executing all consensus propagation stages in a preset order. After completing at least one complete propagation cycle, the evolution trajectory of the node state is analyzed and processed.

[0167] During the analysis, it is possible to identify whether there is a periodic oscillation phenomenon in which the node state changes back and forth repeatedly between different propagation stages, or a continuous degradation phenomenon in which the node state continues to change towards a lower consistency direction as propagation progresses, or a branching phenomenon in which the same node forms multiple contradictory state branches under the influence of different propagation paths.

[0168] Any of the above phenomena can be regarded as a conflict evolution mode that occurs during the consensus propagation process of a node, indicating that the node or combination of nodes is difficult to satisfy multiple propagation goals simultaneously under multimodal consensus constraints.

[0169] It should be noted that the consistency status changes of graph nodes are used to determine whether a node has reached the conflict threshold. The conflict threshold is determined based on the historical changes of the node's consistency status, the correlation between modalities, and the credibility of the candidate annotations.

[0170] S405: When a conflict evolution pattern is detected, the corresponding node combination is determined to be a conflict node combination.

[0171] Specifically, when the analysis of the node state evolution trajectory detects conflict evolution patterns such as periodic oscillations, continuous degradation, or branching among nodes, the relevant node combinations that trigger such conflict evolution patterns are marked as conflict node combinations.

[0172] In this embodiment, the combination of conflicting nodes can be used as input for subsequent consistency correction, node weight adjustment, propagation strategy reconfiguration, or conflict resolution, thereby improving the overall stability and convergence of the multimodal consistency propagation process.

[0173] It should be noted that the determination of conflict node combinations can be carried out in various ways, such as threshold judgment, rule judgment or learning model judgment, depending on the specific implementation. This embodiment of the invention does not limit this.

[0174] This invention performs consistency propagation operations in a multimodal consistency graph and automatically detects conflicting node combinations. It can effectively discover potential semantic, temporal, or logical conflicts in multimodal annotations and avoid erroneous annotation problems caused by deviations in single-modal information.

[0175] It should be noted that by performing multimodal semantic representation on the annotations of each level in the candidate annotation level set, a multimodal consistency graph containing visual nodes, language nodes and action nodes is constructed. The multimodal consistency graph is then organized into a graph set structure including a main consistency graph and multiple auxiliary consistency graphs. This enables the unified modeling of intra-level associations and cross-modal mapping relationships between different levels and different modal semantic units, and assigns corresponding consistency status identifiers to each graph node.

[0176] Based on the aforementioned multimodal consistency graph and its node consistency state initialization, a phased consistency propagation operation is further performed in the graph set structure. This is achieved through intra-layer consistency propagation within the same modal sub-consistency graph and cross-consistency propagation between the main consistency graph and auxiliary consistency graphs. Figure 1 Consistent propagation causes the consistent state of nodes to evolve along different propagation paths, and identifies node combinations with consistency conflicts between multimodal semantics based on the node state evolution trajectory.

[0177] S105, For the detected combination of conflicting nodes, construct counterfactual multimodal samples associated with the candidate labels based on the corresponding candidate label level.

[0178] Specifically, for each set of conflicting nodes, we can backtrack to the candidate labeling level associated with it, and while keeping some modalities unchanged, we can perform counterfactual perturbation on another modality to generate counterfactual multimodal samples.

[0179] Counterfactual multimodal samples are used to simulate the multimodal performance of candidate labels under different assumptions, in order to test the rationality of the candidate labels under different modal combinations.

[0180] It should be noted that the construction methods of counterfactual multimodal samples may include modality replacement, semantic rewriting, or action sequence rearrangement, etc., and this embodiment does not impose specific limitations on them.

[0181] In one embodiment, step S105 may include:

[0182] S1051, Based on the combination of conflict nodes, candidate labels with high correlation to conflict are determined.

[0183] Specifically, when analyzing multimodal data, conflict node combinations can be constructed, where each node corresponds to a feature element or semantic unit in a different modality (e.g., keywords in text, region features in images, time segments in audio). When inconsistencies exist between different modalities in semantics, prediction results, or feature contributions, the corresponding nodes can be identified as constituting a conflict node combination.

[0184] After identifying the conflict node combination, the relationship between the conflict node combination and each individual label can be further analyzed to determine candidate labels with a high degree of correlation with the conflict. The degree of correlation can be determined in various ways, including but not limited to:

[0185] The contribution of conflict nodes to the labeled prediction results during model inference; the degree of similarity, correlation or causal influence between conflict nodes and labels; and the changes in the stability of labeled prediction when conflict nodes appear or change.

[0186] Using the above method, a set of candidate labels highly relevant to the combination of conflicting nodes can be selected from the original label set, providing target objects for the subsequent construction of counterfactual samples. It should be noted that the number of candidate labels can be one or more, and this embodiment of the invention does not limit this.

[0187] S1052, for the determined candidate labels, replace, perturb, or mask at least one modality of data.

[0188] Specifically, after candidate labels are determined, at least one modality of the original multimodal data can be processed using the candidate labels as constraints to weaken or eliminate the influence of conflicting nodes on the candidate labels. Processing methods may include, but are not limited to, replacement, perturbation, or masking.

[0189] As a non-restrictive example:

[0190] When the modal data is text data, synonym replacement, random perturbation, or direct masking can be performed on the words, phrases, or sentences corresponding to conflicting nodes.

[0191] When the modal data is image data, pixel replacement, noise perturbation, or region occlusion can be performed on the image region corresponding to the conflict node.

[0192] When the modal data is audio data, signal perturbation, replacement, or muting can be performed on the corresponding time period.

[0193] It should be noted that in some embodiments, only single-modal data is processed; while in other embodiments, multiple modal data can be processed simultaneously to enhance the diversity and effectiveness of counterfactual samples.

[0194] S1053, Construct counterfactual multimodal samples based on the processed modal data.

[0195] Specifically, after replacing, perturbing, or masking at least one modality of data, the processed modality data can be recombine with the remaining unprocessed modality data to construct counterfactual multimodal samples. The counterfactual multimodal samples maintain the same structural form as the original multimodal samples, but differ in semantic features or key nodes.

[0196] By inputting counterfactual multimodal samples into the target model, the changes in the model's prediction results relative to the original multimodal samples can be observed, thereby assessing the influence of conflict node combinations and candidate labels on the model's decision-making. It should be noted that counterfactual multimodal samples constructed using different processing methods and different candidate labels may produce the same or different prediction results; this embodiment of the invention does not limit this.

[0197] Through the above steps, counterfactual analysis of multimodal conflict problems can be achieved, providing a valid basis for model robustness evaluation, annotation correction, or model optimization.

[0198] In other words, this invention constructs counterfactual multimodal samples associated with candidate labels for detected conflict node combinations, and introduces them into the consistency relationship graph to perform multi-round consistency backtracking verification, thereby performing robust verification of candidate labels under different modal combinations and assumptions, significantly improving the consistency stability evaluation capability of candidate labels.

[0199] S106 introduces counterfactual multimodal samples into the multimodal consistency graph and performs multiple rounds of consistency backtracking verification to determine the consistency stability of candidate labels under different modal combinations.

[0200] Specifically, the constructed counterfactual multimodal samples can be used as new nodes or constraints, introduced into the original multimodal consistency graph, and a multi-round consistency backtracking verification process can be performed.

[0201] By comparing the consistency performance of candidate labels under the original multimodal samples and counterfactual multimodal samples, the consistency stability of the candidate labels under different modal combinations can be evaluated.

[0202] It should be noted that higher consistency stability indicates stronger robustness of the candidate annotation to modal perturbations, making it more suitable as the final annotation result.

[0203] In one embodiment, step S106 may include:

[0204] S1061 introduces counterfactual multimodal samples into the multimodal consistency graph.

[0205] Specifically, after completing feature extraction and cross-modal alignment of the original multimodal samples, a multimodal consistency graph can be constructed to characterize the semantic consistency relationships between different modalities and within the same modality. Nodes in the multimodal consistency graph represent multimodal samples or their feature representations, while edges represent the semantic consistency strength or association between samples.

[0206] Building upon this foundation, counterfactual multimodal samples are introduced. Counterfactual multimodal samples are samples generated by intervening, replacing, or perturbing at least one modality while preserving some modal information. They are used to characterize counterfactual situations where "if modal information changes, does the overall semantics remain consistent?" By introducing counterfactual multimodal samples as new nodes into the multimodal consistency graph, and establishing corresponding edge connections based on their semantic similarity, consistency score, or preset rules with the original multimodal samples, the structure of the multimodal consistency graph is expanded.

[0207] It should be noted that introducing counterfactual multimodal samples does not change the basic structure of the existing nodes and edges in the original multimodal consistency graph. Instead, it enhances the graph structure so that it can simultaneously reflect the consistency differences between real samples and counterfactual samples in the multimodal semantic space.

[0208] S1062 performs multi-round consistency backtracking verification on the multimodal consistency graph after introducing counterfactual multimodal samples.

[0209] Specifically, after introducing the counterfactual multimodal samples, a consistency backtracking verification operation is performed on the updated multimodal consistency graph. Consistency backtracking verification refers to starting from the target node in the graph (e.g., the multimodal sample node corresponding to the candidate label), tracing its associated nodes forward or backward along the consistency relationship edge, and comprehensively analyzing the consistency changes between nodes under different tracing paths.

[0210] In this embodiment, multi-round consistency backtracking verification is achieved by repeatedly executing the aforementioned backtracking process. Each round of backtracking verification can traverse and verify the multimodal consistency graph based on different backtracking starting points, backtracking depths, or edge weight update strategies. Through multiple rounds of backtracking, the randomness caused by a single verification can be gradually eliminated, making the consistency verification results more robust.

[0211] It should be noted that during the backtracking verification process, if the counterfactual multimodal samples and the original multimodal samples show high consistency across multiple backtracking paths, it indicates that the consistency relationship is highly robust to modal perturbations; conversely, it indicates that the consistency relationship is more sensitive to modal changes.

[0212] S1063, evaluate the consistency and stability of candidate annotations based on the results of multiple rounds of backtracking verification.

[0213] Specifically, after completing multiple rounds of consistency backtracking verification, the verification results obtained from each round of backtracking verification can be summarized, and the consistency stability of candidate labels can be evaluated accordingly. Candidate labels can be semantic labels, category labels, or attribute labels generated for multimodal samples.

[0214] In this embodiment, the consistency stability index can be calculated based on the consistency score changes of candidate labels under different backtracking paths and different counterfactual sample conditions during multiple rounds of backtracking verification. For example, when a candidate label consistently maintains a high consistency score during multiple rounds of backtracking verification, and the score fluctuation is within a preset threshold range, the candidate label can be determined to have high consistency stability; otherwise, its consistency stability is determined to be low.

[0215] It should be noted that, based on the consistency stability assessment results, candidate labels can be further screened, confirmed, or corrected, thereby improving the reliability and robustness of multimodal labeling results. This embodiment of the invention does not limit this, as long as the consistency stability of candidate labels can be assessed based on multi-round backtracking verification results.

[0216] S107. Based on the consistency and stability results, perform annotation lifecycle state transition control on candidate annotations to determine the current annotation state of candidate annotations.

[0217] Specifically, a label lifecycle state can be predefined for each candidate label, including but not limited to "initial state", "verification state", "confirmation state" and "frozen state".

[0218] Based on the consistency and stability results of candidate labels, corresponding state transition rules can be triggered, enabling candidate labels to dynamically evolve during their lifecycle.

[0219] It should be noted that different consistency stability intervals may correspond to different state transition paths, and this embodiment does not impose any restrictions on this.

[0220] In one embodiment, step S107 may include:

[0221] S1071, based on the consistency and stability results, the confidence level of the candidate annotation is calculated.

[0222] Specifically, after completing the consistency and stability analysis of candidate annotations, consistency and stability results that characterize the reliability of candidate annotations can be obtained. The consistency and stability results can reflect the stability and consistency of candidate annotations under different time dimensions, different annotation subjects, or different annotation sample conditions.

[0223] In this embodiment, the corresponding trust level can be calculated based on the consistency stability result according to a preset calculation rule or mapping relationship. As a non-limiting example, the consistency stability result may include a consistency score, a stability score, or a weighted result of the two, and the trust level can be obtained by normalizing, segmenting, or classifying the above scores.

[0224] For example, when candidate labels remain highly consistent and show little variation across multiple rounds of labeling results, a higher confidence level can be calculated; conversely, when candidate labels show significant differences or frequent changes across different labeling results, their corresponding confidence level is relatively low.

[0225] It should be noted that the embodiments of the present invention do not limit the specific form of the consistency stability results, nor do they limit the specific calculation method of the credibility level. As long as the credibility of the candidate labels can be quantified or graded based on the consistency stability results, they should be considered to fall within the protection scope of the present invention.

[0226] S1072, based on the matching results of the trust level and multiple preset trust level intervals or thresholds, controls the candidate annotation to perform state transitions between multiple preset annotation lifecycle states.

[0227] Specifically, the system can pre-set multiple confidence level ranges or confidence level thresholds, and configure a labeling lifecycle state for each confidence level range or threshold. The labeling lifecycle state can be used to characterize the current stage or processing strategy of candidate labels in the entire labeling process.

[0228] In this embodiment, the confidence level calculated in step S1071 is matched with multiple preset confidence level intervals or thresholds. Based on the matching results, the lifecycle state of the target label corresponding to the candidate label is determined, and the candidate label is controlled to perform state transition between different label lifecycle states.

[0229] As a non-limiting example, the label lifecycle status can include: pending confirmation status, preliminary trust status, high trust status, need for review status, or invalid status. When the trust level of a candidate label falls into the high trust level range, the candidate label can be transferred from the preliminary trust status to the high trust status; when the trust level is lower than a preset threshold, the candidate label can be transferred to the need for review status or invalid status.

[0230] It should be noted that the state transition of candidate annotations can be unidirectional, bidirectional, or multidirectional, meaning that candidate annotations can be dynamically adjusted based on changes in trust level between different lifecycle states. Furthermore, the specific types and quantities of annotation lifecycle states, as well as the corresponding trust level ranges or thresholds, can be flexibly set according to the actual application scenario; this embodiment of the invention does not impose any limitations on this.

[0231] This invention introduces a labeling lifecycle state transition control mechanism to dynamically adjust the labeling state of candidate labels based on the consistency and stability results, making the process from initial generation to final freezing of labeling results controllable and traceable, thereby effectively avoiding the problems of premature or incorrect freezing of labels.

[0232] S108: When the annotation status of the candidate annotation meets the preset freezing conditions, the candidate annotation is determined as the pre-annotation result of the VLA data and output.

[0233] Specifically, when a candidate label has undergone multiple rounds of consistency verification and its labeling status reaches the preset freezing conditions, such as consistency stability being higher than the threshold and remaining stable over multiple time periods, the candidate label can be determined as the final pre-labeling result.

[0234] The pre-annotated results can serve as high-quality annotation outputs for VLA (Vision-Language-Action) data, which can be used for subsequent model training, evaluation, or manual review processes.

[0235] It should be noted that a freezing condition refers to a condition in which certain operations or states in the system must meet specific criteria or conditions before freezing or final confirmation is allowed. This usually means that the annotations, data, or states have reached preset stability and reliability standards, and only on this basis is the system allowed to proceed to the next step or final stage.

[0236] In this embodiment, the freezing condition refers to a key state transition point in the candidate annotation's lifecycle, namely, the annotation is confirmed to be correct and stable, and is finally finalized as a pre-annotation result. To make the freezing condition clearer and more concrete, it can be defined as:

[0237] Freezing conditions refer to the conditions that a candidate label meets during the labeling process, such as consistency, stability, or verification, allowing the label to transition from a provisional state to a final confirmed state and be output as a valid pre-labeling result. Only when the freezing conditions are met are the labels considered reliable, conflict-free, and stable, thus preventing mislabeled or unstable labels from being output incorrectly.

[0238] Thus, when candidate annotations meet the preset freezing conditions, they are determined as the pre-annotation results of VLA data, making the obtained multimodal pre-annotated data more consistent, stable and reliable, which can effectively reduce the cost of subsequent manual annotation or verification, and improve the efficiency and quality of VLA data construction.

[0239] In summary, the embodiments of the present invention achieve automation, consistency enhancement, and stability assurance of the VLA data annotation process through multimodal consistency modeling, counterfactual verification, and annotation lifecycle control mechanisms. It is particularly suitable for visual-language-action multimodal data annotation and management in complex task scenarios.

[0240] To better understand and illustrate this, we will use an embodied intelligent VLA data pre-labeling implementation scheme based on indoor capture tasks as an example, but it is not limited to this.

[0241] This implementation plan takes indoor desktop grasping tasks as the application scenario. The task object is an embodied intelligent robot that completes a continuous operation process of "identifying the target - moving the robotic arm - grasping - placing" according to voice commands in a desktop environment.

[0242] The system includes: an RGB-D camera for acquiring visual modal data; a voice acquisition module for acquiring human commands and transcribing them into text-based language modal data; a robot controller for outputting the pose changes, joint angles, and grasping actions of the robotic arm to form motion modal data; and a pre-annotation processing module for executing the multimodal VLA data pre-annotation method of this implementation scheme.

[0243] During a complete task execution, the system simultaneously acquires visual modal data, language modal data, and motion modal data, forming a multimodal data stream.

[0244] Specifically: the three modalities of data are timestamped and mapped to the same time axis; the rate of change of the robotic arm's end-effector pose in the motion modality data is used as the main segmentation criterion, and when the pose change exceeds a preset threshold, it is marked as a candidate time segment boundary; the candidate time segment boundaries are corrected by combining the change of target object category in the visual modality (such as from "no target" to "cup") and the end position of the instruction semantic segment in the language modality; finally, the multimodal data stream is divided into multiple continuous time segment data units, and each time segment corresponds to a sub-task stage.

[0245] For each time period data unit, task execution status is identified based on visual and linguistic modalities.

[0246] For example, when a target object is detected in the visual modality and the language modality contains the keyword "pick up the cup", the current state is identified as "target recognition completed"; when the action modality shows that the robotic arm is approaching the target object, the current state is identified as "performing a grasping action".

[0247] Based on the identified task execution status, a set of candidate annotation levels is generated, including but not limited to: task-level annotations (such as "grab task" and "cup"); behavior-level annotations (such as "approaching target" and "closing gripper"); and semantic-level annotations (such as "cup" and "desktop").

[0248] For each candidate annotation level, a corresponding modality sub-consistency graph is constructed, specifically including:

[0249] Visual nodes: represent the object category and location information extracted by the object detection model; Language nodes: represent the keywords and semantic roles in language commands; Action nodes: represent the robotic arm action type and motion parameters.

[0250] Based on the modal sub-consistency graph, a cross-modal consistency mapping graph is constructed according to semantic mapping relationships such as "language keywords - visual target categories".

[0251] The modal sub-consistency graph is combined with the cross-modal consistency mapping graph to form a multimodal consistency relationship graph set structure containing the main consistency graph and multiple auxiliary consistency graphs, and an initial consistency state identifier is assigned to each node.

[0252] In the multimodal consensus graph, consensus propagation operations are performed in a preset order. The specific process includes:

[0253] First, perform intra-layer consistency propagation within the same modal sub-consistency graph to update the consistency state of nodes; then, perform cross-consistency propagation between the primary consistency graph and the secondary consistency graph. Figure 1 Consistent propagation; recording the trajectory information of how the node's consistency state changes as the propagation stage progresses.

[0254] After completing at least one propagation cycle, analyze the node state evolution trajectory: if a visual node's consistency state continuously declines in multiple propagation stages and contradicts language nodes and action nodes, it is determined to be a conflict evolution mode; the corresponding node combination is identified as a conflict node combination.

[0255] For the detected combination of conflicting nodes, the candidate labels with the highest correlation are determined.

[0256] Taking the candidate label "grab a cup" as an example: occlude or replace the target object in the visual modality; or mask the keyword "cup" in the linguistic modality; construct the corresponding counterfactual multimodal sample.

[0257] Counterfactual multimodal samples are introduced into the multimodal consistency graph, and multiple rounds of consistency backtracking verification are performed. The consistency stability of candidate labels under different modal combinations is evaluated based on the backtracking results.

[0258] Based on the consistency and stability results, the credibility level of candidate annotations is calculated. The credibility level is then matched with a preset credibility level range to control the transition of candidate annotations between the following lifecycle states: initial generation state; verification state; stable confirmation state; frozen state.

[0259] When candidate annotations enter the frozen state, they are identified as pre-annotated results of VLA data for the corresponding time period and output for subsequent manual verification or model training.

[0260] The above describes in detail the embodied intelligent VLA data pre-annotation method based on multimodal data through some embodiments. In order to enable those skilled in the art to better understand and implement it, the corresponding device is also described in detail below through some embodiments.

[0261] See Figure 5The schematic diagram shown in Figure 5 illustrates the structure of an embodied intelligent VLA data pre-annotation device based on multimodal data provided in this embodiment of the invention. The embodied intelligent VLA data pre-annotation device 500 based on multimodal data may include:

[0262] The acquisition unit 501 is used to acquire a multimodal data stream containing visual modal data, language modal data and action modal data, and to perform time-series segmentation on the multimodal data stream according to the time order to obtain multiple time-segment data units;

[0263] The generation unit 502 is used to identify the current task execution state based on visual modal data and language modal data for each time period data unit, and generate at least one set of candidate annotation levels corresponding to the task execution state.

[0264] The construction unit 503 is used to perform multimodal semantic representation on each level of annotation in the candidate annotation level set, and to construct a multimodal consistency relationship graph containing visual nodes, language nodes and action nodes.

[0265] The detection unit 504 is used to perform a consistency propagation operation in the multimodal consistency relationship graph to obtain the multimodal consistency constraint relationship between each node and to detect conflicting node combinations.

[0266] Construction unit 505 is used to construct counterfactual multimodal samples associated with candidate labels based on the corresponding candidate label level for detected conflict node combinations.

[0267] The processing unit 506 is used to introduce counterfactual multimodal samples into the multimodal consistency relationship graph, perform multiple rounds of consistency backtracking verification to determine the consistency stability of candidate labels under different modal combinations, and perform label lifecycle state transition control on candidate labels based on the consistency stability results to determine the current labeling state of candidate labels;

[0268] The output unit 507 is used to determine the candidate annotation as the pre-annotation result of VLA data and output it when the annotation status of the candidate annotation meets the preset freezing conditions.

[0269] It is understandable that the above division of units is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the above units can be implemented by the processor calling software.

[0270] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0271] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications, substitutions, and improvements without departing from the concept of the present invention, and these should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of this invention should be determined by the claims.

Claims

1. A method for pre-labeling embodied intelligent VLA data based on multimodal data, characterized in that, include: A multimodal data stream containing visual modal data, language modal data, and action modal data is acquired, and the multimodal data stream is segmented temporally according to the time order to obtain multiple time period data units; For each time period data unit, the current task execution state is identified based on the visual modality data and the language modality data, and at least one set of candidate annotation levels corresponding to the task execution state is generated according to the task execution state. Multimodal semantic representation is performed on each level of the candidate annotation hierarchy set, and a multimodal consistency relationship graph containing visual nodes, language nodes, and action nodes is constructed. A consistency propagation operation is performed in the multimodal consistency relationship graph to obtain the multimodal consistency constraint relationship between each node and to detect conflicting node combinations. For the detected combinations of conflicting nodes, based on the corresponding candidate annotation level, a counterfactual multimodal sample associated with the candidate annotation is constructed; The counterfactual multimodal samples are introduced into the multimodal consistency graph, and multiple rounds of consistency backtracking verification are performed to determine the consistency stability of the candidate labels under different modal combinations. Based on the consistency and stability results, annotation lifecycle state transition control is performed on the candidate annotations to determine the current annotation state of the candidate annotations; When the annotation status of the candidate annotations meets the preset freezing conditions, the candidate annotations are determined as the pre-annotation results of VLA data and output.

2. The pre-annotation method according to claim 1, characterized in that, The obtained multiple time-time data units include: Time synchronization and alignment processing is performed on the multimodal data stream composed of visual modal data, language modal data, and action modal data; Following the temporal order of the multimodal data stream, candidate time period boundaries are determined segment by segment based on the action change information in the action modality data; By combining scene change information in the visual modality data and semantic boundary information in the language modality data, the candidate time period boundary is corrected; Based on the corrected time period boundaries, the multimodal data stream is divided into multiple time period data units.

3. The pre-annotation method according to claim 1, characterized in that, The step of performing multimodal semantic representation on each level of the candidate annotation hierarchy set and constructing a multimodal consistency graph containing visual nodes, language nodes, and action nodes includes: For each level of annotation in the candidate annotation level set, a corresponding modality sub-consistency graph is constructed. The modality sub-consistency graph contains visual semantic units, linguistic semantic units, or action semantic units associated with the annotation at that level. Based on the modal sub-consistency graph, a cross-modal consistency mapping graph is constructed according to the semantic mapping relationship between different modal semantic units to describe the correspondence between different modal sub-consistency graphs; The modal sub-consistency graph is combined with the cross-modal consistency mapping graph to form a graph set structure of multimodal consistency relationship graph, wherein the graph set structure includes at least a primary consistency graph and multiple auxiliary consistency graphs; An initial consistency status identifier is assigned to each graph node in the graph set structure. The consistency status identifier is used to characterize the consistency confidence level of the node under the current candidate annotation conditions. Based on the hierarchical type and task execution status of the candidate annotations, the node states in the graph set structure are grouped and initialized so that nodes at different levels or in different states have different consistency state starting conditions.

4. The pre-annotation method according to claim 3, characterized in that, Each graph node in the graph set structure of the multimodal consistency graph is categorized according to the semantic unit type it represents: Visual nodes represent image features or object recognition results extracted from visual modalities; Language nodes represent lexical or semantic information extracted from language modalities; Action nodes represent motion or behavioral information extracted from action modalities.

5. The pre-annotation method according to claim 1 or 3, characterized in that, The step of performing a consistency propagation operation in the multimodal consistency graph to obtain the multimodal consistency constraint relationships between nodes and detecting conflicting node combinations includes: In the graph set structure of the multimodal consistency relationship graph, multiple consistency propagation stages are executed sequentially according to a preset propagation stage order, and each propagation stage corresponds to a different consistency propagation goal; In each propagation phase, intra-layer consistency propagation and cross-graph consistency propagation are performed according to the consistency constraint type. Intra-layer consistency propagation is used to update the node state within the same modal sub-consistency graph, while cross-graph consistency propagation is used to pass consistency impact information between the primary consistency graph and the auxiliary consistency graph. During the consistency propagation process, the propagation path information corresponding to the changes in node state is recorded to form the node state evolution trajectory; After completing at least one full propagation cycle, the evolution trajectory of the node state is analyzed to identify whether there are conflict evolution patterns in which the node state exhibits periodic oscillations, continuous degradation, or branching during multiple propagation stages. When the conflict evolution pattern is detected, the corresponding node combination is determined to be a conflict node combination.

6. The pre-annotation method according to claim 5, characterized in that, Meet one or more of the following conditions: The intra-layer consistency propagation is used to update the consistency state of nodes within the same modality, and the cross-modal consistency propagation is used to pass consistency between nodes in different modalities. The propagation is executed in a preset priority order, and the propagation priority between different modalities is dynamically adjusted according to the initial state of the nodes. During the propagation process, the initial consistency state of the nodes and the historical propagation results are used to determine whether the propagation conditions are met. When the consistency propagation condition is met, the next propagation phase continues; when the condition is not met, propagation is paused, waiting for more samples or condition updates. Based on the changes in the consistency state of graph nodes, it is determined whether a node has reached a conflict threshold; wherein, the conflict threshold is determined based on the historical changes in the consistency state of the node, the correlation between modalities, and the credibility of the candidate labels.

7. The pre-annotation method according to claim 5, characterized in that, For the detected combinations of conflicting nodes, based on the corresponding candidate annotation level, constructing counterfactual multimodal samples associated with the candidate annotations includes: Based on the combination of conflicting nodes, candidate labels with a high degree of correlation with the conflict are determined; For the identified candidate labels, at least one modality of data is replaced, perturbed, or masked; Construct counterfactual multimodal samples based on the processed modal data.

8. The pre-annotation method according to claim 6, characterized in that, Determining the consistency stability of the candidate labels under different modality combinations includes: The counterfactual multimodal samples are incorporated into the multimodal consistency graph; Perform multi-round consistency backtracking verification on the multimodal consistency graph after introducing counterfactual multimodal samples; The consistency and stability of candidate labels are evaluated based on the results of multiple rounds of backtracking verification.

9. The pre-annotation method according to claim 1, characterized in that, Based on the consistency stability result, the step of performing annotation lifecycle state transition control on the candidate annotations to determine the current annotation state of the candidate annotations includes: Based on the consistency stability results, the confidence level of the candidate labels is calculated; Based on the matching results between the trust level and multiple preset trust level ranges or thresholds, the candidate annotation is controlled to perform state transitions between multiple preset annotation lifecycle states.

10. A pre-labeling device for embodied intelligent VLA data based on multimodal data, characterized in that, include: The acquisition unit is used to acquire a multimodal data stream containing visual modal data, language modal data, and action modal data, and to perform time-series segmentation of the multimodal data stream according to the time order to obtain multiple time-segment data units; The generation unit is used to identify the current task execution state based on the visual modal data and the language modal data for each time period data unit, and generate at least one set of candidate annotation levels corresponding to the task execution state. The construction unit is used to perform multimodal semantic representation on each level of the candidate annotation hierarchy set and construct a multimodal consistency relationship graph containing visual nodes, language nodes and action nodes. The detection unit is used to perform a consistency propagation operation in the multimodal consistency relationship graph to obtain the multimodal consistency constraint relationship between each node and to detect conflicting node combinations. The construction unit is used to construct counterfactual multimodal samples associated with the candidate labels based on the corresponding candidate label level for the detected combination of conflicting nodes. The processing unit is configured to introduce the counterfactual multimodal samples into the multimodal consistency graph, perform multi-round consistency backtracking verification to determine the consistency stability of the candidate labels under different modal combinations, and, based on the consistency stability results, perform label lifecycle state transition control on the candidate labels to determine the current labeling state of the candidate labels; The output unit is used to determine the candidate annotation as the pre-annotation result of VLA data and output it when the annotation status of the candidate annotation meets the preset freezing conditions.

Images