Active learning method and system for multi-modal data collaborative labeling

Abstract

The application discloses a kind of multimodal data collaborative labeling active learning method and system, belong to machine learning and data labeling technical field.The method obtains unlabeled multimodal sample pool and labeled sample set, establishes sample identification and multimodal sample graph for each multimodal sample;Obtain node features, relationship features and atlas alignment features by machine learning feature extraction network;Train atlas active learning labeling model, output node candidate labeling result, relationship candidate labeling result, node confidence information, relationship conflict information and conflict propagation information;Accordingly, construct and merge conflict labeling unit, generate target collaborative labeling package;Then, according to collaborative labeling feedback, update labeled sample set, conflict memory bank and model.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and data annotation technology, and in particular to an active learning method and system for multimodal data collaborative annotation. Background Technology

[0002] With the widespread use of images, text, audio, video, point clouds, and sensor time-series data in applications such as intelligent recognition, knowledge extraction, scene understanding, and human-computer interaction, the demand for multimodal data annotation is constantly increasing. Multimodal data typically supplements the description of the same object, scene, or event from different data sources, providing richer training information for machine learning models. However, in practical applications, multimodal data often suffers from inconsistent collection times, spatial location offsets, inconsistent semantic descriptions, missing modalities, and different annotation granularities across modalities. This makes the multimodal data annotation process not only necessary to determine the label of each individual modality but also to determine whether there are correspondences and consistency relationships between different modalities.

[0003] Active learning is a machine learning approach that reduces the cost of manual annotation. It typically selects valuable samples from unannotated samples based on model confidence, sample uncertainty, or sample representativeness, and then manually annotates these samples, using the annotation results to update the model. Existing technologies already include solutions for entity alignment in multimodal knowledge graphs. For example, patent CN117407689B discloses an active learning method, apparatus, and electronic device for entity alignment. Based on preset active learning rules, it obtains a second entity pair that meets preset conditions from a first entity pair in the knowledge graph, trains a graph neural network model based on the first and second entity pairs, and then obtains a third entity pair based on the active learning rules to update the second entity pair, thereby reducing the annotation cost in the multimodal knowledge graph entity alignment process.

[0004] The aforementioned existing technologies can reduce the annotation cost in the entity alignment process of multimodal knowledge graphs to some extent. However, the active learning query objects of this type of scheme are mainly entity pairs in the knowledge graph. At least for the multimodal samples to be annotated, they do not treat the single modality data within the sample and the correspondence between different modality data as node objects and relation objects in collaborative annotation, respectively. Nor do they treat the temporal, spatial, or semantic relationships between different modality data as independent processing objects for machine learning active query, collaborative verification, and model update.

[0005] In multimodal collaborative annotation scenarios, annotation errors or uncertainties often do not stem solely from a single modality of data, but may arise from incorrect correspondences between different modalities. For example, a target region in an image may not match the object described in the text, a video clip may not occur at the same time as an audio event, the target location in a point cloud may be offset from the target location in the image, or the state reflected in sensor time-series data may not match the state recorded in the text. If the entire sample or entity pair is still used as the minimum query object for active learning, it becomes difficult to distinguish whether the uncertainty lies in a single modality label or in a conflicting cross-modal relationship. It also becomes difficult to determine whether the annotator should confirm the label of the modality data itself or verify the correspondences between different modalities.

[0006] Therefore, existing multimodal active learning annotation methods still have the following technical problems: when multimodal samples contain multiple modal data and cross-modal correspondences, existing methods have difficulty in treating single modal data and cross-modal relationships as queryable, verifiable and updatable annotation objects, respectively. It is difficult to locate the source of cross-modal relationship conflicts, resulting in the machine learning model selecting insufficiently refined collaborative annotation objects during active learning and the model update basis being insufficiently targeted. Summary of the Invention

[0007] To overcome the aforementioned technical deficiencies, the present invention aims to provide an active learning method and system for multimodal data collaborative annotation. This invention establishes a multimodal sample graph for each multimodal sample, including modal data nodes and cross-modal relationship edges. It then utilizes a graph-based active learning annotation model to output node confidence information, relationship conflict information, and conflict propagation information to construct conflict annotation units and target collaborative annotation packages. Combined with collaborative annotation feedback and a conflict memory update model, this addresses the problems of existing multimodal active learning annotation methods, such as difficulty in locating the source of cross-modal relationship conflicts, insufficient precision in collaborative annotation objects, and a lack of targeted basis for model updates.

[0008] This invention discloses an active learning method for collaborative annotation of multimodal data, comprising the following steps: S1. Obtain the unlabeled multimodal sample pool and the labeled sample set. For each multimodal sample in the unlabeled multimodal sample pool, establish a sample identifier and a multimodal sample graph. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. Modal data nodes correspond to single modal data in the multimodal sample. Cross-modal relationship edges are used to represent the temporal, spatial, or semantic relationships between different modal data corresponding to the same sample identifier. S2, input the multimodal sample graph into the machine learning feature extraction network to obtain the node features of the modal data nodes, the relation features of cross-modal relation edges, and the graph alignment features formed by the fusion of node features and relation features; S3, based on the labeled sample set and graph alignment features, train the graph-based active learning annotation model, so that the graph-based active learning annotation model outputs node candidate annotation results, relationship candidate annotation results, node confidence information, relationship conflict information and conflict propagation information; S4. Based on node confidence information, relationship conflict information, and conflict propagation information, conflict annotation units are constructed from the unannotated multimodal sample pool, and conflict annotation units with the same modality data nodes or the same cross-modal relationship edges are merged to obtain the target collaborative annotation package. S5 receives collaborative annotation feedback for the target collaborative annotation package, corrects the node candidate annotation results and relationship candidate annotation results based on the collaborative annotation feedback, updates the labeled sample set and conflict memory, and retrains the graph-based active learning annotation model based on the updated labeled sample set and conflict memory.

[0009] Preferably, In step S1, establishing the multimodal sample map includes: Modal data nodes are established for at least two modalities of image data, text data, audio data, video data, point cloud data, and sensor time-series data. Establish time relationship edges based on the correspondence between acquisition times of data from different modalities; Establish spatial relationship edges based on the correspondence between target locations of different modal data; Establish semantic relationship edges based on object descriptions, event descriptions, or attribute descriptions between different modalities of data; Temporal relation edges, spatial relation edges, and semantic relation edges are treated as cross-modal relation edges.

[0010] Preferably, In step S1, the multimodal sample graph also includes node annotation status and relation annotation status; Node labeling status includes unlabeled status, model pre-labeled status, manual confirmation status, and pending review status. Relationship labeling status includes unverified status, consistency confirmed status, conflict pending status, and conflict correction status; The graph-based active learning annotation model distinguishes between modal data nodes to be annotated and cross-modal relationship edges to be verified based on node annotation status and relationship annotation status.

[0011] Preferably, In step S1, when there is missing modal data in the multimodal samples corresponding to the same sample identifier, a missing placeholder node is established in the multimodal sample graph, and a modal missing attribute is set for the missing placeholder node. In step S2, the machine learning feature extraction network masks the cross-modal relationship edges connecting the missing placeholder nodes according to the modal missing attribute, and retains the sample attribution relationship between the missing placeholder node and the existing modal data node based on the sample identifier.

[0012] Preferably, In step S2, the machine learning feature extraction network includes a node encoding layer, a relation encoding layer, a graph message passing layer, and a gated fusion layer; The node encoding layer is used to extract node features from modal data nodes; The relation encoding layer is used to extract relation features of cross-modal relation edges; The graph message passing layer transmits relation features between different modal data nodes according to the edge type of cross-modal relation edges. The edge type of cross-modal relation edges includes at least one of time relation edge type, spatial relation edge type, and semantic relation edge type. The gated fusion layer generates graph alignment features based on node features, relationship features, and edge types of cross-modal relationship edges.

[0013] Preferably, In step S3, the graph-based active learning annotation model includes a node annotation branch, a relation annotation branch, and a conflict propagation branch; The node annotation branch outputs candidate node annotation results and node confidence information; The relation annotation branch outputs relation candidate annotation results and relation conflict information; The conflict propagation branch identifies cross-modal relationship edges that meet preset conflict triggering conditions as conflicting cross-modal relationship edges based on relationship conflict information, and determines the modal data nodes and cross-modal relationship edges adjacent to the conflicting cross-modal relationship edges, and outputs conflict propagation information.

[0014] Preferably, Relationship conflict information includes at least one of the following: category conflict information, target location conflict information, time segment conflict information, semantic reference conflict information, and quantity attribute conflict information; The relation annotation branch generates relation conflict information based on the candidate annotation results of the modal data nodes at both ends of the same cross-modal relation edge.

[0015] Preferably, Conflict propagation information includes conflict source relationship edges, conflict associated nodes, conflict associated relationship edges, and conflict propagation levels; Conflict source relation edges are one or more of the conflict cross-modal relation edges; The conflict propagation branch starts from the conflict source relation edge, determines the conflict association relation edge among the cross-modal relation edges that share the same modal data node with the conflict source relation edge, and determines the modal data node connecting the conflict association relation edge as the conflict association node.

[0016] Preferably, In step S4, the conflict annotation unit includes sample identifier, focal modal data node, cross-modal relationship edge to be checked, node candidate annotation result, relationship candidate annotation result, relationship conflict information, and annotation action to be performed; A focal modal data node is a modal data node whose node confidence information is lower than a preset node confidence threshold and which belongs to a conflict-related node indicated by conflict propagation information. The cross-modal relationship edges to be verified are conflict-related relationship edges indicating conflict propagation information; When the relationship conflict information of the cross-modal relationship edge to be checked meets the preset conflict triggering condition, the cross-modal relationship edge to be checked is determined as a conflicting cross-modal relationship edge.

[0017] Preferably, In step S4, the annotation actions to be performed include: When the node confidence information of the focus modal data node is lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be checked does not meet the preset conflict triggering condition, the annotation action to be performed is determined to be node label confirmation. When the node confidence information of the focus modal data node is not lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be checked meets the preset conflict triggering condition, the cross-modal relationship edge to be checked is determined as a conflicting cross-modal relationship edge, and the annotation action to be performed is determined as relationship edge checking; When the node confidence information of the focus modal data node is lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified meets the preset conflict triggering condition, the cross-modal relationship edge to be verified is identified as a conflicting cross-modal relationship edge, and the annotation action to be performed is node label confirmation and relationship edge verification.

[0018] Preferably, In step S4, merging conflict annotation units with the same modal data nodes or the same cross-modal relationship edges includes: Multiple conflicting annotation units that share the same focal modality data node are merged into a node-priority annotation unit; Multiple conflicting annotation units that share the same cross-modal relation edge to be checked will be merged into a relation-priority annotation unit; When the same conflicting annotation unit belongs to both the node-priority annotation unit and the relationship-priority annotation unit, the same conflicting annotation unit is set as a composite annotation unit; Generate a target collaborative annotation package based on node-priority annotation units, relationship-priority annotation units, and composite annotation units.

[0019] Preferably, In step S5, the collaborative annotation feedback includes node annotation feedback and relation annotation feedback returned by at least two annotation terminals. The node annotation feedback includes at least two node annotation results, and the relation annotation feedback includes at least two relation annotation results. When at least two node annotation results are consistent and at least two relationship annotation results are not completely consistent, the corresponding conflict annotation unit will be assigned to the cross-modal verification terminal. When at least two node annotation results are not completely consistent and at least two relationship annotation results are consistent, the corresponding conflict annotation unit will be assigned to the modality-specific verification terminal. When at least two node annotation results are not completely consistent and at least two relation annotation results are not completely consistent, the corresponding conflicting annotation units are added to the disputed sample set.

[0020] Preferably, In step S5, updating the conflict memory includes: Write the relationship conflict information corresponding to the conflict annotation unit that has been rectified, the edge type of the cross-modal relationship edge to be checked, the node candidate annotation result before correction, the node annotation result obtained after correction based on collaborative annotation feedback, the relationship candidate annotation result before correction, and the relationship annotation result obtained after correction based on collaborative annotation feedback into the conflict memory. The edge type of the cross-modal relationship edge to be checked includes at least one of the time relationship edge type, spatial relationship edge type, and semantic relationship edge type. Based on historical correction records in the conflict memory that have the same edge type and the same relationship conflict information for cross-modal relation edges to be checked, generate conflict query constraints; In the next round of determining the target collaborative annotation package, conflict annotation units with the same edge type and the same relationship conflict information that have the same cross-modal relationship edge to be checked are merged or excluded based on the conflict query constraint.

[0021] Preferably, In step S5, retraining the graph-based active learning annotation model based on the updated labeled sample set and conflict memory includes: The node annotation results obtained after collaborative annotation feedback correction are used as node supervision information; The relation annotation results obtained after collaborative annotation feedback correction are used as relation supervision information; Construct historical correction records from the conflict memory bank as relational counterexamples; Based on node supervision information, relationship supervision information, and relationship counterexamples, the node annotation branch, relationship annotation branch, and conflict propagation branch are jointly trained.

[0022] Preferably, it includes: The sample graph construction module is used to obtain the unlabeled multimodal sample pool and the labeled sample set, and to build a sample identifier and a multimodal sample graph for each multimodal sample in the unlabeled multimodal sample pool. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. The modal data nodes correspond to the single modal data in the multimodal sample, and the cross-modal relationship edges are used to represent the temporal relationship, spatial relationship or semantic relationship between different modal data corresponding to the same sample identifier. The machine learning feature extraction module is used to input multimodal sample graphs into the machine learning feature extraction network to obtain node features of modal data nodes, relational features of cross-modal relational edges, and graph alignment features formed by fusing node features and relational features. The graph-based active learning annotation module is used to train the graph-based active learning annotation model based on the labeled sample set and graph alignment features, so that the graph-based active learning annotation model outputs node candidate annotation results, relationship candidate annotation results, node confidence information, relationship conflict information and conflict propagation information; The collaborative annotation package generation module is used to construct conflict annotation units from the unannotated multimodal sample pool based on node confidence information, relationship conflict information, and conflict propagation information, and merge conflict annotation units with the same modality data nodes or the same cross-modal relationship edges to obtain the target collaborative annotation package; The feedback update module is used to receive collaborative annotation feedback for the target collaborative annotation package, correct the candidate annotation results of nodes and candidate annotation results of relationships based on the collaborative annotation feedback, update the labeled sample set and conflict memory, and retrain the graph-based active learning annotation model based on the updated labeled sample set and conflict memory.

[0023] Compared with existing technologies, the above technical solution has the following advantages: 1. This invention establishes a multimodal sample graph containing modal data nodes and cross-modal relationship edges for each multimodal sample, enabling different modal data such as image data, text data, audio data, video data, point cloud data, and sensor time series data to form a unified graph representation under the same sample identifier. This refines the traditional active learning query object, which is mainly based on whole samples or entity pairs, into modal data nodes and cross-modal relationship edges, which is beneficial to improving the precision of multimodal data collaborative annotation objects.

[0024] 2. This invention outputs node candidate annotation results, relation candidate annotation results, node confidence information, relation conflict information, and conflict propagation information through a graph-based active learning annotation model. This enables the system to not only determine the annotation uncertainty of a single modality of data itself, but also to determine whether there are conflicts in the temporal, spatial, or semantic relationships between different modalities of data. This is beneficial for locating the source of cross-modal relationship conflicts and avoids coarse-grained annotation selection based solely on the overall sample confidence.

[0025] 3. This invention constructs conflict annotation units based on node confidence information, relationship conflict information, and conflict propagation information, and merges conflict annotation units with the same modal data nodes or the same cross-modal relationship edges to form a target collaborative annotation package. This enables objects entering the manual collaborative annotation process to be organized around the focus modal data nodes and the cross-modal relationship edges to be verified, which helps to reduce duplicate and invalid annotations and improve the targeting of collaborative annotation task allocation.

[0026] 4. By setting up node label confirmation, relationship edge verification, and a combination of node label confirmation and relationship edge verification, this invention enables the annotation terminal to perform different types of annotation tasks based on node confidence information and relationship conflict information. This distinguishes between single-modal label confirmation and cross-modal relationship verification, which helps to improve the clarity of the multimodal collaborative annotation process and the usability of the annotation results.

[0027] 5. This invention determines the conflict source relationship edge, conflict associated node, conflict associated relationship edge, and conflict propagation level through conflict propagation information. This enables the system to start from the cross-modal relationship edge where the conflict occurs and further determine the modal data nodes and relationship edges associated with the conflict. This is beneficial to expanding the ability to identify the scope of potential conflicts and reducing the need for manual review of irrelevant samples or irrelevant modal data.

[0028] 6. This invention sets up a conflict memory bank to record relationship conflict information, edge types of cross-modal relationship edges to be checked, node annotation results before and after correction, and relationship annotation results before and after correction. It also generates conflict query constraints based on historical correction records, which is beneficial for merging or excluding conflict annotation units of the same type in subsequent active learning rounds, thereby reducing the probability of the same conflict source repeatedly entering the manual annotation process. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the active learning method for multimodal data collaborative annotation provided in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the structure of a multimodal sample diagram provided in an embodiment of the present invention.

[0031] Figure 3This is a schematic diagram of the structure of the graph-based active learning annotation model provided in an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram illustrating the process of merging conflict annotation units to generate a target collaborative annotation package, as provided in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram illustrating the process of conflict memory participating in model retraining, as provided in an embodiment of the present invention.

[0034] Figure 6 The graph shows the change in the overall annotation accuracy of the model as a function of the active learning rounds, as provided in the embodiments of the present invention.

[0035] Figure 7 The graph showing the change in the percentage of repeatedly labeled objects with the number of active learning rounds, provided in this embodiment of the invention.

[0036] Figure 8 This is a comparison chart of the accuracy of cross-modal conflict source localization provided in an embodiment of the present invention. Detailed Implementation

[0037] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0039] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0040] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0041] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0042] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0043] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0044] The specific embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention.

[0045] like Figure 1 As shown, this embodiment provides an active learning method for collaborative annotation of multimodal data. This method belongs to the active learning data annotation and model training technology in machine learning, and is applicable to multimodal sample annotation scenarios involving images, text, audio, video, point clouds, and sensor time-series data. In this embodiment, the method no longer uses the entire sample as the active learning query object, but constructs each multimodal sample into a multimodal sample graph containing modal data nodes and cross-modal relationship edges. The multimodal sample graph is used to support the graph-based active learning annotation model in handling the annotation uncertainty of single modal data itself and the conflict between relationships between different modal data, and organizes nodes and relationship edges with conflict associations into target collaborative annotation packages, so that the annotation terminal can confirm, verify, and review the source of conflict.

[0046] In this embodiment, an unlabeled multimodal sample pool and a labeled sample set are first obtained. The unlabeled multimodal sample pool can come from business systems, acquisition devices, data platforms, or historical storage systems. The labeled sample set can be a manually pre-labeled dataset or a historical sample set that has been reviewed and confirmed. To ensure unified management of different modal data under the same object, event, or scenario, a sample identifier is established for each multimodal sample in the unlabeled multimodal sample pool. The sample identifier can be a unique identifier formed by combining the acquisition batch, acquisition time, acquisition object number, and sequence number, or it can be a unique code generated by the data management system. Through the sample identifier, image data, text data, audio data, video data, point cloud data, and sensor time-series data in the same multimodal sample can be classified into the same processing object.

[0047] like Figure 2 As shown, after establishing sample identifiers, a multimodal sample graph is constructed for each multimodal sample. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. Modal data nodes correspond to single-modal data within the multimodal sample; for example, image data nodes correspond to a single image or a set of images, text data nodes correspond to text descriptions or records, audio data nodes correspond to sound clips, video data nodes correspond to video clips, point cloud data nodes correspond to spatial point clouds, and sensor time-series data nodes correspond to continuously acquired state data. Cross-modal relationship edges are used to represent the temporal, spatial, or semantic relationships between different modal data corresponding to the same sample identifier. Through this structure, both the single-modal data itself and the correspondences between different modal data can become processing objects in the active learning process of machine learning.

[0048] In a specific scenario, unlabeled multimodal samples include image data, text records, video clips, laser point clouds, and time-series data from speed sensors in a vehicle traffic scenario. Image data nodes represent vehicle appearance images, text data nodes represent license plate recognition text or manually recorded text, video data nodes represent continuous video clips of vehicles passing through the detection area, point cloud data nodes represent vehicle spatial contours, and sensor time-series data nodes represent continuously acquired information such as speed, distance, or trigger time. Temporal relationship edges are established based on the time correspondence between different modal data, for example, a time relationship edge is established between video clips and sensor time-series data based on timestamps; spatial relationship edges are established based on the target position correspondence between different modal data, for example, a spatial relationship edge is established between image target bounding boxes and point cloud target contours; semantic relationship edges are established based on object descriptions, event descriptions, or attribute descriptions between different modal data, for example, a semantic relationship edge is established between the vehicle type in the text record and the vehicle type recognized in the image. Temporal, spatial, and semantic relationship edges all participate in subsequent processing as cross-modal relationship edges.

[0049] To facilitate control of the subsequent active learning annotation process, the multimodal sample graph can also include node annotation status and relation annotation status. Node annotation status characterizes the annotation progress and confirmation level of modal data nodes, including unannotated, model pre-annotated, manually confirmed, and pending verification status. Relation annotation status characterizes the verification progress and conflict status of cross-modal relation edges, including unverified, consistent, conflict pending, and conflict correction status. The graph-based active learning annotation model distinguishes between modal data nodes to be annotated and cross-modal relation edges to be verified based on node and relation annotation status. For example, when an image data node is in the model pre-annotated state, but the semantic relation edge between the image data node and the text data node is in the conflict pending state, the system will include both the image data node and the semantic relation edge in the subsequent conflict propagation judgment.

[0050] During multimodal data acquisition, data for a particular modality may be missing. For example, a vehicle traffic sample may contain image data and sensor time-series data, but point cloud data may not be obtained due to device occlusion. To avoid structural breaks in the sample graph caused by missing modalities, this embodiment establishes a missing placeholder node in the multimodal sample graph when missing modal data exists in the multimodal samples corresponding to the same sample identifier, and sets a modality missing attribute for the missing placeholder node. The missing placeholder node does not directly participate in the actual label prediction, but retains the sample attribution relationship between it and existing modal data nodes based on the sample identifier. The machine learning feature extraction network masks the cross-modal relationship edges connecting the missing placeholder node according to the modality missing attribute, so that the missing modality does not generate erroneous relationship features, while preserving the complete structure of the multimodal sample graph.

[0051] After constructing the multimodal sample graph, it is input into a machine learning feature extraction network. This network extracts node features from modal data nodes, relational features from cross-modal edges, and graph alignment features formed by fusing node and relational features. Node features represent the content information of a single modality, relational features represent the correspondence between different modalities, and graph alignment features represent the overall alignment status and conflict distribution of the multimodal sample graph.

[0052] like Figure 3As shown, the machine learning feature extraction network can include a node encoding layer, a relation encoding layer, a graph messaging layer, and a gated fusion layer. The node encoding layer extracts node features from modal data nodes. The relation encoding layer extracts relation features from cross-modal relation edges. For temporal relation edges, the relation encoding layer can extract the acquisition time difference, temporal overlap, or event sequence between different modal data; for spatial relation edges, the relation encoding layer can extract target position offset, regional overlap, or coordinate correspondence; for semantic relation edges, the relation encoding layer can extract object description consistency, event description consistency, or attribute description consistency. The edge type of cross-modal relation edges includes at least one of temporal relation edge types, spatial relation edge types, and semantic relation edge types. The graph messaging layer transmits relation features between different modal data nodes according to the edge type of the cross-modal relation edges. The gated fusion layer generates graph alignment features based on node features, relation features, and the edge type of the cross-modal relation edges.

[0053] In a specific calculation example, node confidence information, relationship conflict information, and conflict annotation cell priority values ​​can be calculated as follows. For ease of explanation, node confidence information can be expressed as node confidence values. For quantitative representation, relationship conflict information can be expressed using relationship conflict values. For quantitative representation, the conflict propagation hierarchy can be represented using conflict propagation hierarchy values. For quantitative representation, the ordering of conflict annotation units can be based on the priority value of the conflict annotation units. Quantify the representation.

[0054] For a given image data node, the node annotation branch outputs three candidate categories and their predicted probabilities: "Truck" (0.62), "Small Passenger Vehicle" (0.27), and "Other Vehicles" (0.11). The node confidence value can be the maximum predicted probability, or it can be quantified using the difference in predicted probabilities, prediction entropy, or consistency across multiple predictions. In this embodiment, the maximum predicted probability is used as the quantization method for the node confidence value; therefore, the node confidence value of this image data node is... It can be determined using the following formula: in, Indicates the node confidence value. This indicates that the maximum value is taken. If the preset node confidence threshold is 0.70, then the node confidence value of this image data node is 0.62, which is lower than the preset node confidence threshold. Therefore, this image data node is determined to have node label uncertainty.

[0055] For semantic relationship edges between image data nodes and text data nodes, assuming the candidate category of the image data node is "truck" and the candidate category of the text data node is "passenger car," their categories are inconsistent; simultaneously, the text data node records the number of axles as 2, and the image data node identifies the number of axles as 2, their quantity attributes are consistent. We can set the category conflict score to 1, the quantity attribute conflict score to 0, and set the category conflict weight to 0.70 and the quantity attribute conflict weight to 0.30. (Relationship conflict value) It can be determined using the following formula: in, This represents the relation conflict value. If the preset conflict triggering condition is that the relation conflict value is not lower than 0.60, then the relation conflict value of this semantic relation edge is 0.70, which meets the preset conflict triggering condition. Therefore, this semantic relation edge can be identified as a conflicting cross-modal relation edge. By quantifying attribute consistency using quantitative attribute conflict scores, this example can maintain consistency with quantitative attribute conflict information while still reflecting the quantification process of attribute description consistency or conflict status in semantic relation edges.

[0056] Regarding the time relationship between video and audio data nodes, assuming the video event time segment is from second 10 to second 15 and the audio event time segment is from second 18 to second 22, then the center points of their time segments are 12.5 seconds and 20 seconds respectively, with a time center difference of 7.5 seconds. If the allowable time deviation threshold is 3 seconds, a time segment conflict score can be calculated. : in, Indicates the score for time segment conflicts. This indicates that the smaller value is taken. Since the time segment conflict score reaches 1, it means that the time relationship edge meets the judgment condition of time segment conflict. If the relationship conflict value of the time relationship edge adopts the time segment conflict score, then the relationship conflict value of the time relationship edge meets the preset conflict triggering condition.

[0057] For the spatial relationship edge between point cloud data nodes and image data nodes, assume the projected coordinates of the point cloud target center are 130 (horizontal) and 90 (vertical), and the coordinates of the image target bounding box center are 118 (horizontal) and 86 (vertical). The spatial offset distance between them is... The calculation can be performed using the following formula: After substituting the values, we get: in, This represents the spatial offset distance. If the allowed threshold for spatial offset is 10, then the target location conflict score used to characterize target location conflict information is... It can be determined using the following formula: in, The score represents the target location conflict score. This result indicates that the correspondence between target locations between point cloud data nodes and image data nodes meets the criteria for determining target location conflict.

[0058] In this embodiment, the time segment conflict score and the target location conflict score can be used as the relationship conflict value of the corresponding cross-modal relationship edge. For example, the time segment conflict score can be used as the relationship conflict value corresponding to the time relationship edge, and the target location conflict score can be used as the relationship conflict value corresponding to the spatial relationship edge. In other embodiments, the time segment conflict score, target location conflict score, category conflict score, semantic reference conflict score, and quantity attribute conflict score can also be weighted and combined according to preset weights to form the relationship conflict value of the corresponding cross-modal relationship edge, thereby determining whether the corresponding cross-modal relationship edge meets the preset conflict triggering condition.

[0059] After obtaining node confidence information and relationship conflict information, the conflict propagation branch, based on the relationship conflict information, identifies cross-modal relationship edges that meet preset conflict triggering conditions as conflicting cross-modal relationship edges, and determines the modal data nodes and cross-modal relationship edges adjacent to the conflicting cross-modal relationship edges, outputting conflict propagation information. Conflict propagation information includes conflict source relationship edges, conflict-associated nodes, conflict-associated relationship edges, and conflict propagation hierarchy. Conflict source relationship edges are one or more of the conflicting cross-modal relationship edges. Starting from the conflict source relationship edge, the conflict propagation branch identifies conflict-associated relationship edges among the cross-modal relationship edges that share the same modal data node with the conflict source relationship edge, and identifies the modal data nodes connecting the conflict-associated relationship edges as conflict-associated nodes.

[0060] To ensure the selection of conflict annotation units has a calculable basis, this embodiment can also calculate the priority value of conflict annotation units based on node confidence information, relationship conflict information, and conflict propagation level. In an optional implementation, the conflict annotation unit priority value... It can be determined using the following formula: in, Indicates the priority value of the conflict annotation unit. Indicates the node confidence value. Indicates the value of relationship conflict. Indicates the conflict propagation level value. Represents the uncertainty weight of the node. Indicates the relationship conflict weight. This represents the propagation hierarchy weight. For ease of implementation, the node uncertainty weight can be set to 0.40, the relationship conflict weight to 0.45, and the propagation hierarchy weight to 0.15. These node uncertainty weights, relationship conflict weights, and propagation hierarchy weights can be adjusted based on the annotation accuracy on the validation set, the accuracy of locating cross-modal conflict sources, or the workload of manual annotation.

[0061] Taking a conflict annotation unit as an example, the node confidence value of the focal modal data node is 0.62, the relationship conflict value of the cross-modal relationship edge to be checked is 0.70, and the conflict propagation level value is 1. Then the priority value of the conflict annotation unit can be calculated as follows: If the preset annotation priority threshold is 0.50, then the priority value of this conflict annotation unit is 0.542, which is higher than the preset annotation priority threshold. Therefore, this conflict annotation unit is selected into the target collaborative annotation package. The preset annotation priority threshold can be set according to the capacity of the target collaborative annotation package, the number of annotation terminals, or the manual annotation budget. This calculation method helps to prioritize conflict annotation units with high node uncertainty, strong relationship conflict, and close propagation level into the collaborative annotation process.

[0062] When constructing a conflict annotation unit, the conflict annotation unit includes a sample identifier, a focal modal data node, a cross-modal relationship edge to be verified, candidate node annotation results, candidate relationship annotation results, relationship conflict information, and an annotation action to be performed. A focal modal data node is a modal data node whose node confidence information is lower than a preset node confidence threshold and belongs to a conflict-related node indicated by the conflict propagation information. A cross-modal relationship edge to be verified is a conflict-related relationship edge indicated by the conflict propagation information. When the relationship conflict information of a cross-modal relationship edge to be verified meets a preset conflict triggering condition, the cross-modal relationship edge to be verified is determined as a conflicting cross-modal relationship edge.

[0063] In one specific embodiment, when the node confidence information of a focal modal data node is lower than a preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified does not meet the preset conflict triggering condition, the annotation action to be performed is determined to be node label confirmation. This situation indicates that the annotation of a single modal data node itself is unstable, but no relationship conflict has yet formed between it and other modal data, so there is no need to prioritize relationship edge verification. When the node confidence information of a focal modal data node is not lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified meets the preset conflict triggering condition, the cross-modal relationship edge to be verified is determined to be a conflicting cross-modal relationship edge, and the annotation action to be performed is determined to be relationship edge verification. This situation indicates that the annotation of the modal data node itself is relatively stable, but there is a conflict in the correspondence between different modal data, so the relationship edge verification is performed by the annotation terminal. When the node confidence information of a focal modal data node is lower than a preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified meets the preset conflict triggering condition, the cross-modal relationship edge to be verified is identified as a conflicting cross-modal relationship edge, and the annotation actions to be performed are node label confirmation and relationship edge verification. This situation indicates that both the node's own label and the cross-modal relationship have uncertainties and need to be processed simultaneously.

[0064] like Figure 4 As shown, after constructing multiple conflict annotation units, conflict annotation units with the same modal data node or the same cross-modal relationship edge are merged to obtain the target collaborative annotation package. Specifically, multiple conflict annotation units sharing the same focal modal data node are merged into node-priority annotation units; multiple conflict annotation units sharing the same cross-modal relationship edge to be verified are merged into relationship-priority annotation units; when the same conflict annotation unit belongs to both node-priority and relationship-priority annotation units, it is set as a composite annotation unit. Then, the target collaborative annotation package is generated based on the node-priority annotation units, relationship-priority annotation units, and composite annotation units. Through the above merging method, the repetitive processing of the same focal modal data node or the same cross-modal relationship edge to be verified by multiple annotation terminals can be avoided, improving the organizational efficiency of collaborative annotation tasks.

[0065] After the target collaborative annotation package is generated, it is sent to the annotation terminals. Collaborative annotation feedback includes node annotation feedback and relationship annotation feedback returned by at least two annotation terminals. Node annotation feedback includes at least two node annotation results, and relationship annotation feedback includes at least two relationship annotation results. Annotation terminals may include ordinary annotation terminals, cross-modal verification terminals, modality-specific verification terminals, and disputed sample processing terminals.

[0066] When at least two node annotation results are consistent and at least two relationship annotation results are not entirely consistent, the corresponding conflicting annotation unit is assigned to the cross-modal verification terminal. This indicates that the label confirmation of a single modality data node is relatively stable, but different annotation terminals disagree on whether cross-modal relationship edges are valid. When at least two node annotation results are not entirely consistent and at least two relationship annotation results are consistent, the corresponding conflicting annotation unit is assigned to the modality-specific review terminal. This indicates that cross-modal relationship edges are considered relatively stable, but the label of a certain modality data node itself is still controversial. When at least two node annotation results are not entirely consistent and at least two relationship annotation results are not entirely consistent, the corresponding conflicting annotation unit is added to the disputed sample set. This indicates that both node label and relationship edge verification are controversial and require processing through a higher-level review process.

[0067] In this embodiment, "at least two node annotation results being consistent" can mean that all node annotation results are the same, or it can mean that a preset majority consistency rule is met; similarly, "at least two relationship annotation results being consistent" can mean that all relationship annotation results are the same, or it can mean that a preset majority consistency rule is met. Correspondingly, "at least two node annotation results not being completely consistent" means that either all nodes are not completely consistent or the preset majority consistency rule is not met; similarly, "at least two relationship annotation results not being completely consistent" means either either all nodes are not completely consistent or the preset majority consistency rule is not met. The preset majority consistency rule can be set according to the number of annotation terminals, the type of annotation task, or the review level.

[0068] like Figure 5 As shown, after receiving collaborative annotation feedback, the node candidate annotation results and relation candidate annotation results are corrected based on the feedback, and the annotated sample set and conflict memory are updated. The conflict memory stores the conflict sources, correction processes, and subsequent query constraints of historical conflict samples. When updating the conflict memory, the relation conflict information corresponding to the corrected conflict annotation units, the edge type of the cross-modal relation edge to be checked, the node candidate annotation results before correction, the node annotation results obtained after correction based on collaborative annotation feedback, the relation candidate annotation results before correction, and the relation annotation results obtained after correction based on collaborative annotation feedback are written into the conflict memory. The edge type of the cross-modal relation edge to be checked includes at least one of the following: temporal relation edge type, spatial relation edge type, and semantic relation edge type.

[0069] The conflict memory is also used to generate conflict query constraints for the next round of active learning queries. Specifically, conflict query constraints are generated based on historical correction records in the conflict memory that have the same edge type for cross-modal relationship edges to be checked and the same relationship conflict information. In the next round of determining the target collaborative annotation package, conflict annotation units with the same edge type for cross-modal relationship edges to be checked and the same relationship conflict information are merged or excluded based on the conflict query constraints.

[0070] To generate conflict query constraints based on historical correction records in the conflict memory, conflict repetition rate and actual conflict rate can be set as auxiliary statistical indicators. Conflict repetition rate is used to characterize the proportion of conflict information of a certain edge type and a certain relationship that appears repeatedly in the historical correction records and is confirmed to belong to the same type of conflict cause. Actual conflict rate is used to characterize the proportion of conflict information of a certain edge type and a certain relationship that is confirmed to belong to an actual relationship conflict in the historical correction records.

[0071] In a specific calculation example, the conflict memory contains the following historical correction records: Target location conflict information corresponding to spatial relationship edge types appears 50 times, of which 42 instances, confirmed by collaborative annotation feedback, belong to relationship conflicts caused by the same type of equipment calibration offset. The conflict repetition rate of this type of historical correction record can be calculated. : in, This represents the conflict repetition rate. If the preset repetition rate threshold is 0.80, then the conflict repetition rate is 0.84, which is higher than the preset repetition rate threshold. Based on this, the system generates conflict query constraints, that is, for conflict annotation units that subsequently appear with spatial relationship edge types and whose relationship conflict information is target location conflict information, priority is given to merging processing to avoid multiple conflict annotation units with the same reason being repeatedly assigned to different annotation terminals.

[0072] For example, slight semantic description differences corresponding to semantic relationship edge types occurred 60 times, of which only 5 were confirmed as true relationship conflicts through collaborative annotation feedback. The true conflict rate of this type of historical correction record can be calculated. : in, This represents the actual conflict rate. If the preset actual conflict rate threshold is 0.10, then the actual conflict rate is 0.083, which is lower than the preset threshold. Based on this, the system generates conflict query constraints, meaning that lower-level conflict annotation units with the same edge type and relationship conflict information can be excluded or have their priority lowered when entering the target collaborative annotation package. Through the above calculations, the conflict memory not only serves as a historical data storage unit but also provides calculable constraints for the next round of active learning queries.

[0073] After updating the labeled sample set and conflict memory, the graph-based active learning annotation model is retrained based on the updated labeled sample set and conflict memory. The retraining process includes using node annotation results corrected based on collaborative annotation feedback as node supervision information, relation annotation results corrected based on collaborative annotation feedback as relation supervision information, and constructing relation counterexamples from historical correction records in the conflict memory. Relation counterexamples refer to samples used to train the model to identify erroneous cross-modal or conflicting relations, such as semantic relation edges where the image target and the text description object are inconsistent, temporal relation edges where video event segments and audio event segments are inconsistent, and spatial relation edges where the point cloud target location and the image target region are inconsistent. Based on the node supervision information, relation supervision information, and relation counterexamples, the node annotation branch, relation annotation branch, and conflict propagation branch are jointly trained.

[0074] To make the retraining process feasible, a joint training loss value can be set. The joint training loss value includes the node labeling loss value. Relationship labeling loss value And relational counterexample loss value In one alternative implementation, the joint training loss value... It can be determined using the following formula: in, This represents the joint training loss value. This represents the node labeling loss value. Indicate the loss value of the relationship label. This represents the loss value for counterexamples of the relation. Indicates the node labeling loss weight. Indicate the loss weights of the relation labels. The loss weights for relational negative examples are represented. The node labeling loss weights, relation labeling loss weights, and relational negative example loss weights mentioned above can be adjusted based on the node labeling accuracy, relation verification accuracy, conflict propagation accuracy, or model convergence on the validation set.

[0075] In a specific training epoch, the node annotation loss is 0.35, the relation annotation loss is 0.28, the relation counterexample loss is 0.42, the node annotation loss weight is 0.40, the relation annotation loss weight is 0.35, and the relation counterexample loss weight is 0.25. Therefore, the joint training loss can be calculated as follows: The joint training loss is used to back-update the node labeling branch, relationship labeling branch, and conflict propagation branch, enabling the model to simultaneously learn node label prediction, cross-modal relationship judgment, and conflict propagation range determination. If the joint training loss continues to decrease with each active learning round, it indicates that the collaborative labeling feedback and historical correction records in the conflict memory can participate in model optimization.

[0076] To illustrate the effectiveness of this embodiment in a multimodal collaborative annotation scenario compared to existing schemes that primarily use entity pairs or sample-level objects as the main active learning query objects, the following exemplary test data can be used for explanation. The test data includes 10,000 multimodal samples, each containing at least three modalities: image data, text data, video data, point cloud data, and sensor time-series data. The test tasks are target category annotation, target location verification, and event time verification. The comparison object uses existing entity pair-level active learning schemes, which mainly select objects to be annotated based on entity pairs or the entire sample; this embodiment uses multimodal sample maps, conflict annotation units, and conflict memory for collaborative annotation. The following comparison table (Table 1) shows the data obtained from a set of exemplary tests.

[0077] Table 1 Comparison of Schemes To further illustrate the changing trends of the above exemplary test data, please refer to... Figure 6 , Figure 7 and Figure 8 . Figure 6 The relationship between the number of active learning rounds and the overall model annotation accuracy is shown. In the initial rounds, both schemes use the same initial labeled sample set, and the overall model annotation accuracy is 78.2%. After three rounds of active learning, the overall model annotation accuracy of the existing entity-pair-level active learning scheme improves to 84.9%, while the overall model annotation accuracy of the scheme in this embodiment improves to 91.3%. This curve illustrates that, in this embodiment, the conflict annotation unit and conflict memory can provide the model with node supervision information, relationship supervision information, and relationship counterexamples, which helps the model achieve stable performance improvement in subsequent training rounds.

[0078] Figure 7The relationship between the number of active learning rounds and the percentage of duplicate labeled objects is shown. In the initial round, the percentage of duplicate labeled objects in both schemes is 24.6%. After three rounds of active learning, the percentage of duplicate labeled objects in the existing entity-pair-level active learning scheme decreases to 18.5%, while the percentage of duplicate labeled objects in this embodiment decreases to 7.2%. This curve illustrates that this embodiment helps reduce the situation where the same conflict source is repeatedly assigned to multiple labeling terminals by merging conflict labeling units that share the same focal modality data node or the same cross-modal relationship edge to be checked, and by using a conflict memory to generate conflict query constraints.

[0079] Figure 8 The diagram compares the accuracy of cross-modal conflict source localization between existing entity-pair-level active learning schemes and the scheme of this embodiment. The existing entity-pair-level active learning scheme achieves a cross-modal conflict source localization accuracy of 62.8%, while the scheme of this embodiment achieves 88.6%. This comparison illustrates that this embodiment treats modal data nodes within multimodal samples and cross-modal relationship edges as processing objects, and determines the focal modal data nodes and the cross-modal relationship edges to be verified through relationship conflict information and conflict propagation information, which helps in locating cross-modal conflict sources.

[0080] As can be seen from the above exemplary test data, when there are conflicts in temporal, spatial, and semantic relationships within multimodal samples, this embodiment uses a multimodal sample graph to treat single-modal data and cross-modal relationships as processing objects in the active learning process of machine learning, which helps to locate the source of cross-modal conflicts. The conflict annotation units and target collaborative annotation packages help reduce redundant annotation of the overall samples. The conflict memory can transform historical correction records into conflict query constraints and relational counterexamples, making model retraining more targeted. The above exemplary test data is only used to illustrate the implementation effect of this embodiment in a specific application scenario and does not constitute a limitation on the application scenario, data scale, or model structure of this invention.

[0081] Through the above implementation methods, this invention expands the query objects of machine learning active learning from traditional whole samples or entity pairs to modal data nodes and cross-modal relationship edges during multimodal data collaborative annotation. It constructs conflict annotation units using node confidence information, relationship conflict information, and conflict propagation information, enabling the annotation terminal to perform task-based verification of the consistency of individual modal data labels and cross-modal relationships. Simultaneously, this invention records node annotation results and relationship annotation results before and after correction through a conflict memory, and constructs historical correction records as relational counterexamples to participate in the retraining of the graph-based active learning annotation model. This allows the graph-based active learning annotation model to continuously improve its node annotation, relationship verification, and conflict propagation capabilities by utilizing collaborative annotation feedback.

[0082] This embodiment provides an active learning system for multimodal data collaborative annotation. This system belongs to the same inventive concept as the aforementioned active learning method for multimodal data collaborative annotation, and can be used to perform the processing steps described in the aforementioned method implementation, including data acquisition, sample graph construction, feature extraction, active learning annotation, conflict annotation unit construction, target collaborative annotation package generation, collaborative annotation feedback processing, conflict memory update, and model retraining. The technical content already described in detail in the aforementioned method implementation, such as multimodal sample graphs, modal data nodes, cross-modal relationship edges, node confidence information, relationship conflict information, conflict propagation information, conflict annotation units, target collaborative annotation packages, collaborative annotation feedback, conflict memory, and relationship counterexamples, has the same meaning in this system embodiment.

[0083] In one specific implementation, the active learning system for multimodal data collaborative annotation can be deployed on a server, edge computing device, cloud computing platform, or a distributed computing environment composed of multiple computing nodes. The system may include a processor, memory, data interface, and annotation terminal communication interface. The memory stores program instructions for performing active learning processing of multimodal data collaborative annotation, an annotated sample set, an unannotated multimodal sample pool, a conflict memory, and model parameter files. When the processor invokes the program instructions, it can perform processes such as sample graph construction, machine learning feature extraction, graph-based active learning annotation, collaborative annotation package generation, and feedback updates. The data interface is used to receive at least two modalities of data, including image data, text data, audio data, video data, point cloud data, and sensor time-series data. The annotation terminal communication interface is used to send target collaborative annotation packages to at least two annotation terminals and receive node annotation feedback and relationship annotation feedback.

[0084] The active learning system for multimodal data collaborative annotation in this embodiment includes a sample graph construction module, a machine learning feature extraction module, a graph-based active learning annotation module, a collaborative annotation package generation module, and a feedback update module. Each module can be implemented through software programs or through a combination of software and hardware. In actual deployment, each module can be located on the same server or deployed separately on data preprocessing nodes, model training nodes, annotation task management nodes, and feedback update nodes.

[0085] The sample graph construction module is used to acquire an unlabeled multimodal sample pool and a labeled sample set, and to establish a sample identifier and a multimodal sample graph for each multimodal sample in the unlabeled multimodal sample pool. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. The modal data nodes correspond to single modal data in the multimodal samples, and the cross-modal relationship edges represent the temporal, spatial, or semantic relationships between different modal data corresponding to the same sample identifier. In specific implementations, the sample graph construction module can establish temporal, spatial, and semantic relationship edges based on the acquisition time, acquisition location, object description, event description, and attribute description, and write these relationship edges as cross-modal relationship edges into the multimodal sample graph.

[0086] When missing modality data exists in a multimodal sample, the sample graph construction module can also create missing placeholder nodes in the multimodal sample graph and set a modality missing attribute for these nodes. These placeholder nodes maintain the structural integrity of the multimodal sample graph, enabling the subsequent machine learning feature extraction module to identify the modality missing status of the sample and mask cross-modal relationship edges connecting the missing placeholder nodes. Therefore, missing modality data will not be mistakenly used as real modality data in relation conflict judgment.

[0087] The machine learning feature extraction module is used to input the multimodal sample graph into the machine learning feature extraction network to obtain the node features of the modal data nodes, the relation features of the cross-modal relation edges, and the graph alignment features formed by fusing the node features and the relation features. Specifically, the machine learning feature extraction module may include a node encoding layer, a relation encoding layer, a graph messaging layer, and a gated fusion layer. The node encoding layer is used to extract node features of image data nodes, text data nodes, audio data nodes, video data nodes, point cloud data nodes, or sensor time-series data nodes; the relation encoding layer is used to extract relation features of temporal relation edges, spatial relation edges, and semantic relation edges; the graph messaging layer transmits relation features between different modal data nodes according to the edge type of the cross-modal relation edges; and the gated fusion layer generates graph alignment features based on the node features, relation features, and edge types of the cross-modal relation edges.

[0088] The graph-based active learning annotation module is used to train a graph-based active learning annotation model based on the labeled sample set and the graph alignment features. The model then outputs candidate node annotations, candidate relationship annotations, node confidence information, relationship conflict information, and conflict propagation information. Specifically, the graph-based active learning annotation module may include a node annotation branch, a relationship annotation branch, and a conflict propagation branch. The node annotation branch outputs candidate node annotations and node confidence information for each modal data node; the relationship annotation branch outputs candidate relationship annotations and relationship conflict information for cross-modal relationship edges; and the conflict propagation branch determines conflicting cross-modal relationship edges based on the relationship conflict information and outputs the conflict source relationship edge, conflict-related nodes, conflict-related relationship edges, and conflict propagation level.

[0089] In one implementation, the relation annotation branch can generate relation conflict information based on the candidate node annotation results corresponding to the modal data nodes at both ends of the same cross-modal relation edge. The relation conflict information may include at least one of category conflict information, target location conflict information, time segment conflict information, semantic reference conflict information, and quantity attribute conflict information. For example, when the candidate category corresponding to an image data node is inconsistent with the candidate category corresponding to a text data node, the relation annotation branch can generate category conflict information; when the target position of a point cloud data node deviates from the target position of an image data node by more than a preset range, the relation annotation branch can generate target location conflict information; when the event time segment corresponding to a video data node is inconsistent with the event time segment corresponding to an audio data node, the relation annotation branch can generate time segment conflict information.

[0090] The collaborative annotation package generation module constructs conflict annotation units from the unannotated multimodal sample pool based on node confidence information, relationship conflict information, and conflict propagation information. It then merges conflict annotation units with data nodes of the same modality or the same cross-modal relationship edges to obtain the target collaborative annotation package. In practice, the module first determines the focus modality data nodes based on node confidence information, determines the cross-modal relationship edges to be verified based on conflict propagation information, and then combines the node candidate annotation results, relationship candidate annotation results, relationship conflict information, and the annotation actions to be performed to form conflict annotation units.

[0091] In one specific implementation, when the node confidence information of the focal modal data node is lower than a preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified does not meet the preset conflict triggering condition, the collaborative annotation package generation module determines the annotation action to be performed as node label confirmation; when the node confidence information of the focal modal data node is not lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified meets the preset conflict triggering condition, the collaborative annotation package generation module determines the cross-modal relationship edge to be verified as a conflicting cross-modal relationship edge, and determines the annotation action to be performed as relationship edge verification; when the node confidence information of the focal modal data node is lower than the preset node confidence threshold, and the relationship conflict information of the cross-modal relationship edge to be verified meets the preset conflict triggering condition, the collaborative annotation package generation module determines the annotation actions to be performed as node label confirmation and relationship edge verification.

[0092] The collaborative annotation package generation module can also merge multiple conflicting annotation units sharing the same focal modal data node into a node-priority annotation unit, merge multiple conflicting annotation units sharing the same cross-modal relationship edge to be verified into a relationship-priority annotation unit, and set conflicting annotation units that belong to both node-priority and relationship-priority annotation units as composite annotation units. Then, the collaborative annotation package generation module generates a target collaborative annotation package based on the node-priority annotation units, relationship-priority annotation units, and composite annotation units, and sends the target collaborative annotation package to the annotation terminal.

[0093] The feedback update module receives collaborative annotation feedback for the target collaborative annotation package, corrects the candidate node annotation results and candidate relationship annotation results based on the collaborative annotation feedback, updates the labeled sample set and conflict memory, and retrains the graph-based active learning annotation model based on the updated labeled sample set and conflict memory. In specific implementations, the collaborative annotation feedback may include node annotation feedback and relationship annotation feedback returned by at least two annotation terminals. The node annotation feedback includes at least two node annotation results, and the relationship annotation feedback includes at least two relationship annotation results.

[0094] In one implementation, when at least two node annotation results are consistent and at least two relationship annotation results are not completely consistent, the feedback update module can assign the corresponding conflict annotation unit to the cross-modal verification terminal; when at least two node annotation results are not completely consistent and at least two relationship annotation results are consistent, the feedback update module can assign the corresponding conflict annotation unit to the modality-specific verification terminal; when at least two node annotation results are not completely consistent and at least two relationship annotation results are not completely consistent, the feedback update module can add the corresponding conflict annotation unit to the disputed sample set. Here, "at least two node annotation results are consistent" can mean that all node annotation results are the same, or it can mean that a preset majority consistency rule is met; "at least two relationship annotation results are consistent" can be determined using the same method.

[0095] The feedback update module is also used to update the conflict memory. Specifically, the feedback update module can write the relation conflict information corresponding to the rectified conflict annotation units, the edge type of the cross-modal relation edges to be checked, the node candidate annotation results before correction, the node annotation results obtained after correction based on collaborative annotation feedback, the relation candidate annotation results before correction, and the relation annotation results obtained after correction based on collaborative annotation feedback into the conflict memory. The edge type of the cross-modal relation edges to be checked includes at least one of the following: temporal relation edge type, spatial relation edge type, and semantic relation edge type.

[0096] Historical correction records in the conflict memory can be used to generate conflict query constraints. The feedback update module can generate conflict query constraints based on historical correction records in the conflict memory that have the same edge type and the same relationship conflict information for cross-modal relationship edges to be checked. In the next round of determining the target collaborative annotation package, the collaborative annotation package generation module can merge or exclude conflict annotation units with the same edge type and the same relationship conflict information for cross-modal relationship edges to be checked based on the conflict query constraints. Through this process, the system can reduce the number of times the same conflict source repeatedly enters the manual annotation process, and make the active learning query objects more focused on the modal data nodes and cross-modal relationship edges that need to be checked.

[0097] During model retraining, the feedback update module uses node annotation results obtained after collaborative annotation feedback correction as node supervision information, relation annotation results obtained after collaborative annotation feedback correction as relation supervision information, and constructs relation counterexamples from historical correction records in the conflict memory. Subsequently, the feedback update module jointly trains the node annotation branch, relation annotation branch, and conflict propagation branch of the graph-based active learning annotation model based on the node supervision information, relation supervision information, and relation counterexamples. Through this retraining process, the graph-based active learning annotation model can continuously update node annotation, relation verification, and conflict propagation processes using collaborative annotation feedback and historical correction records from the conflict memory.

[0098] In one specific deployment method, the sample graph construction module can be located in the data access server to receive multimodal data and build a multimodal sample graph; the machine learning feature extraction module and the graph-based active learning annotation module can be located in the model server to complete graph alignment feature extraction and active learning annotation model inference; the collaborative annotation package generation module can be located in the annotation task management server to generate target collaborative annotation packages and distribute them to annotation terminals; and the feedback update module can be located in the model training server to handle collaborative annotation feedback, update the conflict memory, and retrain the graph-based active learning annotation model. These modules can also be integrated into the same computing device, with the corresponding program instructions executed by the same processor.

[0099] The system in this embodiment can be implemented by a computer program. The computer program can be stored in a non-volatile storage medium. When the processor executes the computer program, it can perform processes such as sample graph construction, graph alignment feature extraction, graph-based active learning annotation, target co-annotation package generation, co-annotation feedback processing, conflict memory update, and model retraining. The non-volatile storage medium can be a hard disk, solid-state drive, flash memory, read-only memory, or other media capable of storing program instructions.

[0100] Through the above system implementation, the sample graph construction module, machine learning feature extraction module, graph-based active learning annotation module, collaborative annotation package generation module, and feedback update module work together to enable the system to treat single-modal data and cross-modal relationships within multimodal samples as processing objects in the active learning process, and to determine the target collaborative annotation package based on node confidence information, relationship conflict information, and conflict propagation information. The system can also continuously update the graph-based active learning annotation model through collaborative annotation feedback and a conflict memory, thus forming a corresponding system implementation with the aforementioned active learning method for multimodal data collaborative annotation.

[0101] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. An active learning method for multimodal data collaborative annotation, characterized in that, The steps include the following: S1, obtain the unlabeled multimodal sample pool and the labeled sample set, and establish a sample identifier and a multimodal sample graph for each multimodal sample in the unlabeled multimodal sample pool. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. The modal data nodes correspond to single modal data in the multimodal sample, and the cross-modal relationship edges are used to represent the temporal relationship, spatial relationship or semantic relationship between different modal data corresponding to the same sample identifier. S2, input the multimodal sample graph into a machine learning feature extraction network to obtain the node features of the modal data nodes, the relation features of the cross-modal relation edges, and the graph alignment features formed by fusing the node features and the relation features; S3, based on the labeled sample set and the graph alignment features, train a graph-based active learning annotation model, so that the graph-based active learning annotation model outputs node candidate annotation results, relationship candidate annotation results, node confidence information, relationship conflict information and conflict propagation information; S4. Based on the node confidence information, the relationship conflict information, and the conflict propagation information, construct conflict annotation units from the unlabeled multimodal sample pool, and merge conflict annotation units with the same modal data nodes or the same cross-modal relationship edges to obtain the target collaborative annotation package. S5, receive collaborative annotation feedback for the target collaborative annotation package, correct the node candidate annotation results and the relationship candidate annotation results according to the collaborative annotation feedback, update the labeled sample set and the conflict memory, and retrain the graph-based active learning annotation model based on the updated labeled sample set and the conflict memory.

2. The active learning method for multimodal data collaborative annotation according to claim 1, characterized in that, In step S1, establishing the multimodal sample map includes: Establish the modal data nodes for at least two modalities of image data, text data, audio data, video data, point cloud data, and sensor time-series data, respectively; Establish time relationship edges based on the correspondence between acquisition times of data from different modalities; Establish spatial relationship edges based on the correspondence between target locations of different modal data; Establish semantic relationship edges based on object descriptions, event descriptions, or attribute descriptions between different modalities of data; The temporal relation edge, the spatial relation edge, and the semantic relation edge are used as the cross-modal relation edge.

3. The active learning method for multimodal data collaborative annotation according to claim 1, characterized in that, In step S1, the multimodal sample graph further includes node annotation status and relation annotation status; The node labeling status includes unlabeled status, model pre-labeled status, manual confirmation status, and pending review status. The relationship labeling status includes unverified status, consistent confirmation status, conflict pending status, and conflict correction status. The graph-based active learning annotation model distinguishes between modal data nodes to be annotated and cross-modal relationship edges to be verified based on the node annotation status and the relationship annotation status.

4. The active learning method for multimodal data collaborative annotation according to claim 1, characterized in that, In step S1, when there is missing modal data in the multimodal samples corresponding to the same sample identifier, a missing placeholder node is established in the multimodal sample graph, and a modal missing attribute is set for the missing placeholder node. In step S2, the machine learning feature extraction network performs masking processing on the cross-modal relationship edges connecting the missing placeholder nodes according to the modality missing attribute, and retains the sample attribution relationship formed between the missing placeholder nodes and existing modality data nodes based on the sample identifier.

5. The active learning method for multimodal data collaborative annotation according to claim 1, characterized in that, In step S2, the machine learning feature extraction network includes a node encoding layer, a relation encoding layer, a graph message passing layer, and a gated fusion layer; The node encoding layer is used to extract the node features of the modal data nodes; The relation encoding layer is used to extract the relation features of the cross-modal relation edges; The graph message passing layer transmits relation features between different modal data nodes according to the edge type of the cross-modal relation edge. The edge type of the cross-modal relation edge includes at least one of the following: temporal relation edge type, spatial relation edge type, and semantic relation edge type. The gated fusion layer generates the graph alignment feature based on the node features, the relationship features, and the edge type of the cross-modal relationship edge.

6. The active learning method for multimodal data collaborative annotation according to claim 1, characterized in that, In step S3, the graph-based active learning annotation model includes a node annotation branch, a relation annotation branch, and a conflict propagation branch; The node annotation branch outputs the candidate node annotation results and the node confidence information; The relation annotation branch outputs the candidate relation annotation results and the relation conflict information; The conflict propagation branch determines the cross-modal relationship edge that meets the preset conflict triggering condition as the conflict cross-modal relationship edge based on the relationship conflict information, and determines the modal data node and cross-modal relationship edge adjacent to the conflict cross-modal relationship edge, and outputs the conflict propagation information.

7. The active learning method for multimodal data collaborative annotation according to claim 6, characterized in that, The relationship conflict information includes at least one of the following: category conflict information, target location conflict information, time segment conflict information, semantic reference conflict information, and quantity attribute conflict information; The relation annotation branch generates the relation conflict information based on the candidate annotation results of the node corresponding to the modal data nodes at both ends of the same cross-modal relation edge.

8. The active learning method for multimodal data collaborative annotation according to claim 6, characterized in that, The conflict propagation information includes conflict source relationship edges, conflict associated nodes, conflict associated relationship edges, and conflict propagation levels; The conflict source relationship edge is one or more of the conflict cross-modal relationship edges; The conflict propagation branch starts from the conflict source relation edge, determines the conflict association relation edge among the cross-modal relation edges that share the same modal data node with the conflict source relation edge, and determines the modal data node connecting the conflict association relation edge as the conflict association node.

9. The active learning method for multimodal data collaborative annotation according to claim 8, characterized in that, In step S4, the conflict annotation unit includes a sample identifier, a focal modal data node, a cross-modal relationship edge to be checked, node candidate annotation results, relationship candidate annotation results, relationship conflict information, and annotation action to be performed. The focal modal data node is a modal data node whose node confidence information is lower than a preset node confidence threshold and belongs to the conflict-related node indicated by the conflict propagation information. The cross-modal relationship edge to be verified is the conflict association edge indicated by the conflict propagation information; When the relationship conflict information of the cross-modal relationship edge to be checked meets the preset conflict triggering condition, the cross-modal relationship edge to be checked is determined as a conflicting cross-modal relationship edge.

10. An active learning system for multimodal data collaborative annotation, characterized in that, include: The sample graph construction module is used to obtain an unlabeled multimodal sample pool and a labeled sample set, and to establish a sample identifier and a multimodal sample graph for each multimodal sample in the unlabeled multimodal sample pool. The multimodal sample graph includes modal data nodes and cross-modal relationship edges. The modal data nodes correspond to single modal data in the multimodal sample, and the cross-modal relationship edges are used to represent the temporal relationship, spatial relationship or semantic relationship between different modal data corresponding to the same sample identifier. The machine learning feature extraction module is used to input the multimodal sample graph into the machine learning feature extraction network to obtain the node features of the modal data nodes, the relation features of the cross-modal relation edges, and the graph alignment features formed by fusing the node features and the relation features; The graph-based active learning annotation module is used to train a graph-based active learning annotation model based on the labeled sample set and the graph alignment features, so that the graph-based active learning annotation model outputs node candidate annotation results, relationship candidate annotation results, node confidence information, relationship conflict information and conflict propagation information; The collaborative annotation package generation module is used to construct conflict annotation units from the unlabeled multimodal sample pool based on the node confidence information, the relationship conflict information and the conflict propagation information, and to merge conflict annotation units with the same modality data nodes or the same cross-modal relationship edges to obtain the target collaborative annotation package; The feedback update module is used to receive collaborative annotation feedback for the target collaborative annotation package, correct the node candidate annotation results and the relationship candidate annotation results according to the collaborative annotation feedback, update the labeled sample set and the conflict memory, and retrain the graph-based active learning annotation model based on the updated labeled sample set and the conflict memory.

Images