Man-machine collaborative labeling method and device based on multi-modal fusion and active learning

Abstract

The invention discloses a man-machine collaborative labeling method and device based on multi-modal fusion and active learning, and the method comprises the steps: obtaining a to-be-labeled multi-modal data set, carrying out the decoupling and alignment of multi-modal features, and obtaining a multi-modal sample; unlabeled multi-modal samples are input into the current multi-modal fusion model to obtain corresponding sample fusion representation, the fusion uncertainty of the multi-modal samples is determined according to the sample fusion representation, and candidate samples are screened out; performing cross-modal enhanced pre-annotation on the candidate samples to obtain corresponding pre-annotation results, and displaying the pre-annotation results through a human-computer interaction interface; in response to a labeling operation of a labeling person, obtaining a corresponding multi-modal label, and performing consistency verification on the multi-modal label to obtain a compatibility probability; and when the compatibility probability is smaller than a preset first threshold value, sending out a labeling error prompt through a human-computer interaction interface. According to the method, the efficiency and accuracy of multi-modal data labeling are improved, and the method can be applied to the technical field of data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a human-computer collaborative annotation method and apparatus based on multimodal fusion and active learning. Background Technology

[0002] With the widespread application of deep learning in fields such as computer vision and natural language processing, the demand for large-scale, high-quality labeled data is becoming increasingly urgent. Traditional manual labeling methods are costly, inefficient, and difficult to guarantee the consistency of labels.

[0003] Active learning (AL) prioritizes the annotation of samples with the most information, aiming to achieve optimal model performance with minimal annotation costs. However, existing active learning research largely focuses on single-modal data. For the annotation of multimodal data (such as image-text, video-audio, etc.), the following problems still exist: 1) Sample value is difficult to measure: It is difficult to uniformly assess the uncertainty and information content of heterogeneous data from different modalities; 2) The intermodal relationships are not fully utilized: Existing methods usually perform a simple weighted average of the uncertainties of each mode, failing to deeply explore the complementary and conflicting information between modes to guide sample selection; 3) The annotation process is not optimized: the annotation interface is separated from the model prediction, and the annotators cannot obtain effective intelligent assistance from the model. There is still a lot of room for improvement in annotation efficiency.

[0004] The above problems urgently need to be addressed. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, one objective of this invention is to provide a human-computer collaborative annotation method based on multimodal fusion and active learning, which improves the efficiency and accuracy of multimodal data annotation.

[0007] Another objective of this invention is to provide a human-machine collaborative annotation device based on multimodal fusion and active learning.

[0008] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a human-computer collaborative annotation method based on multimodal fusion and active learning, comprising the following steps: Obtain the multimodal dataset to be labeled, and use a pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples; The unlabeled multimodal samples are input into the current multimodal fusion model to obtain the corresponding sample fusion representation. The fusion uncertainty of the multimodal samples is determined based on the sample fusion representation, and then candidate samples are selected based on the fusion uncertainty. The candidate samples are subjected to cross-modal enhanced pre-labeling to obtain the corresponding pre-labeling results, which are then displayed through a human-computer interaction interface. In response to the annotation operation of the annotator, the corresponding multimodal label is obtained, and the consistency of the multimodal label is checked to obtain the compatibility probability of the multimodal label; When the compatibility probability is less than a preset first threshold, an error warning is issued through the human-computer interaction interface. When the compatibility probability is greater than or equal to the first threshold, the candidate sample is determined to be a labeled sample, and the current multimodal fusion model is incrementally trained using the labeled sample.

[0009] Furthermore, in one embodiment of the present invention, the step of using a pre-trained model to decouple and align multimodal features of data samples in the multimodal dataset to obtain multimodal samples specifically includes: Determine the modal dimensions of the multimodal dataset, wherein the modal dimensions include at least two of the following: image dimension, text dimension, and audio dimension; The pre-trained model is used to extract feature representations of the data samples in different modalities. The feature representation is time-aligned to obtain the multimodal sample corresponding to the data sample.

[0010] Furthermore, in one embodiment of the present invention, the step of determining the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then screening candidate samples based on the fusion uncertainty, specifically includes: The variance of the fused representation of the sample in the representation space is calculated based on the Monte Carlo method to obtain the uncertainty of the fused representation. Intermodal uncertainty is determined based on the standard deviation or variance of the uncertainty index for each mode; Calculate the cosine similarity entropy value between the fused representation of the sample and the fused representation of the labeled sample, and determine the diversity factor based on the negative value of the cosine similarity entropy value; The fusion uncertainty is obtained by weighting and summing the fusion representation uncertainty, the intermodal uncertainty, and the diversity factor according to preset weighting coefficients. The multimodal samples with the greatest fusion uncertainty are selected as candidate samples.

[0011] Furthermore, in one embodiment of the present invention, the cross-modal enhanced pre-labeling of the candidate samples specifically includes: The candidate samples are labeled by performing label prediction on the feature representations of different modalities using classification models of different modalities, and the initial pre-labels of different modalities are obtained. The initial pre-labels include at least two of the following: image pre-labels, text pre-labels, and audio pre-labels. The feature representation is combined with the initial pre-label in a cross-modal manner, and the similarity score between the feature representation and the initial pre-label in the combination is calculated; When the similarity scores of different combinations are all greater than the preset similarity threshold, cross-modal fusion is performed on the initial pre-labeling to obtain the pre-labeling result; If the similarity score of a combination is less than the similarity threshold, the initial pre-labeling is used as the pre-labeling result, and a corresponding inconsistency prompt is generated.

[0012] Furthermore, in one embodiment of the present invention, the step of performing consistency verification on the multimodal tag to obtain the compatibility probability of the multimodal tag specifically includes: Multimodal annotation samples are constructed based on the annotation information of different modalities of the multimodal labels; The multimodal labeled samples are input into a pre-trained multimodal consistency evaluation model to obtain the compatibility probability of the multimodal labels.

[0013] Furthermore, in one embodiment of the present invention, the step of issuing an error notification through the human-computer interaction interface specifically includes: Determine the conflict description of the multimodal tags; An error notification text is generated based on the conflict description, and the error notification text is displayed on the human-computer interaction interface.

[0014] Furthermore, in one embodiment of the present invention, the incremental training of the current multimodal fusion model using the labeled samples specifically includes: The labeled samples are input into the current multimodal fusion model to obtain the model recognition result; The loss value is determined based on the model recognition results and the multimodal labels; The parameters of the current multimodal fusion model are updated based on the loss value.

[0015] On the other hand, embodiments of the present invention provide a human-computer collaborative annotation device based on multimodal fusion and active learning, comprising: The data preprocessing module is used to acquire the multimodal dataset to be labeled, and to use a pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples. The sample screening module is used to input the unlabeled multimodal samples into the current multimodal fusion model to obtain the corresponding sample fusion representation, determine the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then screen out candidate samples based on the fusion uncertainty. The pre-labeling module is used to perform cross-modal enhanced pre-labeling on the candidate samples, obtain the corresponding pre-labeling results, and display them through a human-computer interaction interface; The consistency verification module is used to respond to the annotation operation of the annotator, obtain the corresponding multimodal label, perform consistency verification on the multimodal label, and obtain the compatibility probability of the multimodal label. The reminder module is used to issue a labeling error reminder through the human-computer interaction interface when the compatibility probability is less than a preset first threshold. The incremental training module is used to determine the candidate sample as a labeled sample when the compatibility probability is greater than or equal to the first threshold, and to perform incremental training on the current multimodal fusion model using the labeled sample.

[0016] On the other hand, embodiments of the present invention provide an electronic device, the electronic device including a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory. When the computer program is executed by the processor, it implements the human-computer collaborative annotation method based on multimodal fusion and active learning as described above.

[0017] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the human-computer collaborative annotation method based on multimodal fusion and active learning as described above.

[0018] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the human-computer collaborative annotation method based on multimodal fusion and active learning as described above.

[0019] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention acquires a multimodal dataset to be labeled, uses a pre-trained model to decouple and align multimodal features of data samples in the multimodal dataset to obtain multimodal samples, inputs unlabeled multimodal samples into the current multimodal fusion model to obtain corresponding sample fusion representations, determines the fusion uncertainty of multimodal samples based on the sample fusion representations, and then selects candidate samples based on the fusion uncertainty. Cross-modal augmentation pre-labeling is performed on the candidate samples to obtain the corresponding pre-labeling results, which are displayed through a human-computer interaction interface. Responding to the labeling operation of the labeler, the corresponding multimodal label is obtained, and the consistency of the multimodal label is checked to obtain the compatibility probability of the multimodal label. When the compatibility probability is less than a preset first threshold, a labeling error reminder is issued through the human-computer interaction interface. When the compatibility probability is greater than or equal to the first threshold, the candidate sample is determined to be a labeled sample, and the current multimodal fusion model is incrementally trained using the labeled sample. This invention employs active learning sample selection based on fusion uncertainty and performs cross-modal enhanced pre-annotation to obtain pre-annotation results. This allows annotators to determine multimodal labels based on these pre-annotation results, then perform consistency checks on the multimodal labels to obtain compatibility probabilities, and issue error alerts when the compatibility probability is low, thereby improving the efficiency and accuracy of multimodal data annotation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a human-computer collaborative annotation method based on multimodal fusion and active learning provided in an embodiment of the present invention. Figure 2 A flowchart of step S101 provided in an embodiment of the present invention; Figure 3 A flowchart of step S102 provided in an embodiment of the present invention; Figure 4 A flowchart of step S103 provided in an embodiment of the present invention; Figure 5 A flowchart of step S104 provided in an embodiment of the present invention; Figure 6 A flowchart of step S105 provided in an embodiment of the present invention; Figure 7 A flowchart of step S106 provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of the human-machine collaborative annotation device based on multimodal fusion and active learning provided in an embodiment of the present invention; Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] The human-computer collaborative annotation method based on multimodal fusion and active learning provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a driver state monitoring method based on machine vision, but is not limited to the above forms.

[0025] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0027] like Figure 1 The diagram shows a flowchart of a human-computer collaborative annotation method based on multimodal fusion and active learning provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a human-computer collaborative annotation method based on multimodal fusion and active learning, specifically including the following steps: S101. Obtain the multimodal dataset to be labeled, and use the pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples; S102. Input the unlabeled multimodal samples into the current multimodal fusion model to obtain the corresponding sample fusion representation. Determine the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then select candidate samples based on the fusion uncertainty. S103. Perform cross-modal enhanced pre-labeling on the candidate samples, obtain the corresponding pre-labeling results, and display them through the human-computer interaction interface; S104. In response to the annotation operation of the annotator, obtain the corresponding multimodal label, and perform consistency verification on the multimodal label to obtain the compatibility probability of the multimodal label. S105. When the compatibility probability is less than the preset first threshold, issue a labeling error reminder through the human-computer interaction interface. S106. When the compatibility probability is greater than or equal to the first threshold, the candidate sample is determined as a labeled sample, and the current multimodal fusion model is incrementally trained using the labeled sample.

[0028] Specifically, this invention employs active learning sample selection based on fusion uncertainty. By prioritizing the annotation of samples with the highest information content, it aims to achieve optimal model performance with minimal annotation cost. Based on cross-modal enhanced pre-annotation, it upgrades pre-annotation from "single-modal independent guessing" to "multi-modal cross-validation and mutual reinforcement," improving the efficiency and accuracy of multi-modal data annotation. The specific process is as follows: 1) Multimodal data preprocessing and feature extraction: Obtain unlabeled multimodal datasets and extract feature representations for each modality using pre-trained models; 2) Active learning sample selection based on fusion uncertainty: Input unlabeled samples into the current multimodal model, use the dynamic multimodal fusion engine to calculate the joint representation uncertainty of each sample, and select the most valuable batch of candidate samples by taking into account diversity factors. 3) Cross-modal intelligent pre-annotation: For the candidate samples selected in step 2), high-quality pre-annotation results (such as bounding boxes, labels, text descriptions, etc.) are generated using cross-modal prompting and enhancement algorithms. 4) Human-computer collaborative annotation and real-time verification: The pre-annotation results are presented on a dynamically adaptive annotation interface, where annotators review and correct them. At the same time, a multimodal annotation consistency verification algorithm is used to verify the annotators' modifications in real time and alert potential errors. 5) Incremental model training: Add high-quality labeled data that has been manually reviewed to the training set to perform incremental training on the multimodal model and update the model parameters; 6) Iterative loop: Repeat steps 2 to 5 until the model performance reaches the preset threshold or the annotation budget is exhausted.

[0029] Furthermore, this invention provides an "Intelligent Annotation Assistant (IAA)," which is deeply integrated with an active learning loop and includes: Data storage module: used to store the unlabeled data pool and the labeled dataset; Dynamic multimodal fusion engine: the core computing unit responsible for executing the algorithms in steps 2), 3), and 5); Human-computer interaction interface module: responsible for presenting pre-annotation results, receiving input from annotators, and displaying real-time warnings; Model Management and Incremental Training Module: Responsible for managing model versions and performing incremental learning.

[0030] It can be recognized that the embodiments of the present invention actively learn sample selection based on fusion uncertainty and perform cross-modal enhanced pre-annotation to obtain pre-annotation results, so that annotators can determine multimodal labels based on the pre-annotation results, and then perform consistency verification on the multimodal labels to obtain compatibility probabilities, and issue error reminders when the compatibility probability is low, thereby improving the efficiency and accuracy of multimodal data annotation.

[0031] like Figure 2 The diagram shown is a flowchart of step S101 provided in an embodiment of the present invention. (Refer to...) Figure 2 As an optional implementation, a pre-trained model is used to decouple and align the multimodal features of data samples in the multimodal dataset to obtain multimodal samples, specifically including: S1011. Determine the modal dimensions of the multimodal dataset. The modal dimensions include at least two of the following: image dimension, text dimension, and audio dimension. S1012. Extract feature representations of data samples in different modalities through a pre-trained model; S1013. Perform temporal alignment on the feature representation to obtain the multimodal samples corresponding to the data samples.

[0032] Specifically, the modal dimensions of the multimodal dataset are determined, including at least two of the following: image, text, and audio dimensions. For the image dimension, spatial features are extracted using ViT / ResNet, and multi-scale information is enhanced using a feature pyramid. For the text dimension, semantic vectors are generated using a BERT distillation model, and noisy words are filtered using an attention mask. For the audio dimension, the Mel spectrogram is Conv1D encoded, and then key segments are compressed using temporal pooling. Temporal alignment of the obtained feature representations of different modalities yields the multimodal samples.

[0033] like Figure 3 The diagram shown is a flowchart of step S102 provided in an embodiment of the present invention. (Refer to...) Figure 3 As an optional implementation, the fusion uncertainty of multimodal samples is determined based on the sample fusion representation, and candidate samples are then selected based on the fusion uncertainty. Specifically, this includes: S1021. Calculate the variance of the sample fusion representation in the representation space based on the Monte Carlo method to obtain the uncertainty of the fusion representation; S1022. Determine the intermodal uncertainty based on the standard deviation or variance of the uncertainty index of each mode; S1023. Calculate the cosine similarity entropy value between the fused representation of the sample and the fused representation of the labeled sample, and determine the diversity factor based on the negative value of the cosine similarity entropy value. S1024. The fusion uncertainty is obtained by weighting and summing the fusion representation uncertainty, intermodal uncertainty, and diversity factor according to the preset weight coefficients. S1025. Select several multimodal samples with the greatest fusion uncertainty as candidate samples.

[0034] Specifically, the core of active learning sample selection based on fusion uncertainty lies in calculating the priority score of sample i. The calculation formula is as follows:

[0035] in, To address the uncertainty in the fusion representation, a joint representation of the samples is obtained through a multimodal fusion model (such as Transformer). The variance is calculated in this representation space using Monte Carlo methods. To address intermodal uncertainty, calculate the standard deviation or variance of each modal uncertainty index (such as prediction entropy). The larger the value, the greater the discrepancy in the judgments of the different modal models regarding the sample, and the higher its value. For the diversity factor, calculate The cosine similarity entropy value, which is fused with all samples in the labeled sample pool, is used as a diversity factor by taking its negative value to ensure that samples with large differences from the labeled data are selected; α, β, and γ are learnable or preset weight coefficients used to balance the contributions of the three factors.

[0036] The embodiments of the present invention can more accurately identify "hard samples" that are not only difficult to classify themselves, but also have conflicting information between different modalities and cannot be effectively integrated by the model, thereby improving the selection efficiency of active learning from the source.

[0037] like Figure 4 The diagram shown is a flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, cross-modal enhanced pre-labeling is performed on candidate samples, specifically including: S1031. Using classification models of different modalities, label prediction is performed on the feature representations of candidate samples in different modalities to obtain initial pre-labels for different modalities. The initial pre-labels include at least two of image pre-labels, text pre-labels, and audio pre-labels. S1032. Combine the feature representation with the initial pre-label across modalities, and calculate the similarity score between the combined feature representation and the initial pre-label. S1033. When the similarity scores of different combinations are all greater than the preset similarity threshold, cross-modal fusion is performed on the initial pre-labeling to obtain the pre-labeling results. S1034. When there is a combination whose similarity score is less than the similarity threshold, the initial pre-labeling is used as the pre-labeling result, and the corresponding inconsistency prompt is generated.

[0038] Specifically, different modal classification models are used to predict the labels of the feature representations of candidate samples in different modalities to obtain initial pre-labels for different modalities. The feature representations and initial pre-labels are then combined across modalities, and the similarity score between the combined feature representations and initial pre-labels is calculated.

[0039] Taking image-text pairs (I, T) as an example, image pre-annotation is generated using both visual and text models. and text pre-annotation Then use image I and text pre-annotation Perform cross-modal combination, combining image I and text Input a pre-trained vision-language model (such as CLIP), calculate its similarity score, and simultaneously, from... Extract entity keywords. If the similarity is high, add attributes to the visual detection result (e.g., "dog") using keywords from the text (e.g., "brown"); if the similarity is low, trigger a warning, prompting the annotator to pay close attention to the inconsistency of the sample.

[0040] The embodiments of the present invention upgrade the pre-annotation from "single-modal independent guessing" to "multi-modal cross-validation and mutual reinforcement", which significantly improves the accuracy and richness of pre-annotation and further reduces the correction burden of annotators.

[0041] like Figure 5 The diagram shown is a flowchart of step S104 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, a consistency check is performed on the multimodal tags to obtain the compatibility probability of the multimodal tags, which specifically includes: S1041. Construct multimodal annotation samples based on the annotation information of different modalities of multimodal labels; S1042. Input the multimodal labeled samples into the pre-trained multimodal consistency evaluation model to obtain the compatibility probability of the multimodal labels.

[0042] like Figure 6 The diagram shown is a flowchart of step S105 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, error notifications can be issued via a human-computer interaction interface, specifically including: S1051. Determine the conflict description of multimodal tags; S1052. Generate annotation error reminder text based on the conflict description, and display the annotation error reminder text on the human-computer interaction interface.

[0043] Specifically, obtain a set of multimodal labels after the annotators have completed the annotation. The model constructs L as a multimodal labeled sample and inputs it into another pre-trained multimodal consistency evaluation model to calculate the compatibility probability P(compatible | L). A set threshold τ is obtained; if P < τ, it is determined that there is a potential conflict in the labeled group, and a prompt box automatically pops up on the interface, listing the conflict points (e.g., "The text description is 'quiet,' but the audio tag contains 'explosion sound,' please confirm").

[0044] This invention provides real-time, online annotation quality assistance to prevent error accumulation. It transforms quality control from "post-event sampling" to "real-time assistance during the process," utilizing AI as a "second pair of eyes" for annotation personnel to fundamentally eliminate errors caused by negligence and significantly improve annotation quality.

[0045] like Figure 7 The diagram shown is a flowchart of step S106 provided in an embodiment of the present invention. (Refer to...) Figure 9 As an optional implementation, the current multimodal fusion model can be incrementally trained using labeled samples, specifically including: S1061. Input the labeled samples into the current multimodal fusion model to obtain the model recognition results; S1062. Determine the loss value based on the model recognition results and multimodal labels; S1063. Update the parameters of the current multimodal fusion model based on the loss value.

[0046] This invention extends active learning from a single "sample selection" strategy to a comprehensive human-machine collaborative annotation framework covering the entire process of "pre-annotation-suggestion-interaction-quality inspection." It dynamically optimizes the annotation interface and workflow based on model uncertainty, reducing efficiency losses caused by modality switching. Through a "disputed sample" arbitration mechanism, it transforms the knowledge and judgment of human experts into system rules, achieving self-evolution of annotation quality and solving the industry problem of annotation consistency. It establishes a real-time feedback loop of "annotation-instant training-better pre-annotation," giving the system the unique ability to become faster and more accurate with each annotation.

[0047] The method flow of the embodiments of the present invention has been described above. It can be understood that the embodiments of the present invention actively learn sample selection based on fusion uncertainty and perform cross-modal enhanced pre-annotation to obtain pre-annotation results. This allows annotators to determine multimodal labels based on the pre-annotation results, and then perform consistency verification on the multimodal labels to obtain compatibility probabilities. Error alerts are given when the compatibility probability is low, thereby improving the efficiency and accuracy of multimodal data annotation.

[0048] Compared with the prior art, the embodiments of the present invention have the following advantages: 1) Improved annotation efficiency: This embodiment of the invention integrates the prediction results of various modalities to generate a fused pre-annotation suggestion with the highest confidence, which is then directly presented on the annotation interface. The annotator's work is transformed from "annotating from scratch" to "reviewing and correcting model suggestions," greatly reducing mechanical operations.

[0049] 2) Optimizes human-computer interaction and has a certain degree of practicality.

[0050] 3) Improve annotation quality: When significant inconsistencies occur in the correction results, these samples will be automatically marked as "disputed samples".

[0051] 4) Real-time evolution: After each batch of samples is reviewed and corrected, it is immediately used for incremental training of the model.

[0052] like Figure 8 The diagram shown is a structural schematic of a human-machine collaborative annotation device based on multimodal fusion and active learning provided in an embodiment of the present invention. (Refer to...) Figure 8 This invention provides a human-computer collaborative annotation device based on multimodal fusion and active learning, comprising: The data preprocessing module is used to obtain the multimodal dataset to be labeled, and to use a pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples. The sample selection module is used to input unlabeled multimodal samples into the current multimodal fusion model to obtain the corresponding sample fusion representation, determine the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then select candidate samples based on the fusion uncertainty. The pre-labeling module is used to perform cross-modal enhanced pre-labeling on candidate samples, obtain the corresponding pre-labeling results, and display them through a human-computer interaction interface; The consistency verification module is used to respond to the annotation operations of the annotators, obtain the corresponding multimodal labels, perform consistency verification on the multimodal labels, and obtain the compatibility probability of the multimodal labels. The reminder module is used to issue a labeling error reminder through the human-computer interaction interface when the compatibility probability is less than a preset first threshold. The incremental training module is used to determine the candidate sample as a labeled sample when the compatibility probability is greater than or equal to the first threshold, and to perform incremental training on the current multimodal fusion model using the labeled sample.

[0053] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0054] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned human-computer collaborative annotation method based on multimodal fusion and active learning. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0055] like Figure 9 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 9 This invention provides an electronic device, comprising: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the human-computer collaborative annotation method based on multimodal fusion and active learning according to the embodiments of this invention. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0056] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0057] like Figure 10 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 10 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1001, which can be executed by one or more processors to implement the above-mentioned human-computer collaborative annotation method based on multimodal fusion and active learning.

[0058] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0059] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the aforementioned human-computer collaborative annotation method based on multimodal fusion and active learning.

[0060] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0061] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0063] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0064] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0065] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0066] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0068] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0069] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0070] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0071] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0072] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A human-computer collaborative annotation method based on multimodal fusion and active learning, characterized in that, Includes the following steps: Obtain the multimodal dataset to be labeled, and use a pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples; The unlabeled multimodal samples are input into the current multimodal fusion model to obtain the corresponding sample fusion representation. The fusion uncertainty of the multimodal samples is determined based on the sample fusion representation, and then candidate samples are selected based on the fusion uncertainty. The candidate samples are subjected to cross-modal enhanced pre-labeling to obtain the corresponding pre-labeling results, which are then displayed through a human-computer interaction interface. In response to the annotation operation of the annotator, the corresponding multimodal label is obtained, and the consistency of the multimodal label is checked to obtain the compatibility probability of the multimodal label; When the compatibility probability is less than a preset first threshold, an error warning is issued through the human-computer interaction interface. When the compatibility probability is greater than or equal to the first threshold, the candidate sample is determined to be a labeled sample, and the current multimodal fusion model is incrementally trained using the labeled sample.

2. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The process of using a pre-trained model to decouple and align multimodal features of data samples in the multimodal dataset to obtain multimodal samples specifically includes: Determine the modal dimensions of the multimodal dataset, wherein the modal dimensions include at least two of the following: image dimension, text dimension, and audio dimension; The pre-trained model is used to extract feature representations of the data samples in different modalities. The feature representation is time-aligned to obtain the multimodal sample corresponding to the data sample.

3. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The step of determining the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then selecting candidate samples based on the fusion uncertainty, specifically includes: The variance of the fused representation of the sample in the representation space is calculated based on the Monte Carlo method to obtain the uncertainty of the fused representation. Intermodal uncertainty is determined based on the standard deviation or variance of the uncertainty index for each mode; Calculate the cosine similarity entropy value between the fused representation of the sample and the fused representation of the labeled sample, and determine the diversity factor based on the negative value of the cosine similarity entropy value; The fusion uncertainty is obtained by weighting and summing the fusion representation uncertainty, the intermodal uncertainty, and the diversity factor according to preset weighting coefficients. The multimodal samples with the greatest fusion uncertainty are selected as candidate samples.

4. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The cross-modal enhanced pre-labeling of the candidate samples specifically includes: The candidate samples are labeled by performing label prediction on the feature representations of different modalities using classification models of different modalities, and the initial pre-labels of different modalities are obtained. The initial pre-labels include at least two of the following: image pre-labels, text pre-labels, and audio pre-labels. The feature representation is combined with the initial pre-label in a cross-modal manner, and the similarity score between the feature representation and the initial pre-label in the combination is calculated; When the similarity scores of different combinations are all greater than the preset similarity threshold, cross-modal fusion is performed on the initial pre-labeling to obtain the pre-labeling result; If the similarity score of a combination is less than the similarity threshold, the initial pre-labeling is used as the pre-labeling result, and a corresponding inconsistency prompt is generated.

5. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The process of performing consistency verification on the multimodal tags to obtain the compatibility probability of the multimodal tags specifically includes: Multimodal annotation samples are constructed based on the annotation information of different modalities of the multimodal labels; The multimodal labeled samples are input into a pre-trained multimodal consistency evaluation model to obtain the compatibility probability of the multimodal labels.

6. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The step of issuing error notifications through the human-computer interaction interface specifically includes: Determine the conflict description of the multimodal tags; An error notification text is generated based on the conflict description, and the error notification text is displayed on the human-computer interaction interface.

7. The human-computer collaborative annotation method based on multimodal fusion and active learning according to claim 1, characterized in that, The incremental training of the current multimodal fusion model using the labeled samples specifically includes: The labeled samples are input into the current multimodal fusion model to obtain the model recognition result; The loss value is determined based on the model recognition results and the multimodal labels; The parameters of the current multimodal fusion model are updated based on the loss value.

8. A human-computer collaborative annotation device based on multimodal fusion and active learning, characterized in that, include: The data preprocessing module is used to acquire the multimodal dataset to be labeled, and to use a pre-trained model to decouple and align the multimodal features of the data samples in the multimodal dataset to obtain multimodal samples. The sample screening module is used to input the unlabeled multimodal samples into the current multimodal fusion model to obtain the corresponding sample fusion representation, determine the fusion uncertainty of the multimodal samples based on the sample fusion representation, and then screen out candidate samples based on the fusion uncertainty. The pre-labeling module is used to perform cross-modal enhanced pre-labeling on the candidate samples, obtain the corresponding pre-labeling results, and display them through a human-computer interaction interface; The consistency verification module is used to respond to the annotation operation of the annotator, obtain the corresponding multimodal label, perform consistency verification on the multimodal label, and obtain the compatibility probability of the multimodal label. The reminder module is used to issue a labeling error reminder through the human-computer interaction interface when the compatibility probability is less than a preset first threshold. The incremental training module is used to determine the candidate sample as a labeled sample when the compatibility probability is greater than or equal to the first threshold, and to perform incremental training on the current multimodal fusion model using the labeled sample.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the computer program is executed by the processor, it implements the human-machine collaborative annotation method based on multimodal fusion and active learning as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the human-computer collaborative annotation method based on multimodal fusion and active learning as described in any one of claims 1 to 7.

Images