Multi-modal data labeling method and device, storage medium, equipment and product

Abstract

The invention relates to the technical field of data annotation, and particularly provides a multi-modal data annotation method and device, a storage medium, equipment and a product, and the method can comprise the steps: mapping data in each modal in multi-modal data to a same feature space, and obtaining a feature vector in each modal, the multi-modal data comprises image data, text data and audio data; determining a to-be-fused feature vector in each mode by evaluating the correlation degree between the feature vectors in each mode; fusing the to-be-fused feature vectors in each mode to obtain a fused feature vector; the fusion feature vector is labeled, a labeling result is obtained, and the labeling result comprises a data type label and / or a data description text. According to some embodiments of the invention, accurate labeling of cross-modal data can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of data annotation, and in particular to a method, apparatus, storage medium, device and product for multimodal data annotation. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, more and more application scenarios require the annotation of large-scale multimodal data (such as text, images, audio, and video).

[0003] At present, when annotating data, high-precision automatic image annotation can be achieved through pre-trained models (such as ResNet, Inception, etc.). Natural language processing models based on deep learning (such as BERT, GPT, etc.) can achieve text annotation in tasks such as text classification, named entity recognition, and sentiment analysis. However, these methods can only annotate single-modal data and cannot handle complex multi-modal data annotation tasks.

[0004] Therefore, how to provide a technical solution for an efficient multimodal data annotation method has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of some embodiments of the present application is to provide a method, apparatus, storage medium, device and product for multimodal data annotation. Through the technical solutions of the embodiments of the present application, accurate annotation of multimodal data can be achieved with high practicality.

[0006] In a first aspect, some embodiments of the present application provide a method for labeling multimodal data, comprising: mapping data under each modality in the multimodal data to the same feature space, and obtaining a feature vector under each modality, wherein the multimodal data includes: image data, text data, and audio data; determining the feature vector to be fused under each modality by evaluating the correlation between the feature vectors under each modality; fusing the feature vectors to be fused under each modality to obtain a fused feature vector; labeling the fused feature vector to obtain a labeling result, wherein the labeling result includes: a data type label and / or a data description text.

[0007] Some embodiments of the present application map the data under each modality to the same feature space to obtain a feature vector, then determine the feature vector to be fused under each modality by correlation, then fuse the feature vector to be fused to obtain a fused feature vector; finally, annotate the fused feature vector to obtain an annotation result. The embodiments of the present application can realize automatic and accurate annotation of data under different modalities, reduce labor costs, and have strong versatility and scalability.

[0008] In some embodiments, before mapping the data under each modality in the multimodal data to the same feature space, the method further includes: performing pixel and size adjustment on the original image data to obtain the image data; performing word segmentation and filtering operations on the original text data to obtain the text data; and performing noise reduction and frame division operations on the original audio data to obtain the audio data.

[0009] Some embodiments of the present application obtain corresponding image data, text data, and audio data by preprocessing the original multimodal data, thereby eliminating data interference and improving the quality of subsequent annotation.

[0010] In some embodiments, mapping the data under each modality in the multimodal data to the same feature space to obtain the feature vector under each modality includes: converting the image data into a visual feature vector through an image encoder; converting the text data into a semantic feature vector aligned with the visual feature vector through a text encoder; and converting the audio data into an audio feature vector aligned with the visual feature vector.

[0011] Some embodiments of the present application provide support for subsequent accurate and efficient fusion by converting data in different modes into corresponding feature vectors.

[0012] In some embodiments, the feature vectors to be fused in each modality are determined by evaluating the correlation between the feature vectors in each modality, including: using an attention mechanism to obtain key features in the feature vectors in each modality; calculating the correlation between the key features; and taking the feature vector corresponding to the correlation greater than a correlation threshold as the feature vector to be fused.

[0013] Some embodiments of the present application can improve the accuracy and quality of data annotation by evaluating the correlation between key features under each modality and screening out feature vectors to be fused.

[0014] In some embodiments, labeling the fused feature vector and obtaining the labeling result includes: inputting the fused feature vector into a pre-trained classifier to obtain the data type label; and / or inputting the fused feature vector into a pre-trained generation model to obtain the data description text.

[0015] Some embodiments of the present application realize automatic labeling of fused feature vectors through pre-trained classifiers and pre-trained generation models, and the labeling efficiency is relatively high.

[0016] In some embodiments, after obtaining the labeling result, the method further includes: if it is confirmed that the confidence of the labeling result is greater than a preset threshold, outputting the labeling result; if it is confirmed that the confidence of the labeling result is not greater than the preset threshold, reviewing and verifying the labeling result to obtain feedback information; wherein the feedback information is used to adjust the parameters of the pre-trained classifier and the pre-trained generation model.

[0017] Some embodiments of the present application further verify the annotation results by comparing the confidence level of the annotation results with a preset threshold, thereby ensuring the annotation quality and subsequent annotation accuracy.

[0018] In a second aspect, some embodiments of the present application provide a device for labeling multimodal data, comprising: a feature extraction module, used to map data under each modality in the multimodal data to the same feature space, and obtain a feature vector under each modality, wherein the multimodal data includes: image data, text data and audio data; a correlation evaluation module, used to determine the feature vector to be fused under each modality by evaluating the correlation between the feature vectors under each modality; a fusion module, used to fuse the feature vectors to be fused under each modality, and obtain a fused feature vector; a labeling module, used to label the fused feature vector, and obtain a labeling result, wherein the labeling result includes: a data type label and / or a data description text.

[0019] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0020] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement a method as described in any embodiment of the first aspect when executing the program.

[0021] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A system diagram for multimodal data annotation provided for some embodiments of the present application;

[0024] Figure 2 One of the flow charts of the method for multimodal data annotation provided in some embodiments of the present application;

[0025] Figure 3 A second flowchart of a method for labeling multimodal data provided in some embodiments of the present application;

[0026] Figure 4 A block diagram of a device for multimodal data annotation provided in some embodiments of the present application;

[0027] Figure 5 A schematic diagram of an electronic device is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0029] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0030] In the related technology, manual labeling is the main method of early multimodal data labeling. Although manual labeling can handle complex semantic understanding tasks and has high accuracy in certain specific situations, its limitations are obvious. For example, manual labeling requires processing data one by one, especially when facing large-scale data, the labeling efficiency is extremely low. Moreover, the labeling process is time-consuming and often cannot meet the rapidly developing data needs. It can be seen that manual labeling not only takes a lot of time, but also requires a lot of human resources and is costly. The subjective judgment and experience levels of different annotators vary, which can easily lead to inconsistencies and errors in labeling results. In cases involving multimodal data, such as the association labeling of images and texts, manual labeling faces great challenges and it is difficult to achieve efficient and accurate labeling.

[0031] With the advancement of deep learning technology, deep learning models have begun to be used to automate the annotation of multimodal data. In terms of single-modal data annotation, deep learning models have achieved remarkable results. For example, convolutional neural networks (CNNs) are widely used in tasks such as image classification and object detection. Pre-trained models can achieve high-precision automatic image annotation. Natural language processing models based on deep learning perform well in tasks such as text classification, named entity recognition, and sentiment analysis, and can efficiently annotate text data. However, these methods usually only target a single modality and cannot handle complex multimodal data annotation tasks.

[0032] In view of this, some embodiments of the present application provide a method for multimodal data annotation, which pre-processes the data under each mode in the multimodal data, and then uses a pre-trained large model to map the processed data to the same feature space to obtain a feature vector under each mode. After that, the feature vectors under these different modes are fused after correlation evaluation to obtain a fused feature vector; finally, the fused feature vector is annotated to obtain the annotation result. Some embodiments of the present application greatly reduce manual participation and significantly improve the annotation efficiency. It supports annotation tasks of multiple data modes, such as text, images, etc., and can handle complex correlations between modes, and is suitable for a variety of application scenarios. Utilizing the pre-training capabilities of the large model, the annotation results are more accurate, and consistent annotation across modalities can be achieved to reduce human errors. By reducing the participation of manual annotation, the time and labor costs of large-scale data annotation are significantly reduced.

[0033] The following is combined with Figure 1 The overall composition structure of the multimodal data annotation system provided by some embodiments of the present application is exemplified.

[0034] like Figure 1 As shown, some embodiments of the present application provide a system for multimodal data annotation, which may include: a terminal 100 and an annotation server 200. The terminal 100 may send multimodal data to the annotation server 200. The annotation server 200 may pre-process the multimodal data to obtain processed data; use a pre-deployed pre-trained large model to analyze the processed data to determine feature vectors in the same feature space under different modalities; then fuse the feature vectors to be fused after the correlation evaluation of the feature vectors to obtain a fused feature vector; finally, annotate the fused feature vector to obtain an annotation result.

[0035] In some embodiments of the present application, the terminal 100 may be a mobile terminal or a non-portable computer terminal, which is not specifically limited in the embodiments of the present application. It should be noted that the annotation server 200 is pre-deployed with a pre-trained large model (such as image and text joint learning model CLIP, ViLBERT, etc.) and a pre-trained classifier that meets the annotation requirements, a pre-trained generation model, etc. Specifically, it can be deployed according to the actual application scenario, which is not specifically limited in the embodiments of the present application.

[0036] The following is combined with Figure 2 The implementation process of multimodal data annotation performed by the annotation server 200 provided in some embodiments of the present application is exemplified.

[0037] Please see attached Figure 2 , Figure 2 A flow chart of a method for multimodal data annotation provided for some embodiments of the present application.

[0038] In some embodiments of the present application, the method for multimodal data annotation includes, before executing S210: performing pixel and size adjustment on the original image data to obtain the image data; performing word segmentation and filtering operations on the original text data to obtain the text data; performing noise reduction and frame division operations on the original audio data to obtain the audio data.

[0039] For example, in some embodiments of the present application, first, the multimodal data input module obtains the original text data, original image data and original audio data related to the action. Among them, the original text data can be natural language texts such as action descriptions, instructions or scene annotations. The original image data can be a single frame image or a video frame, for example, capturing visual information of the action execution. The original audio data can be background sound or command audio during the action. The analysis of data under different modes can be used to further improve the semantic richness of action generation. After obtaining the above-mentioned original data, in order to improve the accuracy of annotation, the data preprocessing module needs to preprocess the original data. Specifically, in order to make the image data meet the input requirements of the pre-trained large model when used later. The original image data is normalized, that is, the pixel value of the image is scaled to a fixed range (for example, [0,1]); then the original image data is cropped and scaled to ensure that the image has a suitable size, such as (512, 512). The original text data needs to be segmented, stop words are removed, and word embedding is processed so that the text data can be subsequently input into the encoder, for example, the natural language text is decomposed into words or subword units. The original audio data is subjected to noise reduction, frame division and other operations to obtain audio data.

[0040] The above implementation process is exemplarily described below. The method for multimodal data annotation may include:

[0041] S210, mapping the data in each mode of the multimodal data to the same feature space to obtain a feature vector in each mode, wherein the multimodal data includes: image data, text data, and audio data.

[0042] For example, in some embodiments of the present application, the feature extraction module uses contrastive learning to map data in different modalities to the same feature space through a pre-trained large model CLIP for subsequent fusion and analysis. In other embodiments, data in different modalities (such as images, texts, and audio) are embedded into the same feature space by a joint embedding space method. This method usually relies on a joint learning framework to achieve inter-modal alignment through a common embedding space. For example, a shared embedding space is created, and image, text, and audio data are mapped into this space so that similar images, texts, and audios have close embedding representations, thereby obtaining feature vectors in different modalities. Alternatively, a model is trained using a contrastive learning method so that images, texts, and audios with the same semantics are closer in the embedding space, and images, texts, and audios with different semantics are farther away. It should be understood that the embodiments of the present application are not limited thereto.

[0043] In some embodiments of the present application, S210 may include: converting the image data into a visual feature vector through an image encoder; converting the text data into a semantic feature vector aligned with the visual feature vector through a text encoder; and converting the audio data into an audio feature vector aligned with the visual feature vector.

[0044] For example, in some embodiments of the present application, the image encoder uses a model such as ViT (Vision Transformer) to convert image data into a visual feature vector of a high-dimensional vector. The image encoder can understand key information in the image data, such as action details, through a visual attention mechanism. The text encoder uses a Transformer-based architecture, inputs text data into the Transformer architecture, and converts it into a vector representation aligned with image features (i.e., a visual feature vector), i.e., a semantic feature vector. For example, 3D human motion information (as a specific example of text data) is encoded into a semantic feature vector, i.e., by estimating the 2D key points of the human body and fitting a 3D human body model (SMPL-X), high-precision features of whole-body motion are extracted. The audio data is vectorized based on a pre-trained large model to obtain an audio feature vector.

[0045] S220, determining the feature vectors to be fused in each modality by evaluating the correlation between the feature vectors in each modality.

[0046] For example, in some embodiments of the present application, similar images, texts, and audios will obtain similar feature representations, while unrelated images, texts, and audios will obtain distant feature representations. For action annotation, when the user inputs an action description (such as "a person is running"), the system's multimodal feature fusion module will input the image, text, and audio at the same time, generate corresponding feature vectors, and then determine whether the image, text, and audio match, determine the correlation between the three, and then screen out feature vectors to be fused that have a certain correlation for subsequent fusion.

[0047] In some embodiments of the present application, S220 may include: using an attention mechanism to obtain key features in the feature vector under each modality; calculating the correlation between the key features; and taking the feature vector corresponding to the correlation greater than the correlation threshold as the feature vector to be fused.

[0048] For example, in some embodiments of the present application, through the multi-head attention mechanism in the pre-trained large model, the large model can identify key features in different modalities and calculate the degree of correlation between them (as a specific example of correlation). The co-attention mechanism can establish a connection between image, text, and audio features to ensure cross-modal consistency in action annotation. The feature vector to be fused is determined by comparing the degree of correlation with the set correlation threshold. For example, it is calculated that the degree of correlation of the three key features is 0.8, and the correlation threshold is 0.7. At this time, the feature vector corresponding to the three correlation features can be used as the feature vector to be fused.

[0049] When obtaining the pre-trained large model, the weights of each modality are dynamically adjusted during the training process, so that the system can automatically adjust the attention weight according to the importance of the data of different modalities. For example, in the action labeling task, image information is more critical than text and audio, so more weight can be assigned to the image modality. The pre-trained large model of this application should not only optimize the prediction accuracy of a single modality, but also ensure that the correlation between different modalities is fully utilized. Therefore, the loss function in the training process of this application combines the traditional cross entropy loss with the multimodal correlation loss. The definition of the loss function includes: the traditional cross entropy loss, which is used to evaluate the gap between the predicted label and the true label. Single-modal loss, which is used to evaluate the prediction results of images, texts, and audios respectively. For the image part, the mean square error (MSE) is used to measure the distance between the image features and the label. Multimodal correlation loss, by calculating the cosine similarity between the image, text, and audio feature vectors, evaluates the correlation between the features in the three modalities.

[0050] S230, fusing the feature vectors to be fused in each mode to obtain a fused feature vector.

[0051] For example, in some embodiments of the present application, the multimodal fusion module integrates the feature vectors to be fused from different modalities. Specifically, by fusing the 3D key points of the whole body action (as the feature vector to be fused in the image modality), the text description (as the feature vector to be fused in the text modality) and the sound / command description (as the feature vector to be fused in the audio modality), a unified action feature description is generated, that is, a fused comprehensive vector is obtained (as a specific example of a fused feature vector). The comprehensive vector contains both the visual information of the image, the semantic information of the text, and the sound information.

[0052] S240, annotate the fused feature vector to obtain an annotation result, wherein the annotation result includes: a data type label and / or a data description text.

[0053] For example, in some embodiments of the present application, the fused comprehensive vector is passed to a labeling module to output a labeling result.

[0054] In some embodiments of the present application, S240 may include: inputting the fused feature vector into a pre-trained classifier to obtain the data type label; and / or inputting the fused feature vector into a pre-trained generation model to obtain the data description text.

[0055] For example, in some embodiments of the present application, the annotation module uses a pre-trained classifier (e.g., a classification network based on Transformer) to identify the action type. For example, after inputting the integrated vector of the running image, text description, and sound information fusion, the pre-trained classifier generates the action label "running" (as a specific example of a data type label). Using a pre-trained generative model (e.g., ChatGPT), a more detailed action description can be further generated, such as "a person running at high speed, accompanied by a waving motion" (as a specific example of a data description text).

[0056] For the pre-trained classifiers mentioned above, the input layer receives the comprehensive vector output by the multimodal fusion module, and the dimension of the comprehensive vector is consistent with the fusion feature. Use the fully connected layer (Dense Layer) or convolution layer (ConvLayer) to perform preliminary fusion feature vector transformation and extract high-level feature representation. Introduce Batch Normalization and Dropout technology to prevent overfitting and improve the robustness of the model. After the feature transformation, add a multi-head attention mechanism to make the model focus on the key features of different modalities and automatically adjust the weight of each modality. This ensures that important information will not be ignored during the classification process of the classifier. The pre-trained classifier designs a multi-layer output structure. The first layer outputs a rough classification (for example, the action categories are divided into "movement", "stationary", etc.), and the second layer outputs specific action labels (such as "running", "jumping", etc.). This hierarchical structure can help large models gradually focus on more fine-grained classification targets.

[0057] Considering the lack of data, we use the pre-trained model CLIP for transfer learning. The weights of the pre-trained model are initialized into the custom classifier, and then fine-tuned to adapt the classifier to the new action labeling task.

[0058] In some embodiments of the present application, after executing S240, the method of multimodal data annotation may include: if it is confirmed that the confidence of the annotation result is greater than a preset threshold, outputting the annotation result; if it is confirmed that the confidence of the annotation result is not greater than the preset threshold, reviewing and verifying the annotation result to obtain feedback information; wherein the feedback information is used to adjust the parameters of the pre-trained classifier and the pre-trained generation model.

[0059] For example, in some embodiments of the present application, the output module will decide whether to directly output the annotation result or submit it to the manual verification module based on the confidence of the annotation result. If the model determines that the annotation result has a high confidence level (i.e., greater than a preset threshold) by calculating the confidence level of the annotation result, the annotation result is directly output. If the confidence level is low or the annotation result involves a complex scenario, the system will submit the annotation result to the manual verification module for review. Based on the feedback information of the reviewed annotation results, the parameters of the pre-trained classifier or pre-trained generation model are dynamically adjusted to improve the annotation accuracy in subsequent annotation tasks.

[0060] The following is combined with Figure 3 The specific process of multimodal data annotation provided by some embodiments of the present application is exemplified.

[0061] Please see attached Figure 3 , Figure 3 A flow chart of a method for multimodal data annotation provided for some embodiments of the present application.

[0062] The above process is explained below as an example.

[0063] S310, acquiring multimodal data related to human body movements.

[0064] S320, preprocessing the original data in each mode of the multimodal data to obtain data in each mode.

[0065] S330, mapping the data under each mode to the same feature space to obtain the feature vector under each mode.

[0066] S340, determining the feature vectors to be fused in each modality by evaluating the correlation between the feature vectors in each modality.

[0067] S350, fusing the feature vectors to be fused in each mode to obtain a fused feature vector.

[0068] S360, input the fused feature vector into a pre-trained classifier to obtain an action type label; input the fused feature vector into a pre-trained generation model to obtain an action description text.

[0069] It should be noted that the specific implementation process of S310 to S360 can refer to the method embodiment provided above, and in order to avoid repetition, the detailed description is appropriately omitted here.

[0070] Through some of the above embodiments of the present application, it can be known that the present application adopts fully automated multimodal data annotation, and by introducing a large-scale pre-trained model, fully automated annotation of multimodal data such as text, images and audio is achieved, reducing manual intervention. The present application can effectively process and annotate data involving multiple modalities, and ensure the consistency and relevance of annotations between different modalities. The method of the present application is not only applicable to the currently popular image, text and audio annotation, but can also be extended to other types of multimodal data annotation tasks, such as video. Moreover, through the present application, users can quickly and accurately complete multimodal data annotation tasks in multiple fields and scenarios, significantly improving production efficiency and data annotation quality.

[0071] Please refer to Figure 4 , Figure 4 The block diagram of the composition of the device for multimodal data annotation provided by some embodiments of the present application is shown. It should be understood that the device for multimodal data annotation corresponds to the above method embodiment and can perform each step involved in the above method embodiment. The specific functions of the device for multimodal data annotation can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0072] Figure 4The multimodal data annotation device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the multimodal data annotation device, and the multimodal data annotation device includes: a feature extraction module 410, which is used to map the data under each mode in the multimodal data to the same feature space to obtain the feature vector under each mode, wherein the multimodal data includes: image data, text data and audio data; a correlation evaluation module 420, which is used to determine the feature vector to be fused under each mode by evaluating the correlation between the feature vectors under each mode; a fusion module 430, which is used to fuse the feature vectors to be fused under each mode to obtain a fused feature vector; a labeling module 440, which is used to label the fused feature vector to obtain a labeling result, wherein the labeling result includes: a data type label and / or a data description text.

[0073] In some embodiments of the present application, before the feature extraction module 410, the multimodal data annotation device also includes: a preprocessing module (not shown in the figure) used to: perform pixel and size adjustment on the original image data to obtain the image data; perform word segmentation and filtering operations on the original text data to obtain the text data; perform noise reduction and frame division operations on the original audio data to obtain the audio data.

[0074] In some embodiments of the present application, the feature extraction module 410 is used to convert the image data into a visual feature vector through an image encoder; convert the text data into a semantic feature vector aligned with the visual feature vector through a text encoder; and convert the audio data into an audio feature vector aligned with the visual feature vector.

[0075] In some embodiments of the present application, the correlation evaluation module 420 is used to use the attention mechanism to obtain the key features in the feature vector under each modality; calculate the correlation between the key features; and use the feature vector corresponding to the correlation greater than the correlation threshold as the feature vector to be fused.

[0076] In some embodiments of the present application, the labeling module 440 is used to input the fused feature vector into a pre-trained classifier to obtain the data type label; and / or input the fused feature vector into a pre-trained generation model to obtain the data description text.

[0077] In some embodiments of the present application, the labeling module 440 is used to output the labeling result if it is confirmed that the confidence of the labeling result is greater than a preset threshold; if it is confirmed that the confidence of the labeling result is not greater than the preset threshold, the labeling result is audited and verified to obtain feedback information; wherein the feedback information is used to adjust the parameters of the pre-trained classifier and the pre-trained generation model.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0079] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0080] Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any of the above methods provided in the above embodiments.

[0081] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 can implement a method as described in any of the above embodiments when reading the program from the memory 510 through a bus 530 and executing the program.

[0082] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0083] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0084] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for labeling multimodal data, characterized in that: include: Mapping data in each mode of the multimodal data to the same feature space to obtain a feature vector in each mode, wherein the multimodal data includes: image data, text data, and audio data; By evaluating the correlation between the feature vectors in each mode, the feature vector to be fused in each mode is determined; Fusing the feature vectors to be fused under each mode to obtain a fused feature vector; The fused feature vector is labeled to obtain a labeling result, wherein the labeling result includes: a data type label and / or a data description text.

2. The method according to claim 1, characterized in that Before mapping the data in each mode of the multimodal data to the same feature space, the method further includes: Perform pixel and size adjustment on the original image data to obtain the image data; Performing word segmentation and filtering operations on the original text data to obtain the text data; The original audio data is subjected to noise reduction and frame division operations to obtain the audio data.

3. The method according to claim 2, characterized in that Mapping the data in each mode of the multimodal data to the same feature space to obtain the feature vector in each mode includes: Converting the image data into a visual feature vector by an image encoder; Converting the text data into a semantic feature vector aligned with the visual feature vector by a text encoder; The audio data is converted into an audio feature vector aligned with the visual feature vector.

4. The method according to claim 2 or 3, characterized in that The step of determining the feature vector to be fused in each mode by evaluating the correlation between the feature vectors in each mode includes: Using the attention mechanism to obtain key features in the feature vector under each modality; Calculating the correlation between the key features; The feature vector corresponding to the correlation degree being greater than the correlation threshold is used as the feature vector to be fused.

5. The method according to any one of claims 1 to 3, characterized in that The step of labeling the fused feature vector to obtain a labeling result includes: Inputting the fused feature vector into a pre-trained classifier to obtain the data type label; and / or, The fused feature vector is input into a pre-trained generation model to obtain the data description text.

6. The method according to claim 5, characterized in that After obtaining the annotation result, the method further includes: If it is confirmed that the confidence of the annotation result is greater than a preset threshold, outputting the annotation result; If it is confirmed that the confidence of the annotation result is not greater than the preset threshold, the annotation result is reviewed and verified to obtain feedback information; wherein the feedback information is used to adjust the parameters of the pre-trained classifier and the pre-trained generation model.

7. A device for multimodal data annotation, characterized in that: include: A feature extraction module, used to map the data in each mode of the multimodal data to the same feature space to obtain a feature vector in each mode, wherein the multimodal data includes: image data, text data and audio data; A correlation evaluation module is used to determine the feature vectors to be fused in each mode by evaluating the correlation between the feature vectors in each mode; A fusion module, used for fusing the feature vectors to be fused under each mode to obtain a fused feature vector; The labeling module is used to label the fused feature vector and obtain a labeling result, wherein the labeling result includes: a data type label and / or a data description text.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program executes the method according to any one of claims 1 to 6 when being run by the processor.

10. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

Images