Data annotation method and device, electronic equipment and storage medium
Through the multimodal feature fusion, the problems of high cost and low efficiency of multimodal data annotation in the existing technology are solved, and the accuracy of automated annotation and labeling results are improved.
Patent Information
- Application Number
- CN202411865893.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-29
AI Technical Summary
In the prior art, multimodal data annotation requires manual operation, which is costly and inefficient, and cannot fully consider other modal data, which can easily cause labeling errors.
By obtaining the data to be marked and determining its annotation type and output format, using the preset speech model, text model and visual model for feature extraction, obtaining speech, text and visual modal features, and then fusion modal features are generated through multimodal features, and finally generating the annotation result based on the annotation type and output format.
Automatic labeling of multimodal data is realized, and the accuracy of labeling results is improved through multimodal feature fusion, cost is reduced, and labeling efficiency is improved.
Smart Images
Figure CN119903339A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data annotation, and in particular to a data annotation method and device, electronic equipment, and storage medium. Background Art
[0002] In the rapidly developing field of artificial intelligence, data labeling, as the basis for training machine learning and deep learning algorithms, plays a vital role in improving model performance. Whether in industry or academia, appropriately labeled data plays an important role in market development and the advancement of scientific research.
[0003] The data annotation methods in the related art generally rely on manual operations to complete data classification and annotation; they usually process data of different modalities separately, such as using independent tools to annotate text, voice and image data respectively.
[0004] However, since the related technology requires not only manual labeling to complete multimodal data labeling, but also the labeling personnel need to be familiar with the use of each tool respectively, the cost of labeling is high and the labeling efficiency is poor. Summary of the invention
[0005] In view of the above problems, a data labeling method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:
[0006] A data labeling method, the method comprising:
[0007] Acquire the data to be annotated, and determine the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type;
[0008] Determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model, and a preset visual model to obtain a first speech modality feature, a first text modality feature, and a first visual modality feature;
[0009] Obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature, and obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature, and the shared feature;
[0010] According to the annotation type, the output format and the fusion modality feature, an annotation result of the data to be annotated is obtained.
[0011] Optionally, obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature includes:
[0012] The first speech modality feature, the first text modality feature, and the first visual modality feature are merged to obtain the shared feature.
[0013] Optionally, obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature includes:
[0014] Under the condition that the shared feature is used as a query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature and a second visual modality feature;
[0015] The fused modality feature is obtained according to the second speech modality feature, the second text modality feature and the second visual modality feature.
[0016] Optionally, obtaining the fused modality feature according to the second speech modality feature, the second text modality feature and the second visual modality feature includes:
[0017] Extracting the second speech modality feature, the second text modality feature, and the second visual modality feature in sequence through a progressive fusion network and a self-attention mechanism to obtain a third speech modality feature, a third text modality feature, and a third visual modality feature;
[0018] According to the third speech modality feature, the third text modality feature and the third visual modality feature, an original fused modality feature is obtained, and the original fused modality feature is orthogonalized to obtain the fused modality feature.
[0019] Optionally, before extracting features from the data set file using a preset voice model, a preset text model, and a preset visual model, the method further includes:
[0020] Perform data enhancement processing on the dataset file.
[0021] Optionally, the merging the first speech modality feature, the first text modality feature, and the first visual modality feature to obtain the shared feature includes:
[0022] The first speech modality feature, the first text modality feature and the first visual modality feature are merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain the shared feature.
[0023] Optionally, the first speech modality feature, the first text modality feature, and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature, and a second visual modality feature, including:
[0024] Through any one of cross-attention mechanism fusion, weighted fusion, feature pooling fusion and feature transformation fusion, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain the second speech modality feature, the second text modality feature and the second visual modality feature.
[0025] A data labeling device, comprising:
[0026] A module for acquiring data to be annotated, used to acquire data to be annotated, and determine the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type;
[0027] A feature extraction module, used to determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model and a preset visual model to obtain a first speech modality feature, a first text modality feature and a first visual modality feature;
[0028] A fusion modality feature acquisition module, used to obtain a shared feature according to the first speech modality feature, the first text modality feature and the first visual modality feature, and to obtain a fusion modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature;
[0029] The annotation result generating module is used to obtain the annotation result of the data to be annotated according to the annotation type, the output format and the fusion modality feature.
[0030] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the data labeling method as described above when executed by the processor.
[0031] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data labeling method as described above is implemented.
[0032] A computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the data labeling method as described above.
[0033] The embodiments of the present invention have the following advantages:
[0034] The present invention provides a data annotation method, which obtains the data to be annotated and determines the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type and an image type; then, according to the annotation type of the data to be annotated, obtains a data set file of the data to be annotated, and extracts features of the data set file through a preset voice model, a preset text model and a preset visual model, so as to obtain a first voice modality feature, a first text modality feature and a first visual modality feature; thereby, according to the first voice modality feature, the first text modality feature and the first visual modality feature, a shared feature is obtained, and according to the first voice modality feature, the first text modality feature, the first visual modality feature and the shared feature, a fused modality feature is obtained; and then, according to the annotation type, the output format and the fused modality feature, the annotation result of the data to be annotated is obtained. The automatic annotation of multimodal data is realized, and through the fusion of multimodal features, the understanding of the data by the annotation result generation model is further improved, thereby improving the accuracy of data annotation, without the need for manual multimodal annotation process, reducing the cost of data annotation, and improving the annotation efficiency of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0036] Figure 1 is a flowchart of steps of a data labeling method provided by some embodiments of the present invention;
[0037] Figure 2 is an example diagram of a product interface provided by some embodiments of the present invention;
[0038] Figure 3 is an example diagram of an execution flow of a data labeling method provided by some embodiments of the present invention;
[0039] Figure 4 is a structural schematic diagram of a data labeling device provided by some embodiments of the present invention;
[0040] Figure 5is a block diagram of an electronic device provided in some embodiments of the present invention;
[0041] Figure 6 is a schematic diagram of a computer-readable medium provided in some embodiments of the present invention. DETAILED DESCRIPTION
[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. This is only for the purpose of clearly describing the technical solutions of the embodiments of the present application, and shall not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.
[0044] In the embodiments of the present application, the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0045] In the rapidly developing field of artificial intelligence, data labeling, as the basis for training machine learning and deep learning algorithms, plays a vital role in improving model performance. Whether in industry or academia, appropriately labeled data plays an important role in market development and the advancement of scientific research.
[0046] The data annotation methods in the related art generally rely on manual operations to complete data classification and annotation; they usually process data of different modalities separately, such as using independent tools to annotate text, voice and image data respectively.
[0047] However, since the relevant technology requires not only manual labeling to complete multimodal data labeling, but also the labeling personnel need to be familiar with the use of each tool respectively, the cost of labeling is high and the labeling efficiency is poor. In addition, since other modal data cannot be fully considered when labeling data, it is often only possible to focus on one dimension of the data, which easily leads to labeling errors.
[0048] In order to solve the above problems, the present invention is based on the core technical concept of realizing data annotation by fusing multimodal features, and improves the data annotation method in the related art. The present invention will be described in detail below with reference to the accompanying drawings:
[0049] Reference Figure 1 , shows a flowchart of a data labeling method provided by some embodiments of the present invention, which may specifically include the following steps:
[0050] Step 101, obtaining data to be annotated, and determining the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type;
[0051] In the specific implementation, Figure 2 As shown in the figure, you can select the annotation type, data set and output format on the product interface based on the specific business type. The annotation type itself can include voice type, text type and image type, and the selected annotation type should be one or more of voice type, text type and image type during the specific selection; the data set is the data set file corresponding to the corresponding annotation type, for example, the voice data set is usually an audio file, such as WAV, MP3, etc.; the text data set is usually a txt, csv, xml or json file, etc.; the image data set is usually a picture file, such as JPEG, PNG, etc. The output format includes all the formats of the annotation type, such as json, txt, csv, etc.
[0052] Step 102, determining a data set file of the data to be annotated according to the annotation type of the data to be annotated, and performing feature extraction on the data set file through a preset speech model, a preset text model, and a preset visual model to obtain a first speech modality feature, a first text modality feature, and a first visual modality feature;
[0053] In the specific implementation, after selecting the annotation type according to the data to be annotated on the product interface, the data set file corresponding to the corresponding annotation type can be determined. For example, the voice data set is usually an audio file, such as WAV, MP3, etc.; the text data set is usually a txt, csv, xml or json file, etc.; the image data set is usually a picture file, such as JPEG, PNG, etc. The output format includes all the formats of the annotation type, such as json, txt, csv, etc. Since the selected annotation type should be one or more of the voice type, text type and image type during the specific selection, the corresponding data set file can also be confirmed according to the corresponding logic to cover a variety of different application scenarios.
[0054] Further, such as Figure 3As shown, the data set file can be used as input data into the pre-trained preset voice model, preset text model and preset visual model, so as to extract features from the data set file and obtain the first voice modality feature, the first text modality feature and the first visual modality feature. It should be emphasized that in the visual model, the extracted features can be of three scales to facilitate the labeling of objects of different scales. In the voice and text models, the extracted features can be of a single scale.
[0055] In some embodiments of the present invention, before extracting features from the data set file using a preset voice model, a preset text model, and a preset visual model, the method further includes:
[0056] Perform data enhancement processing on the dataset file.
[0057] In practical applications, after determining the data type and data set annotated on the product interface, data enhancement preprocessing can be performed on the data set to make it meet the input requirements of the subsequent network model.
[0058] For example, when the dataset file is insufficient, dataset generation processing can be performed, that is, the jsonl format can be generated through a script, where each line represents an instance, and the instance has at least one of the text name, voice name, and image name information. If there is no such information, it is represented by NULL. The dataset data name should be consistent with the name in the jsonl file.
[0059] Furthermore, the model's generalization ability for different modalities can be enhanced through multimodal data augmentation technology, especially in modalities with scarce data, for example:
[0060] A multimodal data enhancement framework can be built, which can include three parts: speech generation, text generation, and image generation. For the speech generation part, the generator can use an architecture similar to WaveNet, and the discriminator can use a convolutional neural network (CNN) (the loss function can be binary cross entropy loss). For speech, the interface of CycleGAN-VC2 can also be encapsulated (the loss function can be adversarial loss, cycle consistency loss, mapping consistency loss, and second adversarial loss).
[0061] The image generator can use the StyleGAN architecture, the discriminator can use PatchGAN (the loss function can be feature matching loss and binary cross entropy loss), and the image model can also encapsulate the StyleGAN-XL interface (the loss function can be adversarial loss and feature matching loss).
[0062] The text generator can use the TextGAN architecture, and the discriminator can use a Transformer-based model (the loss function can be binary cross entropy loss, diversity loss, and adversarial loss). The TextGAN interface can also be encapsulated for text (the loss function can be adversarial loss or diversity loss).
[0063] Based on the above content, users can choose different solutions to implement data enhancement according to actual needs, or they can proceed directly to subsequent processes without performing data enhancement.
[0064] Step 103, obtaining a shared feature according to the first speech modality feature, the first text modality feature and the first visual modality feature, and obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature;
[0065] In the specific implementation, Figure 3 As shown, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged and output through a multimodal attention mechanism to obtain a shared feature. Specifically, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain a shared feature.
[0066] Furthermore, under the condition that the shared feature is used as the query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, the first speech modality feature, the first text modality feature and the first visual modality feature are fused with the shared feature respectively by using the cross-attention mechanism fusion method to obtain the second speech modality feature, the second text modality feature and the second visual modality feature; when the first speech modality feature, the first text modality feature and the first visual modality feature are fused with the shared feature respectively, they can be selectively weighted according to the importance of the task, and the default weights are all 1. At the same time, in addition to the cross-attention mechanism fusion, weighted fusion, feature pooling fusion, feature transformation fusion, local and shared feature fusion strategies, etc. can also be used in the fusion strategy.
[0067] Furthermore, the second speech modality feature, the second text modality feature and the second visual modality feature can be extracted in sequence through the progressive fusion network and the self-attention mechanism, and the more advanced features of each modality can be extracted through the progressive fusion network, so as to obtain the third speech modality feature, the third text modality feature and the third visual modality feature;
[0068] Then, the third speech modality feature, the third text modality feature and the third visual modality feature can be concatenated, and after concatenation, the attention mechanism is used to perform feature fusion to obtain the original fused modality feature, and then the original fused modality feature can be orthogonalized to obtain the fused modality feature. Among them, orthogonalization is mainly used to reduce the redundancy between different features in multimodal learning and enhance the generalization ability of the model; in addition, when the attention mechanism is used to perform feature fusion after concatenation, selective weighting can also be performed according to the importance of the task, and the default weights can all be 1.
[0069] In some embodiments of the present invention, obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature includes:
[0070] The first speech modality feature, the first text modality feature, and the first visual modality feature are merged to obtain the shared feature.
[0071] In a specific implementation, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged and output through a multimodal attention mechanism to obtain a shared feature. Specifically, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain a shared feature.
[0072] In some embodiments of the present invention, obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature includes:
[0073] Under the condition that the shared feature is used as a query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature and a second visual modality feature;
[0074] The fused modality feature is obtained according to the second speech modality feature, the second text modality feature and the second visual modality feature.
[0075] In practical applications, the shared features can be used as query vectors, and the first speech modality features, the first text modality features and the first visual modality features can be used as key-value pairs. The first speech modality features, the first text modality features and the first visual modality features can be fused with the shared features by using a cross-attention mechanism fusion method to obtain the second speech modality features, the second text modality features and the second visual modality features; thus, the fused modality features can be obtained according to the second speech modality features, the second text modality features and the second visual modality features; wherein, when the first speech modality features, the first text modality features and the first visual modality features are fused with the shared features respectively, they can be selectively weighted according to the importance of the task, and the default weights are all 1. At the same time, in addition to the cross-attention mechanism fusion, weighted fusion, feature pooling fusion, feature transformation fusion, local and shared feature fusion strategies, etc. can also be used in the fusion strategy.
[0076] In some embodiments of the present invention, obtaining the fused modality feature according to the second speech modality feature, the second text modality feature and the second visual modality feature includes:
[0077] Extracting the second speech modality feature, the second text modality feature, and the second visual modality feature in sequence through a progressive fusion network and a self-attention mechanism to obtain a third speech modality feature, a third text modality feature, and a third visual modality feature;
[0078] According to the third speech modality feature, the third text modality feature and the third visual modality feature, an original fused modality feature is obtained, and the original fused modality feature is orthogonalized to obtain the fused modality feature.
[0079] In a specific implementation, the second speech modality feature, the second text modality feature, and the second visual modality feature can be extracted in sequence through a progressive fusion network and a self-attention mechanism, and a more advanced feature of each modality can be extracted through a progressive fusion network, thereby obtaining a third speech modality feature, a third text modality feature, and a third visual modality feature;
[0080] Then, the third speech modality feature, the third text modality feature and the third visual modality feature can be spliced, and after splicing, the attention mechanism is used to perform feature fusion to obtain the original fused modality feature, and then the original fused modality feature can be orthogonalized to obtain the fused modality feature. Among them, orthogonalization is mainly used to reduce the redundancy between different features in multimodal learning and enhance the generalization ability of the model; in addition, when the attention mechanism is used for feature fusion after splicing, it can also be selectively weighted according to the importance of the task, and the default weight can be 1; thereby, hierarchical feature integration is achieved through progressive fusion, and the model's understanding of complex data (multiple modalities) is enhanced. In addition, through sequential processing (i.e., the second speech modality feature, the second text modality feature and the second visual modality feature are extracted in turn through a progressive fusion network and a self-attention mechanism; wherein the self-attention mechanism is mainly used to integrate modality information) and orthogonalization, the model can enhance the extraction effect of modality-specific information.
[0081] In some embodiments of the present invention, the step of merging the first speech modality feature, the first text modality feature, and the first visual modality feature to obtain the shared feature includes:
[0082] The first speech modality feature, the first text modality feature and the first visual modality feature are merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain the shared feature.
[0083] In practical applications, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged and output through a multimodal attention mechanism to obtain a shared feature. Specifically, the first speech modality feature, the first text modality feature and the first visual modality feature can be merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain a shared feature.
[0084] In some embodiments of the present invention, the first speech modality feature, the first text modality feature, and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature, and a second visual modality feature, including:
[0085] Through any one of cross-attention mechanism fusion, weighted fusion, feature pooling fusion and feature transformation fusion, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain the second speech modality feature, the second text modality feature and the second visual modality feature.
[0086] In a specific implementation, the shared feature can be used as a query vector, and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs. The first speech modality feature, the first text modality feature and the first visual modality feature are fused with the shared feature by using a cross-attention mechanism fusion method to obtain the second speech modality feature, the second text modality feature and the second visual modality feature; when the first speech modality feature, the first text modality feature and the first visual modality feature are fused with the shared feature, they can be selectively weighted according to the importance of the task, and the default weight can be 1. At the same time, in addition to the cross-attention mechanism fusion, weighted fusion, feature pooling fusion, feature transformation fusion, local and shared feature fusion strategies, etc. can also be used in the fusion strategy, so as to effectively extract text, image and speech features and improve the accuracy of annotation.
[0087] Step 104: Obtain the annotation result of the data to be annotated according to the annotation type, the output format and the fusion modality feature.
[0088] In practical applications, the obtained fused modal features can be sent to the network corresponding to the determined annotation type for predictive annotation according to the previously confirmed annotation type and output format, and the results consistent with the output format, such as JSON files, can be returned, thereby automatically completing the overall annotation process of multimodal data.
[0089] In general, the data annotation method provided by the above embodiment of the present invention first extracts features from speech, text, and image respectively and outputs shared features through a multimodal attention mechanism; then, a progressive fusion method and an orthogonal sequence fusion method are respectively adopted, and dynamic weight allocation is combined with a multi-task learning framework to output corresponding annotation results for text, image, and sound respectively. The model is enabled to process image, speech, and text data containing multiple scales and complex backgrounds; a single model processes multiple tasks, reducing the cost of model development and maintenance; dynamic weight allocation ensures that resources are effectively utilized and improves the performance of the model on specific tasks; multi-task learning and adaptive weight allocation enhance the model's generalization ability for different types of data. It also reduces the intermediate steps in the traditional data processing process, and automated annotation reduces manual intervention, improving production efficiency and annotation speed; end-to-end training also allows the model to better generalize to new and unseen data, simplifying the entire development process from data collection to annotation results, without manual intervention, and can significantly improve data annotation efficiency and accuracy.
[0090] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0091] Reference Figure 4 , shows a schematic diagram of the structure of a data labeling device provided by some embodiments of the present invention; specifically, it may include the following modules:
[0092] The to-be-annotated data acquisition module 401 is used to acquire the to-be-annotated data and determine the annotation type and output format of the to-be-annotated data; wherein the annotation type includes one or more of a voice type, a text type and an image type;
[0093] A feature extraction module 402 is used to determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model, and a preset visual model to obtain a first speech modality feature, a first text modality feature, and a first visual modality feature;
[0094] A fusion modality feature acquisition module 403 is used to obtain a shared feature according to the first speech modality feature, the first text modality feature and the first visual modality feature, and to obtain a fusion modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature;
[0095] The annotation result generating module 404 is used to obtain the annotation result of the data to be annotated according to the annotation type, the output format and the fusion modality feature.
[0096] In some embodiments of the present invention, the fusion modality feature acquisition module 403 includes:
[0097] The shared feature acquisition submodule is used to merge the first speech modality feature, the first text modality feature and the first visual modality feature to obtain the shared feature.
[0098] In some embodiments of the present invention, the fusion modality feature acquisition module 403 includes:
[0099] a feature fusion submodule, for fusing the first speech modality feature, the first text modality feature and the first visual modality feature with the shared feature respectively, under the condition that the shared feature is used as a query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, to obtain a second speech modality feature, a second text modality feature and a second visual modality feature;
[0100] The fused modal feature acquisition submodule is used to obtain the fused modal feature according to the second speech modal feature, the second text modal feature and the second visual modal feature.
[0101] In some embodiments of the present invention, the fusion modality feature acquisition submodule includes:
[0102] A feature extraction unit, configured to extract the second speech modality feature, the second text modality feature, and the second visual modality feature in sequence through a progressive fusion network and a self-attention mechanism to obtain a third speech modality feature, a third text modality feature, and a third visual modality feature;
[0103] The fused modal feature acquisition unit is used to obtain the original fused modal feature according to the third speech modal feature, the third text modal feature and the third visual modal feature, and to perform orthogonal processing on the original fused modal feature to obtain the fused modal feature.
[0104] In some embodiments of the present invention, the device further comprises:
[0105] The data enhancement processing module is used to perform data enhancement processing on the data set file.
[0106] In some embodiments of the present invention, the shared feature acquisition submodule includes:
[0107] A shared feature acquisition unit is used to merge the first speech modality feature, the first text modality feature and the first visual modality feature through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain the shared feature.
[0108] In some embodiments of the present invention, the feature fusion submodule includes:
[0109] A feature fusion unit is used to fuse the first speech modality feature, the first text modality feature and the first visual modality feature with the shared feature respectively through any one of cross-attention mechanism fusion, weighted fusion, feature pooling fusion and feature transformation fusion to obtain the second speech modality feature, the second text modality feature and the second visual modality feature.
[0110] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above data labeling method when executed by a processor.
[0111] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.
[0112] Memory 503, used for storing computer programs;
[0113] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:
[0114] Acquire the data to be annotated, and determine the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type;
[0115] Determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model, and a preset visual model to obtain a first speech modality feature, a first text modality feature, and a first visual modality feature;
[0116] Obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature, and obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature, and the shared feature;
[0117] According to the annotation type, the output format and the fusion modality feature, an annotation result of the data to be annotated is obtained.
[0118] In an optional embodiment of the present invention, obtaining a shared feature according to the first speech modality feature, the first text modality feature and the first visual modality feature includes:
[0119] The first speech modality feature, the first text modality feature, and the first visual modality feature are merged to obtain the shared feature.
[0120] In an optional embodiment of the present invention, obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature includes:
[0121] Under the condition that the shared feature is used as a query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature and a second visual modality feature;
[0122] The fused modality feature is obtained according to the second speech modality feature, the second text modality feature and the second visual modality feature.
[0123] In an optional embodiment of the present invention, obtaining the fused modality feature according to the second speech modality feature, the second text modality feature and the second visual modality feature includes:
[0124] Extracting the second speech modality feature, the second text modality feature, and the second visual modality feature in sequence through a progressive fusion network and a self-attention mechanism to obtain a third speech modality feature, a third text modality feature, and a third visual modality feature;
[0125] According to the third speech modality feature, the third text modality feature and the third visual modality feature, an original fused modality feature is obtained, and the original fused modality feature is orthogonalized to obtain the fused modality feature.
[0126] In an optional embodiment of the present invention, before extracting features from the data set file using a preset voice model, a preset text model, and a preset visual model, the method further includes:
[0127] Perform data enhancement processing on the dataset file.
[0128] In an optional embodiment of the present invention, the step of merging the first speech modality feature, the first text modality feature, and the first visual modality feature to obtain the shared feature includes:
[0129] The first speech modality feature, the first text modality feature and the first visual modality feature are merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain the shared feature.
[0130] In an optional embodiment of the present invention, the first speech modality feature, the first text modality feature, and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature, and a second visual modality feature, including:
[0131] Through any one of cross-attention mechanism fusion, weighted fusion, feature pooling fusion and feature transformation fusion, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain the second speech modality feature, the second text modality feature and the second visual modality feature.
[0132] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0133] The communication interface is used for communication between the above terminal and other devices.
[0134] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0135] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0136] like Figure 6 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 601 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the data labeling method described in the above embodiment.
[0137] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0138] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0139] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0140] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0144] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.
[0145] The above provides a detailed introduction to a data annotation method and device, electronic device, and storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data labeling method, characterized in that: The method comprises: Acquire the data to be annotated, and determine the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type; Determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model, and a preset visual model to obtain a first speech modality feature, a first text modality feature, and a first visual modality feature; Obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature, and obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature, and the shared feature; According to the annotation type, the output format and the fusion modality feature, an annotation result of the data to be annotated is obtained.
2. The method according to claim 1, characterized in that The step of obtaining a shared feature according to the first speech modality feature, the first text modality feature, and the first visual modality feature includes: The first speech modality feature, the first text modality feature, and the first visual modality feature are merged to obtain the shared feature.
3. The method according to claim 2, characterized in that The step of obtaining a fused modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature includes: Under the condition that the shared feature is used as a query vector and the first speech modality feature, the first text modality feature and the first visual modality feature are used as key-value pairs, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain a second speech modality feature, a second text modality feature and a second visual modality feature; The fused modality feature is obtained according to the second speech modality feature, the second text modality feature and the second visual modality feature.
4. The method according to claim 3, characterized in that The step of obtaining the fused modality feature according to the second speech modality feature, the second text modality feature and the second visual modality feature includes: Extracting the second speech modality feature, the second text modality feature, and the second visual modality feature in sequence through a progressive fusion network and a self-attention mechanism to obtain a third speech modality feature, a third text modality feature, and a third visual modality feature; According to the third speech modality feature, the third text modality feature and the third visual modality feature, an original fused modality feature is obtained, and the original fused modality feature is orthogonalized to obtain the fused modality feature.
5. The method according to claim 1, characterized in that Before extracting features from the data set file using a preset voice model, a preset text model, and a preset visual model, the method further includes: Perform data enhancement processing on the dataset file.
6. The method according to claim 2, characterized in that The step of merging the first speech modality feature, the first text modality feature, and the first visual modality feature to obtain the shared feature includes: The first speech modality feature, the first text modality feature and the first visual modality feature are merged through any one of multimodal attention mechanism fusion, deep balanced multimodal fusion and linear fusion to obtain the shared feature.
7. The method according to claim 3, characterized in that The step of fusing the first speech modality feature, the first text modality feature, and the first visual modality feature with the shared feature to obtain a second speech modality feature, a second text modality feature, and a second visual modality feature comprises: Through any one of cross-attention mechanism fusion, weighted fusion, feature pooling fusion and feature transformation fusion, the first speech modality feature, the first text modality feature and the first visual modality feature are respectively fused with the shared feature to obtain the second speech modality feature, the second text modality feature and the second visual modality feature.
8. A data labeling device, characterized in that: The device comprises: A module for acquiring data to be annotated, used to acquire data to be annotated, and determine the annotation type and output format of the data to be annotated; wherein the annotation type includes one or more of a voice type, a text type, and an image type; A feature extraction module, used to determine a data set file of the data to be annotated according to the annotation type of the data to be annotated, and perform feature extraction on the data set file through a preset speech model, a preset text model and a preset visual model to obtain a first speech modality feature, a first text modality feature and a first visual modality feature; A fusion modality feature acquisition module, used to obtain a shared feature according to the first speech modality feature, the first text modality feature and the first visual modality feature, and to obtain a fusion modality feature according to the first speech modality feature, the first text modality feature, the first visual modality feature and the shared feature; The annotation result generating module is used to obtain the annotation result of the data to be annotated according to the annotation type, the output format and the fusion modality feature.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data labeling method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data labeling method according to any one of claims 1 to 7 is implemented.