Data annotation method and device, medium and product

Abstract

The invention provides a data annotation method and device, a medium and a product, relates to the technical field of artificial intelligence, and aims to solve the problems that generalization ability is insufficient and data annotation cannot be accurately performed in a general technology. The data annotation method comprises the steps of obtaining to-be-annotated data; and determining a to-be-selected annotation result of the to-be-annotated data based on the multi-modal large model, and correcting the to-be-selected annotation result based on a pre-constructed annotation database to obtain a target annotation result of the to-be-annotated data. Wherein the multi-modal large model is obtained by training multi-modal data, and the multi-modal data is obtained by integrating training data of multiple modals.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data annotation method, device, medium and product. Background Art

[0002] With the continuous development of artificial intelligence technology, multi-modal data (e.g., text data, image data, voice data, video data) are increasingly used in machine learning of artificial intelligence technology. Data annotation (e.g., adding labels to image data, marking the location of individual information in the image) is a key step in machine learning, especially deep learning, and has a crucial impact on model performance. Therefore, the demand for annotation of multi-modal data is growing.

[0003] General technology can use simple fusion strategies to process multi-modal data, and train deep learning models with the processed multi-modal data to perform data annotation based on the deep learning model.

[0004] However, general technology adopts a simple fusion strategy that cannot fully utilize the complementary information between different modal data, resulting in the loss of data information of different modal data. Especially in vertical fields where data is more professional and complex, general technology lacks generalization ability and cannot accurately label data. Summary of the invention

[0005] The embodiments of the present disclosure provide a data labeling method, device, medium and product, which are intended to solve the problem that general technologies have insufficient generalization capabilities and cannot accurately label data.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a data labeling method is provided, including: obtaining data to be labeled; determining candidate labeling results of the data to be labeled based on a multimodal big model; the multimodal big model is obtained by training multimodal data; the multimodal data is obtained by integrating training data of multiple modalities; based on a pre-constructed labeling database, the candidate labeling results are corrected to obtain target labeling results of the data to be labeled.

[0008] Optionally, the data labeling method also includes: obtaining multiple training sets; each training set includes training data of multiple modalities with an associated relationship; the training data of multiple modalities includes at least two of the following: text data, image data, audio data, video data; integrating the training data of multiple modalities in the training set into multimodal data; and training a large multimodal model based on the multimodal data.

[0009] Optionally, the training data of multiple modalities in the training set are integrated into multimodal data, including: determining the weight of the training data of each modality according to the data quality of the training data of each modality in the training data of multiple modalities; the data quality is positively correlated with the weight; the data quality is used to represent the proportion of data noise in the training data; the data noise is used to represent the interference data in the training data; and according to the weight of the training data of each modality, the training data of multiple modalities are integrated into multimodal data.

[0010] Optionally, the data labeling method also includes: obtaining data noise of the training data; determining the data quality of the training data based on the data noise; and the data noise is negatively correlated with the data quality.

[0011] Optionally, based on the weight of the training data of each modality, the training data of multiple modalities are integrated into multimodal data, including: extracting feature data of the training data of each modality; determining the sum of the products of the feature data and the weight as the feature data of the multimodal data; and determining the multimodal data based on the feature data of the multimodal data.

[0012] Optionally, a multimodal large model is trained based on the multimodal data, including: inputting the multimodal data into the multimodal large model to obtain predicted labeling results of the multimodal data, and obtaining actual labeling results of the multimodal data; determining the loss value of the multimodal large model based on the predicted labeling results and the actual labeling results, and iteratively training the multimodal large model based on the loss value and the multimodal data until the model convergence conditions are met to obtain the multimodal large model.

[0013] Optionally, the data labeling method also includes: adjusting model parameters of the multimodal large model according to the target labeling result.

[0014] In a second aspect, a data labeling device is provided, which includes: a communication unit and a processing unit; the communication unit is used to obtain the data to be labeled; the processing unit is used to determine the candidate labeling results of the data to be labeled based on the multimodal large model; the multimodal large model is obtained by training with multimodal data; the multimodal data is obtained by integrating training data of multiple modalities; the processing unit is also used to correct the candidate labeling results based on a pre-constructed labeling database to obtain a target labeling result of the data to be labeled.

[0015] Optionally, the communication unit is also used to obtain multiple training sets; each training set includes training data of multiple modalities with an associated relationship; the training data of multiple modalities includes at least two of the following: text data, image data, audio data, and video data; the processing unit is also used to integrate the training data of multiple modalities in the training set into multimodal data; the processing unit is also used to train a multimodal large model based on the multimodal data.

[0016] Optionally, the processing unit is specifically used to: determine the weight of the training data of each modality according to the data quality of the training data of each modality in the training data of multiple modalities; the data quality is positively correlated with the weight; the data quality is used to represent the proportion of data noise in the training data; the data noise is used to represent the interference data in the training data; according to the weight of the training data of each modality, integrate the training data of multiple modalities into multimodal data.

[0017] Optionally, the communication unit is further used to obtain data noise of the training data; the processing unit is further used to determine the data quality of the training data based on the data noise; the data noise is negatively correlated with the data quality.

[0018] Optionally, the processing unit is specifically used to: extract feature data of the training data of each modality; determine the sum of the products of the feature data and the weight as feature data of the multimodal data; and determine the multimodal data based on the feature data of the multimodal data.

[0019] Optionally, the processing unit is specifically used to: input the multimodal data into the multimodal large model to obtain the predicted labeling results of the multimodal data, and obtain the actual labeling results of the multimodal data; determine the loss value of the multimodal large model according to the predicted labeling results and the actual labeling results, and iteratively train the multimodal large model based on the loss value and the multimodal data until the model convergence conditions are met to obtain the multimodal large model.

[0020] Optionally, the processing unit is further used to adjust model parameters of the multimodal large model according to the target labeling result.

[0021] In a third aspect, a data labeling device is provided, comprising a memory and a processor; the memory is used to store computer execution instructions, and the processor is connected to the memory through a bus; when the data labeling device is running, the processor executes the computer execution instructions stored in the memory, so that the data labeling device performs the data labeling method of the first aspect.

[0022] The data labeling device may be an electronic device, or a part of an electronic device, such as a chip system in an electronic device. The chip system is used to support the electronic device to implement the functions involved in the first aspect and any possible implementation thereof, for example, to obtain and determine the data and / or information involved in the above-mentioned data labeling method. The chip system includes a chip, and may also include other discrete devices or circuit structures.

[0023] In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium comprising computer execution instructions, and when the computer execution instructions are executed on a computer, the computer executes the data labeling method described in the first aspect.

[0024] In a fifth aspect, a computer program product is also provided, which includes a computer program or instructions. When the computer instructions are executed on a data labeling device, the data labeling device executes the data labeling method as described in the first aspect above.

[0025] It should be noted that the above-mentioned computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the data labeling device, or may be packaged separately from the processor of the data labeling device, which is not limited in the embodiments of the present application.

[0026] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect.

[0027] In the embodiments of the present application, the name of the above-mentioned data annotation device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. For example, the receiving unit may also be called a receiving module, a receiver, etc. As long as the functions of each device or functional module are similar to those of the present application, they fall within the scope of the claims of the present application and their equivalent technologies.

[0028] The technical solution provided by this application brings at least the following beneficial effects:

[0029] Based on any of the above aspects, an embodiment of the present application provides a data labeling method, first, obtaining the data to be labeled. Then, based on the multimodal large model, determining the selected labeling results of the data to be labeled. Next, based on the pre-constructed labeling database, the selected labeling results are corrected to obtain the target labeling results of the data to be labeled. Among them, the multimodal large model is obtained by training multimodal data, and the multimodal data is obtained by integrating the training data of multiple modalities.

[0030] From the above, it can be seen that the present application can determine the selected annotation results of the data to be labeled based on the multimodal large model. Among them, the multimodal large model is obtained by training with multimodal data, and the multimodal data is obtained by integrating the training data of multiple modal data. It can be understood that the multimodal data fully contains the data information of different modal data. Therefore, the selected annotation results determined by the present application based on the multimodal large model obtained by training with multimodal data are more accurate, and the present application can also correct the selected annotation results based on the pre-built annotation database to obtain the target annotation results of the data to be labeled. Therefore, the present application can determine the target annotation results of the data to be labeled more efficiently and accurately.

[0031] The beneficial effects of the first, second, third, fourth and fifth aspects of the present application can all be referred to the analysis of the above beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of a data annotation system provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of a hardware structure of a communication device provided in an embodiment of the present application;

[0034] Figure 3 A flowchart of a data labeling method provided in an embodiment of the present application;

[0035] Figure 4 A flowchart of another data labeling method provided in an embodiment of the present application;

[0036] Figure 5 A flowchart of another data labeling method provided in an embodiment of the present application;

[0037] Figure 6 A flowchart of another data labeling method provided in an embodiment of the present application;

[0038] Figure 7 A flowchart of another data labeling method provided in an embodiment of the present application;

[0039] Figure 8 A flowchart of another data labeling method provided in an embodiment of the present application;

[0040] Fig. 9 A flowchart of another data labeling method provided in an embodiment of the present application;

[0041] Fig.10 A schematic diagram of a process for constructing a multimodal large model provided in an embodiment of the present application;

[0042] Fig.11 A flowchart of another data labeling method provided in an embodiment of the present application;

[0043] Fig.12 A schematic diagram of a model fine-tuning process provided in an embodiment of the present application;

[0044] Fig.13 A schematic diagram of the structure of a data labeling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0047] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and order of execution.

[0048] Before introducing the data annotation method provided by this application in detail, a brief introduction to the application scenarios and implementation environment involved in this application is first given.

[0049] First, a brief introduction to the application scenarios involved in this application is given.

[0050] As described in the background technology, with the continuous development of artificial intelligence technology, multi-modal data (for example, text data, image data, voice data, video data) is increasingly widely used in machine learning of artificial intelligence technology. Data annotation is a key step in machine learning, especially deep learning, and has a crucial impact on model performance. Therefore, the demand for annotation of multi-modal data is growing.

[0051] The traditional method is to manually annotate data, but manual annotation of data is costly and inefficient.

[0052] General technology can use simple fusion strategies to process multi-modal data, and train deep learning models with the processed multi-modal data to perform data annotation based on the deep learning model.

[0053] However, general technology adopts a simple fusion strategy that cannot fully utilize the complementary information between different modal data, resulting in the loss of data information of different modal data. Especially in vertical fields where data is more professional and complex, general technology lacks generalization ability and cannot accurately label data.

[0054] In view of the above problems, an embodiment of the present application provides a data labeling method. First, the data to be labeled is obtained. Then, based on the multimodal large model, the selected labeling results of the data to be labeled are determined. Next, based on the pre-built labeling database, the selected labeling results are corrected to obtain the target labeling results of the data to be labeled. Among them, the multimodal large model is obtained by training multimodal data, and the multimodal data is obtained by integrating the training data of multiple modalities.

[0055] From the above, it can be seen that the present application can determine the selected annotation results of the data to be labeled based on the multimodal large model. Among them, the multimodal large model is obtained by training with multimodal data, and the multimodal data is obtained by integrating the training data of multiple modal data. It can be understood that the multimodal data fully contains the data information of different modal data. Therefore, the selected annotation results determined by the present application based on the multimodal large model obtained by training with multimodal data are more accurate, and the present application can also correct the selected annotation results based on the pre-built annotation database to obtain the target annotation results of the data to be labeled. Therefore, the present application can determine the target annotation results of the data to be labeled more efficiently and accurately.

[0056] The implementation environment of the above data annotation method can be the data annotation system provided in the embodiment of the present application.

[0057] Figure 1 FIG. 1 shows a schematic diagram of the structure of a data annotation system provided by an embodiment of the present application. Figure 1 As shown, the data annotation system includes: a data annotation device 101, a data storage device 102, and a business demand device 103.

[0058] The data annotation device 101 is communicatively connected with the data storage device 102 and the business demand device 103 respectively.

[0059] The data annotation device 101 may include: a high-performance processor, a graphics processor, and a large-capacity memory.

[0060] Among them, high-performance processors and graphics processors are used to meet the real-time operation requirements of multimodal big data, and large-capacity memory is used to store multimodal data and model parameters of multimodal large models.

[0061] Optionally, the data annotation device 101 also includes: a user interaction interface.

[0062] The user interface can provide users with an intuitive data display interface. Users can load the data to be labeled and view the target labeling results through the user interface.

[0063] The user interface can provide users with a variety of annotation tools, such as rectangles, polygons, and text input boxes. Users can also input annotation data through the user interface to build an annotation database.

[0064] The user interaction interface can also provide a user feedback function to collect users' opinions and suggestions on the data annotation results, so as to optimize the model parameters of the multimodal large model and improve the data annotation quality of the data annotation device 101.

[0065] The data storage device 102 may be a pre-built annotation database.

[0066] In practical applications, the data annotation device 101 can be connected to any number of business demand devices 103. For ease of understanding, Figure 1 An example of a data labeling device 101 connected to a business demand device 103 is used for explanation.

[0067] In an embodiment of the present application, the business demand device 103 is used to provide the data labeling device 101 with data for determining the data to be labeled (for example, the data to be labeled, the training set), and the data storage device 102 is used to provide the data labeling device 101 with data for correcting the data to be labeled.

[0068] Optionally, the physical device of the data annotation device 101 may be a server, a terminal, or other types of electronic devices, which is not limited in the embodiments of the present application.

[0069] Optionally, the terminal may be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connection function, or other processing device connected to a wireless modem. The wireless terminal may communicate with one or more core networks via a radio access network (RAN). The wireless terminal may be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, or a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges language and / or data with a radio access network, such as a mobile phone, tablet computer, laptop computer, netbook, personal digital assistant (PDA).

[0070] Optionally, the above-mentioned server can be a server in a server cluster (consisting of multiple servers), or a chip in the server, or a system on a chip in the server, or can be implemented by a virtual machine (VM) deployed on a physical machine, which is not limited in the embodiments of the present application.

[0071] The data annotation equipment in the data annotation system includes: Figure 2 The following are the components included. Figure 2 Taking the communication device shown in the figure as an example, the hardware structure of the data labeling device is introduced.

[0072] Figure 2 , which is a schematic diagram of a hardware structure of a communication device provided in an embodiment of the present application. The communication device includes a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, the memory 22, and the communication interface 23 can be connected via a bus 24.

[0073] The processor 21 is the control center of the communication device, which can be a processor or a general term for multiple processing elements. For example, the processor 21 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0074] As an embodiment, the processor 21 may include one or more CPUs, such as Figure 2 CPU0 and CPU1 are shown in the figure.

[0075] The memory 22 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0076] In one possible implementation, the memory 22 may exist independently of the processor 21, and the memory 22 may be connected to the processor 21 via a bus 24 to store instructions or program codes. When the processor 21 calls and executes the instructions or program codes stored in the memory 22, the data labeling method provided in the following embodiments of the present application can be implemented.

[0077] In the embodiment of the present application, for the communication device, the software programs stored in the memory 22 are different, so the functions implemented by the communication device are different. The functions performed by each device will be described in conjunction with the following flowchart.

[0078] In another possible implementation, the memory 22 may also be integrated with the processor 21 .

[0079] The communication interface 23 is used for the communication device to connect with other devices through a communication network, and the communication network may be Ethernet, wireless access network, wireless local area network (WLAN), etc. The communication interface 23 may include a receiving unit for receiving data and a sending unit for sending data.

[0080] The bus 24 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] It should be pointed out that Figure 2 The structure shown in the figure does not constitute a limitation on the communication device, except Figure 2 In addition to the components shown, the communication device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0082] The data labeling method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0083] The data annotation method provided in the embodiment of the present application is applied to Figure 1 The data annotation device 101 in the data annotation system shown in FIG. Figure 3 As shown, the data labeling method provided in the embodiment of the present application includes:

[0084] S301: Obtain data to be labeled.

[0085] The data to be annotated can be text data, image data, audio data, or video data.

[0086] Combination Figure 1 , the data labeling device 101 can obtain the data to be labeled from the business demand device 103.

[0087] S302: Determine candidate labeling results for the data to be labeled based on the multimodal large model.

[0088] The multimodal large model is obtained by training with multimodal data, and multimodal data is obtained by integrating training data of multiple modalities.

[0089] In an embodiment of the present application, the data annotation device can determine the candidate annotation results of the data to be annotated based on the multimodal large model. Among them, the multimodal large model is obtained by training with multimodal data, and the multimodal data is obtained by integrating the training data of multiple modal data. It can be understood that the multimodal data fully contains the data information of different modal data. Therefore, the candidate annotation results determined by the data annotation device based on the multimodal large model obtained by training with multimodal data are more accurate.

[0090] The data labeling device can obtain the data to be labeled and determine the candidate labeling results of the data to be labeled based on the multimodal large model.

[0091] When the data to be annotated is text data, the annotation results may be key information of the text data, sentiment tendency of the text data, sentiment intensity of the text data, the subject of the text data, entity information in the text data, and the intention expressed by the text data.

[0092] When the data to be annotated is image data, the annotation result can be the subject information in the image data, the selection result of the target object in the image data, the key feature position of the target object in the image data, the image quality of the image data, and the scene information of the image data.

[0093] When the data to be annotated is audio data, the annotation result may be text information of the audio data, the intention expressed by the audio data, and the audio quality of the audio data.

[0094] When the data to be labeled is video data, the labeling result may be the classification of target objects in the video data, the selection result of target objects in the video data, the action behavior of people or objects in the video data, and the scene information of the video data.

[0095] Optionally, the data labeling device may also obtain a labeling task. After obtaining the labeling task, the data labeling device may determine a candidate labeling result for the data to be labeled based on the multimodal large model.

[0096] S303: Based on the pre-built annotation database, the annotation result to be selected is modified to obtain a target annotation result of the data to be annotated.

[0097] In the embodiment of the present application, the data labeling device can modify the selected labeling result based on the pre-built labeling database to obtain the target labeling result of the data to be labeled. The labeling database stores the data labeling results obtained based on professional judgment. Therefore, the data labeling device can more accurately determine the target labeling result of the data to be labeled by modifying the selected labeling result based on the pre-built labeling database.

[0098] The annotation database is used to store original data and annotated data, including original text data, annotated text data, original image data, annotated image data, original audio data, annotated audio data, original video data, and annotated video data.

[0099] In some embodiments, in combination Figure 3 ,like Figure 4 As shown, the data annotation method also includes:

[0100] S401. Obtain multiple training sets.

[0101] Each training set includes training data of multiple modalities with associated relationships.

[0102] The multi-modal training data includes at least two of the following: text data, image data, audio data, and video data.

[0103] The data annotation device can directly obtain the training set.

[0104] Optionally, the data annotation device can obtain training data of multiple modalities and process the training data of multiple modalities, including: preprocessing the training data of multiple modalities and extracting features of the training data of multiple modalities.

[0105] Specifically, first, the data annotation device can obtain training data of multiple modalities in a vertical field, including: image data (for example, equipment images), text data (equipment operation manuals, equipment failure reports), and audio data (audio recordings). Then, the data annotation device can clean the acquired training data of multiple modalities, that is, delete duplicate, invalid, or poor quality training data. Next, the data annotation device can extract corresponding features for training data of different modalities. For example, device edges, device textures, and device colors in image data, and keywords and key phrases in text data.

[0106] Combination Figure 1 , the data annotation device 101 can obtain multiple training sets from the business demand device 103.

[0107] In one conceivable manner, a data annotation device may obtain training data of multiple modalities from various sources in vertical fields, such as equipment manufacturers, operators, and research institutions.

[0108] S402: Integrate the training data of multiple modalities in the training set into multimodal data.

[0109] In an embodiment of the present application, the data annotation device can integrate the training data of multiple modes in the training set into multimodal data. It can be understood that the multimodal data includes data information of the training data of multiple modes, so that the data annotation device can train the multimodal large model based on the multimodal data, so that the multimodal large model can accurately determine the candidate annotation results of the data to be annotated based on the information of the multimodal data.

[0110] The data annotation device can use Orthogonal Sequential Fusion (OSF) to integrate the training data of multiple modalities in the training set into multimodal data.

[0111] OSF is a multimodal data fusion strategy. In the process of fusing training data of multiple modalities, the training data of different modalities are fused step by step in sequence, and the data features of the training data of each modality are guaranteed to be orthogonal to each other, so as to give full play to the advantages of the training data of each modality and avoid information redundancy and mutual interference.

[0112] Specifically, the data annotation device can sequentially merge training data of different modalities.

[0113] Exemplarily, the data annotation device can integrate the training data of three different modalities into multimodal data, namely, the training data i1 of the first modality, the training data i2 of the second modality, and the training data i3 of the third modality. The data annotation device can first integrate the training data i1 of the first modality and the training data i2 of the second modality into the multimodal data z of the first modality and the second modality. 12 The data annotation device can then transform the multimodal data of the first modality and the second modality into 12 The training data i3 of the third modality is integrated into the multimodal data z of the first modality, the second modality, and the third modality 123 .

[0114] Among them, the multimodal data z of the first mode and the second mode 12 Satisfies the following formula:

[0115] z 12 =f(i1,i2);

[0116] Among them, f is used to represent the fusion function.

[0117] Multimodal data z of the first modality, the second modality, and the third modality 123 Satisfies the following formula:

[0118] z 123 =g(z 12 , i3);

[0119] Among them, g is used to represent the fusion function.

[0120] S403: Train a large multimodal model based on the multimodal data.

[0121] In an embodiment of the present application, a data annotation device can train a multimodal large model based on multimodal data. The multimodal data fully contains data information of data of different modalities. In this way, the present application trains a multimodal large model based on the above multimodal data, so that the multimodal large model can fully learn multiple features of a data, that is, features under different modalities. This makes the candidate annotation results determined by the data annotation device based on the multimodal large model more accurate.

[0122] Optionally, before training the multimodal large model based on the multimodal data, the data annotation device may construct the multimodal large model.

[0123] Specifically, the data annotation device can integrate the Segment Anything Model (SAM), the Grounding with DINO (Grounding DINO) model, and the Stable Diffusion model into a deep learning framework (e.g., TensorFlow, Pythontorch (PyTorch)) to obtain a multimodal large model. The multimodal large model has the image segmentation capability of SAM, the zero-shot visual understanding capability of Grounding DINO, and the text-to-image generation capability of Stable Diffusion.

[0124] In the above deep learning framework, a unified input and output interface is designed for each model to ensure that the above models can work together.

[0125] The data annotation device can also design algorithms to enable the above-mentioned SAM model, Grounding DINO model, and Stable Diffusion model to work together.

[0126] For example, first, the data annotation device can generate relevant image data based on the text-to-image generation capability of the Stable Diffusion model. Then, the data annotation device can segment the image data based on the image segmentation capability of the SAM model. Next, the data annotation device can use the zero-shot learning capability of the Grounding DINO model to associate the text data with the key parts in the image data.

[0127] As can be seen from the above, the multimodal large model builds an image segmentation module based on SAM to process image data. It also integrates the zero-shot learning capability of Grounding DINO, so that the multimodal large model can handle new categories or tasks that are not in the training set. The multimodal large model combines the Stable Diffusion technology to generate related images based on text data to enrich the data set.

[0128] In some embodiments, in combination Figure 4 ,like Figure 5 As shown, in S402, the training data of multiple modes in the training set are integrated into multimodal data, including:

[0129] S501 . Determine the weight of the training data of each modality according to the data quality of the training data of each modality among the training data of multiple modalities.

[0130] Data quality is positively correlated with weight.

[0131] Data quality is used to indicate the proportion of data noise in training data, and data noise is used to indicate interference data in training data.

[0132] In an embodiment of the present application, the data annotation device can integrate the training data of multiple modes in the training set into multimodal data. Since the feature information contained in the training data of different modes is different, the information of the same person or object can be reflected in different aspects and to different degrees. However, the data quality of the training data of different modes is different. The higher the data quality, the better it can reflect the information of the person or object. Therefore, the data annotation device can determine the weight of the training data of each mode according to the data quality of the training data of each mode in the training data of multiple modes.

[0133] In one possible implementation, training data of different modalities have different uncertainty measures. The uncertainty measure can reflect the data quality of the data. The higher the uncertainty measure value of the training data, the lower the data quality, and the lower its weight.

[0134] The uncertainty measure of the training data of a modality may be the prediction confidence of the training data, the Dempster-Shafer (DS) evidence theory of the training data, or the energy score of the training data.

[0135] The data annotation device can determine the weight of the training data of each modality based on the uncertainty measurement of the training data of each modality. The relationship between the weight and the uncertainty measurement satisfies the following formula:

[0136]

[0137] Among them, ω m The weight used to represent the training data of the mth modality (also called fusion weight), μ m (x) is used to represent the uncertainty measure of the training data of the m-th mode, and α is used to represent the specific hyperparameter of the training data of a mode.

[0138] S502: Integrate the training data of multiple modalities into multimodal data according to the weight of the training data of each modality.

[0139] The data annotation device can integrate the training data of multiple modes into multimodal data based on quality-aware multimodal fusion (QMF). In the process of integration, the weights of the training data of different modes can be dynamically adjusted. In this way, the data annotation device can improve the quality and accuracy of multimodal data. In subsequent applications, the multimodal large model can be enhanced to extract unique information from training data of different modes, so that the data annotation device can achieve efficient and accurate data annotation based on the multimodal large model.

[0140] In one achievable manner, the data annotation device may integrate the training data of multiple modalities into multimodal data according to the weight of the training data of each modality, that is, integrate the training data of multiple modalities according to the weight at the feature level.

[0141] In another possible implementation, the data labeling device can predict the training data of each modality based on the multimodal large model, and determine the final prediction result according to the prediction result of the training data of each modality and the weight of the training data of each modality, that is, determine the final prediction value of the multimodal data (which can also be understood as the candidate labeling result) in the decision and according to the weight.

[0142] The above fusion processes all satisfy the following formula:

[0143]

[0144] Among them, M is used to represent the number of modal categories, ω m The weight used to represent the training data of the mth mode, f m (x m ) is used to represent the m-th mode (or the prediction result of the training data of the m-th mode).

[0145] In one achievable manner, the data annotation device may sort the training data of multiple modalities to determine the modalities ordering of the training data of multiple modalities. Further, the data annotation device may integrate the training data of multiple modalities into multimodal data based on the modalities ordering result.

[0146] Optionally, the data annotation device can train a single modality model and determine the modality sorting result based on the individual performance of the training data of multiple modalities in the task. The modality sorting result can be expressed as S=sort(i1, i2, ..., i N ), where i1 is the training data of the first modality, i2 is the training data of the second modality, and i N It is the training data of the Nth modality, and the individual performance of the data is getting better and better from the training data of the first modality to the training data of the Nth modality. It can also be understood that the data quality of the data is getting higher and higher from the training data of the first modality to the training data of the Nth modality.

[0147] In one practicable manner, the data annotation device, in the process of integrating the training data of multiple modalities, starts from the worst performing training data and gradually transitions to the best performing training data. In this way, the data annotation device can improve the extraction of feature information of the training data of each modality and improve the quality of multimodal data.

[0148] In some embodiments, in combination Figure 5 ,like Figure 6 As shown, the data annotation method also includes:

[0149] S601: Obtain data noise of training data.

[0150] Data noise is used to represent interference data in training data.

[0151] Combination Figure 1 , the data annotation device 101 can obtain the data noise of the training data from the business demand device.

[0152] S602: Determine the data quality of the training data according to the data noise.

[0153] Data noise is negatively correlated with data quality.

[0154] In one achievable manner, data noise can be used to reflect the data quality of the training data. The greater the data noise, the lower the data quality of the training data. Correspondingly, the smaller the data noise, the smaller the data noise of the training data.

[0155] Data noise in text data includes: spelling errors, semantic ambiguity, grammatical errors, and irrelevant information in the text.

[0156] The data noise of image data includes: image blur, uneven image lighting, image distortion, and image missing.

[0157] The data noise of audio data includes: environmental noise, electromagnetic interference noise, and audio loss.

[0158] The data noise of video data includes: video frame noise, video missing, and video blur.

[0159] In some embodiments, in combination Figure 6 ,like Figure 7 As shown, in S502, according to the weight of the training data of each modality, the training data of multiple modalities are integrated into multimodal data, including:

[0160] S701. Extract feature data of training data of each modality.

[0161] Each modality of training data contains its own unique feature data, and different modal training data contain different types and forms of information. To a certain extent, different modal training data are complementary. Integrating the feature data of multiple modal training data can obtain more comprehensive and accurate data. For example, image data provides visual information, and text data provides descriptive or semantic information. Image data and text data can complement each other, allowing multimodal large models to better understand data.

[0162] S702: Determine the sum of the products of the feature data and the weight as feature data of the multimodal data.

[0163] The data labeling device can determine the sum of the products of the feature data and the weight as the feature data of the multimodal data. The specific process can refer to the above-mentioned process of integrating the training data of multiple modalities into multimodal data according to the weights of the training data of each modality.

[0164] S703: Determine multimodal data according to feature data of the multimodal data.

[0165] In the embodiment of the present application, since the training data of different modalities have different physical properties and information characteristics, multimodal data cannot be obtained by simply superimposing the training data of different modalities. In order to integrate the training data of different modalities into multimodal data, the data annotation device can extract the feature data of the training data of each modality, integrate the feature data, and then determine the multimodal data based on the integrated feature data.

[0166] In some embodiments, in combination Figure 7 ,like Figure 8 As shown, in S403, training a multimodal large model based on the multimodal data includes:

[0167] S801: Input the multimodal data into the multimodal large model to obtain the predicted labeling result of the multimodal data and obtain the actual labeling result of the multimodal data.

[0168] S802. Determine the loss value of the multimodal large model based on the predicted labeling results and the actual labeling results, and iteratively train the multimodal large model based on the loss value and the multimodal data until the model convergence conditions are met to obtain the multimodal large model.

[0169] The convergence conditions include: the number of iterations is greater than the iteration number threshold, or the loss value is less than the loss value threshold.

[0170] In one achievable method, the data labeling device can determine the loss value of the multimodal large model based on the predicted labeling results and the actual labeling results, and iteratively train the multimodal large model based on the loss value and the multimodal data until the model convergence conditions are met to obtain the multimodal large model.

[0171] In another achievable method, the data labeling device may introduce an orthogonal loss function during the iterative training of the multimodal large model, thereby optimizing the multimodal data and further optimizing the multimodal large model by continuously optimizing the fusion process of training data of different modalities.

[0172] The orthogonality loss function is used to represent the orthogonality loss of two feature vectors (vector representation of feature data).

[0173] In some embodiments, the data annotation device may integrate training data of N different modalities into multimodal data, which involves N-1 fusion layers.

[0174] The orthogonality loss function satisfies the following formula:

[0175]

[0176] Among them, T is used to represent the T task, L is used to represent the loss function of task T, preds is used to represent the predicted labeling result, outputs is used to represent the actual labeling result, L1 is used to represent the regularization term of the first fusion layer, and L2 is used to represent the regularization term of the first fusion layer.

[0177] The regularization term L1 of the first fusion layer satisfies the following formula:

[0178] L1=λ 1,i |embd 1,i *embd 2,i |;

[0179] Among them, λ 1,i Hyperparameter used to represent the regularization strength of the first fusion layer, embd 1,i Embedded representation of training data of one modality used to represent the i-th fusion layer, embd 2,i The embedding representation of the training data of the two modalities used to represent the i-th fusion layer.

[0180] The regularization term L2 of the second fusion layer satisfies the following formula:

[0181] L2=λ 2,i (embd 1,i *embd 2,i ) 2 ;

[0182] Among them, λ 2,i Hyperparameter used to represent the regularization strength of the second fusion layer, embd 1,i Embedded representation of training data of one modality used to represent the i-th fusion layer, embd 2,i The embedding representation of the training data of the two modalities used to represent the i-th fusion layer.

[0183] With the first fusion layer regularization term forcing the absolute value of the dot product of the embeddings to be close to zero, and the second fusion layer regularization term encouraging the square of the dot product to be close to zero, the training data of different modalities are complementary.

[0184] The data annotation device enhances the adaptability of the multimodal large model to low-quality training data by using regularization terms to reduce the excessive reliance of the multimodal large model on unreliable training data.

[0185] In some embodiments, in combination Figure 8 ,like Fig. 9 As shown, the data annotation method also includes:

[0186] S901. Adjust model parameters of the multimodal large model according to the target labeling result.

[0187] In one feasible method, as the annotation database continues to accumulate, the data annotation device corrects the annotation results to be selected based on the annotation database to obtain the target annotation results of the data to be annotated, and continuously adjusts the model parameters of the multimodal large model based on the target annotation results.

[0188] From the above, we can see that as the annotation database continues to accumulate, the data annotation equipment can iteratively train the multimodal large model and use the newly added target annotation data to fine-tune the model parameters of the model to improve the ability of the multimodal large model to process the data to be annotated.

[0189] In another possible implementation, the data labeling device can also provide intelligent labeling prompts and suggestions based on the user's labeling habits and labeling preferences to reduce the user's learning cost and labeling difficulty.

[0190] Optionally, the data annotation device can evaluate the multimodal large model using the new training set to determine the accuracy and recall of the multimodal large model.

[0191] The accuracy rate is used to indicate the accuracy rate of the candidate labeling results determined by the data labeling device, and the recall rate is used to indicate the proportion of the candidate labeling results determined by the data labeling device that are recalled and corrected.

[0192] For example, Fig.10 A schematic diagram of a process for building a multimodal large model is shown.

[0193] First, the data annotation equipment obtains the data and processes it.

[0194] Specifically, the data annotation equipment collects data in multiple modalities, including: text data, image data, audio data, and video data.

[0195] Then, the data annotation equipment processes the multi-modal data.

[0196] Specifically, the processing operations of the data annotation device on text data include: deduplication and word segmentation, that is, deleting duplicate text data and performing word segmentation on the text data. The processing operations of the data annotation device on image data include: denoising and enhancement, that is, deleting noise data in the image data and enhancing the main content of the image data. The processing operations of the data annotation device on video data include: extracting key frames, that is, extracting key images in video data. The processing operations of the data annotation device on audio data include: enhancing speech, that is, enhancing speech signals in audio data.

[0197] Next, the data annotation equipment can build a large multimodal model.

[0198] The multimodal large model is integrated by three basic models: image segmentation model, positioning model based on target detection model, and stable diffusion model.

[0199] Next, the data annotation equipment trains the model based on multi-modal data.

[0200] Specifically, the data labeling device first initializes the model and then performs full batch training based on multiple modal data.

[0201] Next, the data labeling device can obtain actual data labeling (which can also be understood as manual data labeling) to perform model evaluation and fine-tune model parameters (also known as model fine-tuning) on ​​the multimodal large model based on the actual labeling results.

[0202] Among them, the data labeling device can adjust the model parameters based on low-rank adaptive technology, and can also adjust the model parameters based on quantized low-rank adaptive technology.

[0203] For example, Fig.11 A schematic diagram of another data annotation process is shown.

[0204] First, the data labeling device can obtain the data to be labeled (also referred to as loading the data to be labeled).

[0205] Then, the data labeling device can determine the candidate labeling results (also called preliminary labeling results) of the data to be labeled based on the multimodal large model.

[0206] Specifically, the data annotation device can perform image segmentation on image data. The data annotation device can perform semantic understanding on text data and audio data and annotate the data.

[0207] Next, the data labeling device may correct the candidate labeling result (ie, correct the candidate labeling result) based on the pre-built labeling database to obtain the target labeling result.

[0208] Among them, the annotation data in the annotation database is obtained by users through correction and optimization based on the candidate annotation results and professional knowledge.

[0209] Then, the data labeling device can save and export the target labeling results.

[0210] Optionally, the target annotation results can be used for model evaluation and model fine-tuning.

[0211] Data annotation equipment can fine-tune large multimodal models based on model evaluation results to optimize model parameters and model structure.

[0212] For example, Fig.12 A schematic diagram of a model fine-tuning process is shown.

[0213] First, the feature vectors of training data of different modalities are input.

[0214] Then, the input feature vector is multiplied by the pre-trained weight W to get an intermediate result.

[0215] Next, the intermediate result is added to the product of matrix A and matrix B to obtain the prediction result after weight update, and the prediction result is output.

[0216] The internal dimension r of matrix A and matrix B is a hyperparameter. By adjusting r, the data annotation device can control the number of parameters updated during the fine-tuning process of the multimodal large model. Matrix A and matrix B are the feature vectors of the feature data of the training data of different modalities.

[0217] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0218] The embodiment of the present application can divide the functional modules of the data annotation device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0219] Fig.13 FIG. 1 shows a schematic diagram of the structure of a data labeling device provided in an embodiment of the present application. Fig.13 As shown, the data labeling device includes: a communication unit 1301 and a processing unit 1302;

[0220] The communication unit 1301 is used to obtain the data to be labeled; the processing unit 1302 is used to determine the candidate labeling results of the data to be labeled based on the multimodal large model; the multimodal large model is obtained by training with multimodal data; the multimodal data is obtained by integrating training data of multiple modalities; the processing unit 1302 is also used to correct the candidate labeling results based on a pre-constructed labeling database to obtain the target labeling results of the data to be labeled.

[0221] Optionally, the communication unit 1301 is also used to obtain multiple training sets; each training set includes training data of multiple modalities with an associated relationship; the training data of multiple modalities includes at least two of the following: text data, image data, audio data, and video data; the processing unit 1302 is also used to integrate the training data of multiple modalities in the training set into multimodal data; the processing unit 1302 is also used to train a multimodal large model based on the multimodal data.

[0222] Optionally, the processing unit 1302 is specifically used to: determine the weight of the training data of each modality according to the data quality of the training data of each modality in the training data of multiple modalities; the data quality is positively correlated with the weight; the data quality is used to represent the proportion of data noise in the training data; the data noise is used to represent the interference data in the training data; and according to the weight of the training data of each modality, integrate the training data of multiple modalities into multimodal data.

[0223] Optionally, the communication unit 1301 is further used to obtain data noise of the training data; the processing unit 1302 is further used to determine the data quality of the training data according to the data noise; the data noise is negatively correlated with the data quality.

[0224] Optionally, the processing unit 1302 is specifically used to: extract feature data of the training data of each modality; determine the sum of the products of the feature data and the weight as feature data of the multimodal data; and determine the multimodal data based on the feature data of the multimodal data.

[0225] Optionally, the processing unit 1302 is specifically used to: input the multimodal data into the multimodal large model to obtain the predicted labeling results of the multimodal data, and obtain the actual labeling results of the multimodal data; determine the loss value of the multimodal large model according to the predicted labeling results and the actual labeling results, and iteratively train the multimodal large model based on the loss value and the multimodal data until the model convergence conditions are met to obtain the multimodal large model.

[0226] Optionally, the processing unit 1302 is further used to adjust the model parameters of the multimodal large model according to the target labeling result.

[0227] An embodiment of the present application also provides a computer-readable storage medium, which includes computer execution instructions. When the computer execution instructions are executed on a computer, the computer executes the data labeling method provided in the above embodiment.

[0228] An embodiment of the present application also provides a computer program, which can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program can implement the data labeling method provided in the above embodiment.

[0229] Those skilled in the art should be aware that in one or more of the above examples, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0230] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0231] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0232] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the general technology or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a disk, or an optical disk.

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data labeling method, characterized in that: The method comprises: Obtain the data to be labeled; Determining the selected annotation results of the data to be annotated based on the multimodal large model; the multimodal large model is obtained by training the multimodal data; the multimodal data is obtained by integrating the training data of multiple modalities; Based on a pre-constructed annotation database, the candidate annotation result is modified to obtain a target annotation result of the data to be annotated.

2. The method according to claim 1, characterized in that The method further comprises: Acquire multiple training sets; each training set includes training data of multiple modes with associated relationships; the training data of the multiple modes includes at least two of the following: text data, image data, audio data, and video data; Integrating the training data of multiple modalities in the training set into multimodal data; The multimodal large model is trained based on the multimodal data.

3. The method according to claim 2, characterized in that The integrating the training data of multiple modes in the training set into multimodal data comprises: Determine the weight of the training data of each modality according to the data quality of the training data of each modality in the training data of the multiple modalities; the data quality is positively correlated with the weight; the data quality is used to represent the proportion of data noise in the training data; the data noise is used to represent interference data in the training data; The training data of the multiple modalities are integrated into multimodal data according to the weight of the training data of each modality.

4. The method according to claim 3, characterized in that: The method further comprises: Obtaining data noise of the training data; The data quality of the training data is determined according to the data noise; the data noise is negatively correlated with the data quality.

5. The method according to claim 3, characterized in that: The step of integrating the training data of the multiple modalities into multimodal data according to the weight of the training data of each modality includes: Extracting feature data of the training data of each modality; Determining the sum of the products of the feature data and the weight as feature data of the multimodal data; The multimodal data is determined according to feature data of the multimodal data.

6. The method according to claim 2, characterized in that The step of training the multimodal large model according to the multimodal data comprises: Inputting the multimodal data into a multimodal large model to obtain a predicted annotation result of the multimodal data, and obtaining an actual annotation result of the multimodal data; According to the predicted labeling result and the actual labeling result, the loss value of the multimodal large model is determined, and the multimodal large model is iteratively trained based on the loss value and the multimodal data until the model convergence condition is met to obtain the multimodal large model.

7. The method according to claim 1, characterized in that The method further comprises: According to the target labeling result, the model parameters of the multimodal large model are adjusted.

8. A data labeling device, characterized in that: The device comprises: a communication unit and a processing unit; The communication unit is used to obtain the data to be labeled; The processing unit is used to determine the candidate labeling results of the data to be labeled based on the multimodal large model; the multimodal large model is obtained by training multimodal data; the multimodal data is obtained by integrating training data of multiple modalities; The processing unit is further used to modify the candidate annotation result based on a pre-built annotation database to obtain a target annotation result of the data to be annotated.

9. A data annotation device, characterized in that: include: A processor and a memory; wherein the memory is used to store one or more programs, and the one or more programs include computer-executable instructions. When the device is running, the processor executes the computer-executable instructions stored in the memory to enable the device to perform the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of a data labeling device, the data labeling device can execute the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises: a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Images