Automatic data labeling method based on multi-modal model

Abstract

The invention relates to an automatic data annotation method based on a multi-modal model, and belongs to the field of data annotation and artificial intelligence. According to the invention, to-be-labeled image data and potential labels are input into an automatic data labeling system; the automatic data labeling system scores and sorts the potential labels through a text-image multi-mode model, and returns a potential label probability sorting result; and the labeler manually audits the automatic labeling label, confirms a labeling result and completes labeling. According to the method, the data labeling time efficiency can be improved; the labor cost is reduced; and the annotation data volume is expanded, and more downstream artificial intelligence models are efficiently driven.

Description

Technical Field

[0001] The present invention belongs to the field of data annotation and artificial intelligence, and specifically relates to an automatic data annotation method based on a multimodal model. Background Art

[0002] Artificial intelligence models require large-scale labeled data for training, and manual data labeling usually requires a lot of manpower and material resources. The introduction of automatic data labeling products is precisely to help data labelers increase labeling efficiency and shorten labeling time. Current mainstream artificial intelligence methods are based on the closed set assumption, following the assumption that training data and test data meet the "independent and identically distributed" assumption. However, real scenes are open and complex, containing many unknown objects and distributions, such as the appearance of wild animals that have not been included in surveillance images.

[0003] The strong robustness, transferability, and deductive learning capabilities demonstrated by large artificial intelligence models can help detect unknown classes and discover new categories in real open scenarios, achieving the goal of autonomously distinguishing and learning unknown classes. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide a method for automatic data annotation based on a multimodal model to solve the problem of automatic image annotation, especially the annotation of unknown objects.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a method for automatic data annotation based on a multimodal model, which comprises the following steps:

[0008] Step S101: Automatic labeling system for image data to be labeled and potential label input data;

[0009] Step S102: The data automatic annotation system scores and ranks potential labels through the text-image multimodal model, and returns the potential label probability ranking result; the text-image multimodal model processes text and image data simultaneously, uses a pre-trained language model and a convolutional neural network to extract features, and uses an attention mechanism to fuse features; the model infers the association between text and image, and scores the relevance of the input image and text, and finally obtains the most likely image description through scoring and ranking; the model outputs the label with the highest relevance as the automatic annotation label of the image to be annotated;

[0010] Step S103: The annotator manually reviews the automatically annotated labels, confirms the annotated results and completes the annotating.

[0011] (III) Beneficial effects

[0012] The present invention proposes an automatic data annotation method based on a multimodal model. The present invention adopts a large text-image multimodal model supporting both Chinese and English to provide recommended tags for annotators, simplify the user annotation process, and ultimately achieve the effect of improving annotation efficiency.

[0013] The present invention relates to a method, system and storage medium for automatic data annotation based on a multimodal model. The automatic annotation system adopts a B / S architecture to provide a callable interface for users. The user sends pictures to be annotated and potential tags to the automatic annotation system by initiating an HTTP request. The automatic annotation system returns a possible probability ranking of tags to the user, so that the user can improve the annotation efficiency.

[0014] An automatic data labeling method based on a multimodal model can be used in the field of data labeling and can bring the following expected effects: 1) improve the time efficiency of data labeling; 2) reduce labor cost overhead; 3) expand the amount of labeled data and efficiently drive more downstream artificial intelligence models. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the product process of automatic data annotation based on the artificial intelligence big model of the present invention;

[0016] Figure 2 This is the architecture diagram of the data automatic annotation system based on the artificial intelligence big model of the present invention;

[0017] Figure 3 An electronic device;

[0018] Figure 4 A computer-readable storage medium. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The automatic data labeling method based on the multimodal model relies on the multimodal model's ability to parse unknown class labels to effectively associate the Chinese and English labels input by the user with the images to be labeled, and give the target label with the highest probability for the images to be labeled.

[0021] The present invention proposes an automatic data annotation method based on a multimodal model, aiming to help annotators improve the annotation efficiency of image classification tasks.

[0022] The present invention uses a multimodal model to provide annotators with the potential label prediction probability of the data to be annotated. Annotators can quickly screen out accurate labels by sorting the probability, effectively reducing the cost of labeling work and ultimately improving data labeling efficiency.

[0023] The present invention involves an automatic data annotation system based on a multimodal model, which adopts B / S architecture mode, asynchronous resource scheduling, Docker container deployment and other technologies, and is committed to providing users with efficient, convenient and fast-response automatic data annotation services to ensure the robustness of the annotation results.

[0024] Figure 1 The present invention provides a method for automatic data annotation based on a multimodal model, the method comprising the following steps:

[0025] Step S101: the image data to be annotated (original image data without labels) and potential labels (possible label descriptions of the original image) are input into the data automatic annotation system.

[0026] Step S102: The data automatic annotation system scores and ranks potential labels through the text-image multimodal model, and returns the potential label probability ranking result; the text-image multimodal model processes text and image data simultaneously, uses a pre-trained language model and convolutional neural network to extract features, and uses the attention mechanism to fuse features. Specifically, the model can infer the association between text and image, and the multimodal model involved in the present invention can score the relevance of the input image and text, and finally obtain the most likely image description through scoring and sorting; the model outputs the label with the highest relevance as the automatic annotation label of the image to be annotated.

[0027] Step S103: The annotator manually reviews the automatically annotated labels, confirms the annotated results and completes the annotating.

[0028] The present invention covers a text-image multimodal model. In the input processing part, the model uses the pre-trained language large model BERT to encode the text input information and convert it into a vector representation; the pre-trained visual large model ViT is used to extract the features of the image input information to obtain the corresponding feature vector. In the feature fusion stage, the attention mechanism is used to weightedly combine the text and image features, and through a multimodal learning strategy, a text-image annotation dataset is used for cross-modal comparative learning to obtain a joint representation between the modalities. Finally, in the model reasoning stage, the model can obtain the corresponding correlation degree score for the paired input image and text label.

[0029] When processing text data, the text-image multimodal model of the present invention processes Chinese and English texts simultaneously.

[0030] Embodiment 1:

[0031] A method for automatic data annotation based on a multimodal model, which specifically provides users with automatic data annotation services.

[0032] Furthermore, the automatic labeling capability of image classification tasks is provided for the original image data to be labeled.

[0033] Furthermore, the automatic annotation results provided by the data automatic annotation product are adopted as the final annotation results after manual review.

[0034] Furthermore, the automatic data annotation capability is provided by a multimodal model, which can score labels in multiple languages ​​and obtain potential labels for images based on the scoring order.

[0035] Furthermore, the data automatic annotation method adopts a B / S architecture model to provide users with a service calling interface.

[0036] Furthermore, the data automatic annotation method uses resource asynchronous scheduling on the server side to call the automatic annotation service.

[0037] Furthermore, the data automatic labeling product uses Docker containerization to deploy the image data automatic labeling service based on the multimodal model.

[0038] Embodiment 2:

[0039] The present invention also provides an electronic device corresponding to the method provided in the above-mentioned embodiment. Figure 3 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0040] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0041] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any of the above embodiments of the present invention may be applied to the processor 200, or implemented by the processor 200.

[0042] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 200 or the instruction in the form of software. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0043] The electronic device provided by the present invention and the method provided by the present invention are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0044] Embodiment 3:

[0045] The present invention also provides a computer-readable storage medium corresponding to the method provided in the above-mentioned embodiment, please refer to Figure 4 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0046] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0047] The present invention uses an image classification automatic annotation technology based on a multimodal model to provide automatic annotation capabilities for image classification tasks;

[0048] The present invention adopts a text-image multimodal model to provide the ability to process Chinese and English texts;

[0049] The present invention adopts a multimodal model-based data automatic annotation method and has the ability to annotate unknown domain data;

[0050] The data automatic annotation system involved in the present invention adopts a B / S architecture mode, in which the automatic annotation service uses a multimodal model as a service backend to perform probability scoring and sorting for input tags, and the browser side serves as a client to receive the automatic annotation service results;

[0051] The data automatic annotation system involved in the present invention uses an asynchronous call method on the server side to schedule the automatic annotation service based on the multimodal model;

[0052] The data automatic annotation system involved in the present invention adopts Docker containerization to deploy automatic annotation services, and realizes cross-platform rapid deployment and operation by encapsulating the automatic annotation system based on the text-image multimodal model in the Docker image.

[0053] The present invention adopts a text-image multimodal large model that supports both Chinese and English to provide recommended tags for annotators, simplify the user annotation process, and ultimately achieve the effect of improving annotation efficiency.

[0054] The present invention relates to a method, system and storage medium for automatic data annotation based on a multimodal model. The automatic annotation system adopts a B / S architecture to provide a callable interface for users. The user sends pictures to be annotated and potential tags to the automatic annotation system by initiating an HTTP request. The automatic annotation system returns a possible probability ranking of tags to the user, so that the user can improve the annotation efficiency.

[0055] An automatic data labeling method based on a multimodal model can be used in the field of data labeling and can bring the following expected effects: 1) improve the time efficiency of data labeling; 2) reduce labor cost overhead; 3) expand the amount of labeled data and efficiently drive more downstream artificial intelligence models.

[0056] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for automatic data annotation based on a multimodal model, characterized in that: The method comprises the following steps: Step S101: Automatic labeling system for image data to be labeled and potential label input data; Step S102: The data automatic annotation system scores and ranks potential labels through the text-image multimodal model, and returns the potential label probability ranking result; the text-image multimodal model processes text and image data simultaneously, uses a pre-trained language model and convolutional neural network to extract features, and uses the attention mechanism to fuse features; the model infers the association between text and image, and scores the association between the input image and text, and finally obtains the most likely image description through scoring and ranking; The model outputs the label with the highest correlation as the automatic label of the image to be labeled; Step S103: The annotator manually reviews the automatically annotated labels, confirms the annotated results and completes the annotating.

2. The method for automatic data annotation based on a multimodal model according to claim 1, characterized in that: The image data to be annotated is the original image data without labels, and the potential labels are possible label descriptions of the original images.

3. The method for automatic data annotation based on a multimodal model according to claim 1, characterized in that: In the input processing part, the model uses the pre-trained language large model BERT to encode the text input information and convert it into a vector representation; it uses the pre-trained visual large model ViT to extract the features of the image input information and obtain the corresponding feature vector.

4. The method for automatic data annotation based on a multimodal model as claimed in claim 3, characterized in that: In the feature fusion stage, the attention mechanism is used to weightedly combine text and image features. Through a multimodal learning strategy, a text-image annotation dataset is used for cross-modal comparative learning to obtain a joint representation between the modalities.

5. The method for automatic data annotation based on a multimodal model as claimed in claim 4, characterized in that: In the model inference stage, the model obtains corresponding correlation scores for paired input images and text labels.

6. The method for automatic data annotation based on a multimodal model according to claim 1, characterized in that: This text-image multimodal model processes both Chinese and English texts when processing text data.

7. The method for automatic data annotation based on a multimodal model according to claim 1, characterized in that: The automatic annotation system adopts a B / S architecture model, in which the automatic annotation service uses a multimodal model as the service backend to perform probability scoring and sorting for input tags, and the browser side acts as the client to receive the automatic annotation service results.

8. The method for automatic data annotation based on a multimodal model according to claim 1, characterized in that: The automatic annotation system adopts B / S architecture to provide users with a callable interface. Users send the images to be annotated and potential tags to the automatic annotation system by initiating HTTP requests, and the automatic annotation system returns the possible probability ranking of the tags to the user.

9. The method for automatic data annotation based on a multimodal model according to claim 7, characterized in that: The automatic annotation system uses asynchronous calling on the server side to schedule the automatic annotation service based on the multimodal model.

10. The method for automatic data annotation based on a multimodal model according to claim 7, characterized in that: The automatic annotation system uses Docker containerization to deploy automatic annotation services. By encapsulating the automatic annotation system based on the text-image multimodal model in a Docker image, it can be quickly deployed and run across platforms.

Images