Multi-modal automatic labeling method and device, storage medium and electronic equipment

Abstract

The invention provides a multi-modal automatic labeling method and device, a storage medium and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the feature enhancement of a to-be-processed text feature and a to-be-processed image feature through a cross-modal decoder, obtaining an enhanced text feature and an enhanced image feature, and carrying out the feature enhancement of the to-be-processed text feature and the to-be-processed image feature; performing feature fusion by using a cross attention mechanism to obtain correlation scores between the enhanced text features and enhanced image features corresponding to different image regions; selecting an enhanced image feature having the highest correlation with the enhanced text feature, and generating query information corresponding to each selected image feature; and obtaining annotation information corresponding to each piece of query information through a multi-modal target detection unit. According to the multi-modal automatic labeling method and device, the storage medium and the electronic equipment provided by the invention, the information is automatically extracted from the text description and is converted into the image labeling, so that the efficiency and accuracy of image labeling are greatly improved, and the labor cost is also reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal automatic annotation method, device, storage medium and electronic device. Background Art

[0002] Artificial Intelligence (AI) is a rapidly growing field that uses computer science and data to mimic human intelligence. The applications of AI are very wide, from virtual assistants and recommendation systems in everyday life to professional fields such as healthcare, finance and manufacturing.

[0003] In the field of automated object detection and recognition, image annotation technology has become an indispensable key link. The image annotation methods in related technologies mainly rely on manual operations, which are to manually select specific areas in the image to annotate the name and location of the object.

[0004] However, the labeling methods in related technologies can only automatically label fixed object categories. New categories need to be manually labeled and retrained. In addition, they lack the ability to process multimodal data, that is, they cannot process image and text information at the same time. Not only is the labeling cost high, but it also limits its application in complex scenarios. Summary of the invention

[0005] The purpose of this application is to provide a multimodal automatic annotation method, device, storage medium and electronic device for automatically extracting information from text descriptions and converting them into image annotations, which not only greatly improves the efficiency and accuracy of image annotation, but also reduces labor costs.

[0006] The present application provides a multimodal automatic annotation method, comprising: The method comprises the steps of: obtaining a text to be processed and an image to be processed, extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and processing the image to be processed by the image processing unit to obtain features of the image to be processed; inputting the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, performing feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and performing feature fusion by using a cross-attention mechanism to obtain correlation scores between the enhanced text features and enhanced image features corresponding to different image regions; based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, selecting enhanced image features with the highest correlation with the enhanced text features, and generating query information corresponding to each selected image feature; inputting the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0007] Optionally, the text features to be processed and the image features to be processed are input into the cross-modal decoder, and the text features to be processed and the image features to be processed are enhanced to obtain enhanced text features and enhanced image features, including: inputting the image features to be processed into multiple attention branch units of the cross-modal decoder to obtain regional image features output by each attention branch unit; fusing the regional image features output by each attention branch unit to obtain the enhanced image features; wherein the image to be processed is divided into multiple image regions, and one attention branch unit corresponds to one image region.

[0008] Optionally, the cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: taking the image feature to be processed as a query and the text feature to be processed as a key and a value, and using the cross-attention mechanism to calculate a first correlation score between the image features and text features corresponding to different image regions.

[0009] Optionally, the cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: taking the text feature to be processed as a query and the image feature to be processed as a key and a value, and using the cross-attention mechanism to calculate a second correlation score between the text feature and the image features corresponding to different image regions.

[0010] Optionally, the cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: calculating the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions based on the first correlation score and the second correlation score.

[0011] Optionally, based on the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, the enhanced image feature with the highest correlation with the enhanced text feature is selected, and query information corresponding to each selected image feature is generated, including: based on the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, at least one image region with the highest correlation with the enhanced text feature is screened out from the multiple image regions, and query information corresponding to each image region of the at least one image region is generated; wherein the query information includes: feature parameters and position information; the feature parameters are updated by back propagation during the training phase.

[0012] Optionally, the multimodal target detection unit includes: a prediction head and multiple decoders; each decoder includes: cross-attention from image to text, cross-attention from text to image; the query information corresponding to the enhanced text feature, the enhanced image feature and each selected image feature is input into the multimodal target detection unit to obtain the annotation information corresponding to each query information, including: inputting the enhanced text feature, the enhanced image feature and the query information corresponding to each selected image feature into the multiple decoders in sequence to obtain multimodal features; inputting the multimodal features into the prediction head to obtain the annotation information corresponding to each query information; wherein the multiple decoders are connected in sequence, and the output of the previous decoder is used as the input of the subsequent decoder.

[0013] The present application also provides a multi-modal automatic annotation device, comprising: A feature extraction module is used to obtain the text to be processed and the image to be processed, and to extract the features of the text to be processed by the natural language processing unit to obtain the features of the text to be processed, and to process the image to be processed by the image processing unit to obtain the features of the image to be processed; a feature fusion module is used to input the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, perform feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and perform feature fusion using a cross-attention mechanism to obtain the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions; a query information generation module is used to select the enhanced image features with the highest correlation with the enhanced text features based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, and generate query information corresponding to each selected image feature; an annotation information generation module is used to input the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0014] Optionally, the feature fusion module is specifically used to input the features of the image to be processed into multiple attention branch units of the cross-modal decoder to obtain regional image features output by each attention branch unit; the feature fusion module is also specifically used to fuse the regional image features output by each attention branch unit to obtain the enhanced image features; wherein, the image to be processed is divided into multiple image regions, and one attention branch unit corresponds to one image region.

[0015] Optionally, the feature fusion module is specifically used to use the image features to be processed as queries and the text features to be processed as keys and values, and use a cross-attention mechanism to calculate a first correlation score between image features and text features corresponding to different image regions.

[0016] Optionally, the feature fusion module is specifically used to use the text features to be processed as queries and the image features to be processed as keys and values, and to use a cross-attention mechanism to calculate a second correlation score between the text features and the image features corresponding to different image regions.

[0017] Optionally, the feature fusion module is specifically configured to calculate a correlation score between the enhanced text feature and enhanced image features corresponding to different image regions based on the first correlation score and the second correlation score.

[0018] Optionally, the query information generation module is specifically used to screen out at least one image region with the highest correlation with the enhanced text feature from the multiple image regions based on the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, and generate query information corresponding to each image region in the at least one image region; wherein the query information includes: feature parameters and position information; the feature parameters are updated by back propagation during the training phase.

[0019] Optionally, the annotation information generation module is specifically used to input the enhanced text feature, the enhanced image feature and the query information corresponding to each selected image feature into the multiple decoders in sequence to obtain multimodal features; the annotation information generation module is also specifically used to input the multimodal features into the prediction head to obtain annotation information corresponding to each query information; wherein the multiple decoders are connected in sequence, and the output of the previous decoder serves as the input of the subsequent decoder.

[0020] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the multimodal automatic annotation methods described above.

[0021] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the multimodal automatic annotation methods described above are implemented.

[0022] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the multimodal automatic annotation methods described above are implemented.

[0023] The multimodal automatic annotation method, device, storage medium and electronic device provided by the present application obtain a text to be processed and an image to be processed, and perform feature extraction on the text to be processed by the natural language processing unit to obtain features of the text to be processed, and process the image to be processed by the image processing unit to obtain features of the image to be processed; input the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, perform feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and use a cross-attention mechanism to perform feature fusion to obtain correlation scores between the enhanced text features and enhanced image features corresponding to different image regions; based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, select the enhanced image features with the highest correlation with the enhanced text features, and generate query information corresponding to each selected image feature; input the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information. In this way, information can be automatically extracted from text descriptions and converted into image annotations, which not only greatly improves the efficiency and accuracy of image annotation but also reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 This is one of the flowcharts of the multimodal automatic annotation method provided by this application; Figure 2 This is the second flowchart of the multimodal automatic annotation method provided by this application; Figure 3 It is a structural schematic diagram of a multi-modal automatic annotation device provided by the present application; Figure 4 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] With the rapid development of artificial intelligence technology, especially the continuous progress in the field of computer vision, image annotation technology has become a key link in realizing automatic target detection and recognition. Existing image annotation methods mainly rely on manual operation, which is to manually select specific areas in the image to annotate the name and location of the object. Most of the existing automatic annotation systems focus on fixed objects. The object must be manually annotated first, and the target detection model is trained by annotated data to obtain a pre-trained model with recognition function, which further realizes the automatic annotation of the target object.

[0029] The labeling methods in related technologies can only automatically label fixed object categories. New categories need to be manually labeled and retrained. Moreover, the labeling methods in related technologies lack the ability to process multimodal data, that is, they cannot process image and text information at the same time. Not only is the labeling efficiency low and the cost high, but it also limits its application in complex scenarios.

[0030] In view of the above technical problems existing in the related art, the embodiments of the present application provide a multimodal automatic annotation system and an automatic annotation method performed by the system, which can automatically extract information from text descriptions and convert them into image annotations, and has important practical significance and application value for improving the efficiency and accuracy of image annotation and reducing labor costs. Figure 1As shown, the method includes: 1. A multimodal input interface receives text data and image data to be annotated; 2. Extracting text features in the text data and image features in the image data; 3. A cross-modal decoder fuses the features of the above two modalities, identifies the target object specified in the text in the image, and generates annotation information.

[0031] The following is a detailed description of the multimodal automatic annotation method for a multimodal automatic annotation system provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings. The multimodal automatic annotation system includes: a natural language processing unit, an image processing unit, a cross-modal decoder, and a multimodal target detection unit.

[0032] like Figure 2 As shown, an embodiment of the present application provides a multimodal automatic annotation method, which may include the following steps 201 to 204: Step 201, obtaining a text to be processed and an image to be processed, and extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and processing the image to be processed by the image processing unit to obtain features of the image to be processed.

[0033] Exemplarily, the user can input a text description and an input image through the system interface to obtain the above-mentioned text to be processed and image to be processed. The text description contains the name, attributes or other relevant information of the object, and multiple objects can be detected at the same time. The input image is the target scene to be annotated, and multiple image data can be input at the same time.

[0034] For example, after obtaining the text to be processed and the image to be processed, the natural language processing unit can analyze the input text description to extract keywords and features, such as object name, color and other attributes. The image processing unit extracts features from the input image to obtain multi-scale image features, i.e., the above-mentioned image features to be processed.

[0035] Step 202: input the to-be-processed text features and the to-be-processed image features into the cross-modal decoder, perform feature enhancement on the to-be-processed text features and the to-be-processed image features to obtain enhanced text features and enhanced image features, and perform feature fusion using a cross-attention mechanism to obtain a correlation score between the enhanced text features and enhanced image features corresponding to different image regions.

[0036] Exemplarily, after receiving the above-mentioned text features to be processed and image features to be processed, the multimodal feature fusion unit of the cross-modal decoder uses the self-attention mechanism to enhance the text features and uses the enhanced multi-head self-attention module to enhance the image features. Then, feature fusion is achieved by adding cross-attention from image to text and cross-attention from text to image.

[0037] Specifically, in the above step 202, the step of enhancing the image features may include the following steps 202a1 and 202a2: Step 202a1: input the image features to be processed into multiple attention branch units of the cross-modal decoder to obtain regional image features output by each attention branch unit.

[0038] Step 202a2: After fusing the regional image features output by each attention branch unit, the enhanced image features are obtained.

[0039] The image to be processed is divided into multiple image regions, and one attention branch unit corresponds to one image region.

[0040] Exemplarily, an embodiment of the present application provides an enhanced multi-head self-attention module, which combines deformable convolution and channel attention mechanisms, and uses the dynamic adjustment capability of deformable convolution to adapt to the specific features of the image content, such as edges, corners, or specific texture patterns. Deformable convolution is a special convolution operation that allows the shape of the convolution kernel to be dynamically adjusted based on the input feature map. This adjustment is achieved by adding additional offsets that can learn how to better align to the boundaries of the target object. The advantage of this is that the model can more flexibly capture the shape and position changes of the target, thereby improving the ability to locate and recognize the target.

[0041] Exemplarily, the same structure and parameters are used for each branch of the multi-head self-attention module. Each branch only focuses on a local area of ​​the original image features, thereby enhancing the model's perception of local features of the image and improving the model's ability to recognize small but critical differences in the image. The multi-scale image features obtained in the above steps are input into the multi-head self-attention module. The input image features will be divided into multiple local areas, so that the attention can be limited to the local area. Each local area enters a branch, and each branch is an enhanced attention module. Finally, the outputs of all branches are merged to form the final enhanced image feature representation.

[0042] Exemplarily, after obtaining the enhanced text features and enhanced image features, feature fusion is required. Specifically, the step of fusing the enhanced text features and enhanced image features in the above step 202 may include the following steps 202b1 and 202b2: Step 202b1: using the image features to be processed as queries and the text features to be processed as keys and values, and using a cross-attention mechanism to calculate a first correlation score between image features and text features corresponding to different image regions.

[0043] For example, in the image-to-text cross attention, image features are used as queries, text features are used as keys and values, and the correlation scores between image features and text features are calculated through the cross attention mechanism.

[0044] Step 202b2: using the to-be-processed text features as queries and the to-be-processed image features as keys and values, and using a cross-attention mechanism to calculate a second correlation score between the text features and the image features corresponding to different image regions.

[0045] For example, in the cross-attention from text to image, text features are used as queries, image features are used as keys and values, and the correlation scores between text features and image features are calculated through the cross-attention mechanism.

[0046] It can be understood that through the above two cross-attention mechanisms, the model can effectively fuse image features and text features, improve the model's understanding of image content, and enhance its responsiveness to text descriptions. This tightly coupled feature fusion method helps the model achieve better performance in open set object detection tasks, especially when dealing with categories that have not been seen during training.

[0047] Specifically, in the above step 202, the step of calculating the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions may further include the following step 202c: Step 202c: Calculate the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions based on the first correlation score and the second correlation score.

[0048] Exemplarily, in the above steps, based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, the region most relevant to the input text can be selected through the text features. The text features and the dynamic anchor boxes are combined to determine which anchor box regions are most relevant to the input text through language-guided query selection.

[0049] Step 203: Based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, the enhanced image features with the highest correlation with the enhanced text features are selected, and query information corresponding to each selected image feature is generated.

[0050] For example, based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, the image features most correlated with the text features can be selected. These image features are considered to be the regions most likely to contain the object described by the text.

[0051] Specifically, the above step 203 may further include the following step 203a: Step 203a: based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, at least one image region having the highest correlation with the enhanced text features is screened out from the multiple image regions, and query information corresponding to each image region in the at least one image region is generated.

[0052] The query information includes: feature parameters and location information; the feature parameters are updated by back propagation during the training phase.

[0053] Exemplarily, in an embodiment of the present application, a query information is generated for each selected image feature, and each query information contains the following content: learnable parameters and position information. The learnable parameters are learnable in the training stage of the model, which means that during the training process, the model can learn how to better match the object described by the input text. When the model starts training, the learnable parameters are randomly initialized and updated through the back propagation algorithm during the training process. The loss function tells the model the difference between the current learnable parameters and the actual target object to capture the local feature representation of the image that is most relevant to the input text. These learnable parameters can be regarded as the model's "understanding" or "representation" of the object described by the input text, and they are continuously optimized as the training data accumulates. Through repeated training and adjustment, these content parts gradually learn how to capture the key features of the input text description, thereby more accurately identifying and locating the target object. The position information is represented by a dynamic anchor box. The dynamic anchor box is a flexible bounding box representation that allows the model to generate bounding boxes of different positions and sizes for each object, increasing the model's adaptability to changes in the shape and size of various objects.

[0054] Step 204: input the query information corresponding to the enhanced text feature, the enhanced image feature and each selected image feature into the multimodal object detection unit to obtain annotation information corresponding to each query information.

[0055] Exemplarily, after obtaining the query information corresponding to each selected image feature, the enhanced text feature, the enhanced image feature and the query information corresponding to each selected image feature can be input into the multimodal target detection unit for target detection to generate annotation information corresponding to each query information.

[0056] Specifically, the multimodal target detection unit includes: a prediction head and multiple decoders; each decoder includes: image-to-text cross attention, text-to-image cross attention, the above step 204 may also include the following steps 204a1 and 204a2: Step 204a1: input the enhanced text feature, the enhanced image feature and the query information corresponding to each selected image feature into the multiple decoders in sequence to obtain a multimodal feature.

[0057] Step 204a2: input the multimodal features into the prediction head to obtain target labeling information corresponding to each query information.

[0058] The multiple decoders are connected in sequence, and the output of the previous decoder serves as the input of the subsequent decoder.

[0059] Exemplarily, the multimodal target detection unit is used to perform accurate target detection, and is composed of multiple decoders and a prediction head. It inputs the enhanced text features and enhanced image features obtained in step 3, and the query information obtained in the above steps, and outputs the position, size and category of the final predicted target of each query.

[0060] Exemplarily, each decoder consists of a self-attention, an image-to-text cross-attention, a text-to-image cross-attention, and a fully connected layer. Through multiple decoder layers, the model can gradually deepen the fusion of image and text features at different stages. Each layer can add more feature fusion, helping the model to better align and understand cross-modal information. More decoder layers mean that the model has a stronger ability to express complex relationships and patterns, especially when dealing with complex interactions between vision and language. At the end of multiple decoders, the output of the decoder, that is, the final multimodal feature, is passed to the prediction head, which outputs the predicted bounding box, category, and category probability.

[0061] For example, after obtaining the above-mentioned annotation information, a YOLO-formatted annotation file and a result image with a detection frame can be generated. These annotation data files can be directly used for training and verification of the YOLO target detection model.

[0062] The multimodal automatic annotation method provided in the embodiment of the present application obtains the text to be processed and the image to be processed, and extracts the features of the text to be processed by the natural language processing unit to obtain the features of the text to be processed, and processes the image to be processed by the image processing unit to obtain the features of the image to be processed; the features of the text to be processed and the features of the image to be processed are input into the cross-modal decoder, the features of the text to be processed and the features of the image to be processed are enhanced to obtain enhanced text features and enhanced image features, and the cross-attention mechanism is used to perform feature fusion to obtain the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions; based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, the enhanced image features with the highest correlation with the enhanced text features are selected, and query information corresponding to each selected image feature is generated; the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature are input into the multimodal target detection unit to obtain the annotation information corresponding to each query information. In this way, information can be automatically extracted from the text description and converted into image annotation, which not only greatly improves the efficiency and accuracy of image annotation, but also reduces labor costs.

[0063] It should be noted that the multimodal automatic annotation method provided in the embodiment of the present application can be executed by a multimodal automatic annotation device, or a control module in the multimodal automatic annotation device for executing the multimodal automatic annotation method. In the embodiment of the present application, the multimodal automatic annotation device executing the multimodal automatic annotation method is taken as an example to illustrate the multimodal automatic annotation device provided in the embodiment of the present application.

[0064] It should be noted that in the embodiments of the present application, the multimodal automatic annotation methods shown in the above-mentioned method drawings are all illustrated by taking one of the drawings in the embodiments of the present application as an example. In specific implementation, the multimodal automatic annotation methods shown in the above-mentioned method drawings can also be implemented in combination with any other drawings that can be combined as shown in the above-mentioned embodiments, which will not be repeated here.

[0065] The multimodal automatic annotation device provided by the present application is described below, and the multimodal automatic annotation method described below and above can be referenced to each other.

[0066] Figure 3 A schematic diagram of the structure of a multi-modal automatic annotation device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, specifically including: The feature extraction module 301 is used to obtain the text to be processed and the image to be processed, and to extract the features of the text to be processed by the natural language processing unit to obtain the features of the text to be processed, and to process the image to be processed by the image processing unit to obtain the features of the image to be processed; the feature fusion module 302 is used to input the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, perform feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and use the cross-attention mechanism to perform feature fusion to obtain the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions; the query information generation module 303 is used to select the enhanced image features with the highest correlation with the enhanced text features based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, and generate query information corresponding to each selected image feature; the annotation information generation module 304 is used to input the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0067] Optionally, the feature fusion module 302 is specifically used to input the image features to be processed into multiple attention branch units of the cross-modal decoder to obtain regional image features output by each attention branch unit; the feature fusion module 302 is also specifically used to fuse the regional image features output by each attention branch unit to obtain the enhanced image features; wherein, the image to be processed is divided into multiple image regions, and one attention branch unit corresponds to one image region.

[0068] Optionally, the feature fusion module 302 is specifically used to use the image features to be processed as queries and the text features to be processed as keys and values, and use a cross-attention mechanism to calculate a first correlation score between image features and text features corresponding to different image regions.

[0069] Optionally, the feature fusion module 302 is specifically used to use the text features to be processed as queries and the image features to be processed as keys and values, and use a cross-attention mechanism to calculate a second correlation score between the text features and the image features corresponding to different image regions.

[0070] Optionally, the feature fusion module 302 is specifically configured to calculate a correlation score between the enhanced text feature and enhanced image features corresponding to different image regions based on the first correlation score and the second correlation score.

[0071] Optionally, the query information generation module 303 is specifically used to screen out at least one image region with the highest correlation with the enhanced text feature from the multiple image regions based on the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, and generate query information corresponding to each image region in the at least one image region; wherein the query information includes: feature parameters and position information; the feature parameters are updated by back propagation during the training phase.

[0072] Optionally, the annotation information generation module 304 is specifically used to input the enhanced text feature, the enhanced image feature and the query information corresponding to each selected image feature into the multiple decoders in sequence to obtain multimodal features; the annotation information generation module 304 is also specifically used to input the multimodal features into the prediction head to obtain annotation information corresponding to each query information; wherein the multiple decoders are connected in sequence, and the output of the previous decoder serves as the input of the subsequent decoder.

[0073] The multimodal automatic annotation device provided by the present application obtains the text to be processed and the image to be processed, and extracts the features of the text to be processed by the natural language processing unit to obtain the features of the text to be processed, and processes the image to be processed by the image processing unit to obtain the features of the image to be processed; the features of the text to be processed and the features of the image to be processed are input into the cross-modal decoder, and the features of the text to be processed and the features of the image to be processed are enhanced to obtain enhanced text features and enhanced image features, and the cross-attention mechanism is used to perform feature fusion to obtain the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions; based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, the enhanced image features with the highest correlation with the enhanced text features are selected, and query information corresponding to each selected image feature is generated; the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature are input into the multimodal target detection unit to obtain the annotation information corresponding to each query information. In this way, information can be automatically extracted from the text description and converted into image annotation, which not only greatly improves the efficiency and accuracy of image annotation, but also reduces labor costs.

[0074] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the multimodal automatic annotation method, which includes: obtaining a text to be processed and an image to be processed, and extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and processing the image to be processed by the image processing unit to obtain features of the image to be processed; inputting the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, performing feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and using a cross-attention mechanism to perform feature fusion to obtain a correlation score between the enhanced text features and enhanced image features corresponding to different image regions; based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, selecting an enhanced image feature with the highest correlation with the enhanced text feature, and generating query information corresponding to each selected image feature; inputting the enhanced text features, the enhanced image features, and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0075] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multimodal automatic annotation method provided by the above methods, and the method includes: obtaining a text to be processed and an image to be processed, and extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and processing the image to be processed by the image processing unit to obtain features of the image to be processed; inputting the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, and extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed; and processing the image to be processed by the image processing unit to obtain features of the image to be processed; The method further comprises the steps of: performing feature enhancement on the processed text features and the image features to be processed to obtain enhanced text features and enhanced image features, and performing feature fusion using a cross-attention mechanism to obtain correlation scores between the enhanced text features and enhanced image features corresponding to different image regions; selecting an enhanced image feature with the highest correlation with the enhanced text feature based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, and generating query information corresponding to each selected image feature; inputting the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0077] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the multimodal automatic annotation method provided above, the method comprising: obtaining a text to be processed and an image to be processed, and extracting features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and processing the image to be processed by the image processing unit to obtain features of the image to be processed; inputting the features of the text to be processed and the features of the image to be processed into the cross-modal decoder, performing feature enhancement on the features of the text to be processed and the features of the image to be processed to obtain enhanced text features and enhanced image features, and performing feature fusion using a cross-attention mechanism to obtain a correlation score between the enhanced text features and enhanced image features corresponding to different image regions; based on the correlation score between the enhanced text features and the enhanced image features corresponding to different image regions, selecting an enhanced image feature with the highest correlation with the enhanced text feature, and generating query information corresponding to each selected image feature; inputting the enhanced text features, the enhanced image features and the query information corresponding to each selected image feature into the multimodal target detection unit to obtain annotation information corresponding to each query information.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal automatic annotation method, characterized in that: Applied to a multimodal automatic annotation system, the automatic annotation system comprises: a natural language processing unit, an image processing unit, a cross-modal decoder and a multimodal target detection unit; The method comprises: Acquire a text to be processed and an image to be processed, and extract features of the text to be processed by the natural language processing unit to obtain features of the text to be processed, and process the image to be processed by the image processing unit to obtain features of the image to be processed; Inputting the to-be-processed text features and the to-be-processed image features into the cross-modal decoder, performing feature enhancement on the to-be-processed text features and the to-be-processed image features to obtain enhanced text features and enhanced image features, and performing feature fusion using a cross-attention mechanism to obtain correlation scores between the enhanced text features and enhanced image features corresponding to different image regions; Based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, the enhanced image features with the highest correlation with the enhanced text features are selected, and query information corresponding to each selected image feature is generated; The enhanced text feature, the enhanced image feature and query information corresponding to each selected image feature are input into the multimodal object detection unit to obtain annotation information corresponding to each query information.

2. The method according to claim 1, characterized in that The step of inputting the to-be-processed text features and the to-be-processed image features into the cross-modal decoder, and performing feature enhancement on the to-be-processed text features and the to-be-processed image features to obtain enhanced text features and enhanced image features comprises: Inputting the image features to be processed into multiple attention branch units of the cross-modal decoder to obtain regional image features output by each attention branch unit; After fusing the regional image features output by each attention branch unit, the enhanced image features are obtained; The image to be processed is divided into multiple image regions, and one attention branch unit corresponds to one image region.

3. The method according to claim 1 or 2, characterized in that: The cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: The image features to be processed are used as queries and the text features to be processed are used as keys and values, and a cross-attention mechanism is used to calculate first correlation scores between image features and text features corresponding to different image regions.

4. The method according to claim 3, characterized in that The cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: The text features to be processed are used as queries and the image features to be processed are used as keys and values, and a cross-attention mechanism is used to calculate a second correlation score between the text features and the image features corresponding to different image regions.

5. The method according to claim 4, characterized in that The cross-attention mechanism is used to perform feature fusion to obtain a correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, including: Based on the first correlation score and the second correlation score, a correlation score between the enhanced text feature and enhanced image features corresponding to different image regions is calculated.

6. The method according to claim 1 or 5, characterized in that: The step of selecting an enhanced image feature with the highest correlation with the enhanced text feature based on the correlation score between the enhanced text feature and the enhanced image features corresponding to different image regions, and generating query information corresponding to each selected image feature includes: Based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, at least one image region having the highest correlation with the enhanced text features is screened out from the multiple image regions, and query information corresponding to each image region in the at least one image region is generated; The query information includes: feature parameters and location information; the feature parameters are updated by back propagation during the training phase.

7. The method according to claim 6, characterized in that The multimodal object detection unit includes: a prediction head and a plurality of decoders; each decoder includes: image-to-text cross attention, text-to-image cross attention; The step of inputting the query information corresponding to the enhanced text feature, the enhanced image feature, and each selected image feature into the multimodal object detection unit to obtain annotation information corresponding to each query information includes: Inputting the enhanced text feature, the enhanced image feature and query information corresponding to each selected image feature into the multiple decoders in sequence to obtain a multimodal feature; Inputting the multimodal features into the prediction head to obtain labeling information corresponding to each query information; The multiple decoders are connected in sequence, and the output of the previous decoder serves as the input of the subsequent decoder.

8. A multi-modal automatic annotation device, characterized in that: Applied to a multimodal automatic annotation system, the automatic annotation system comprises: a natural language processing unit, an image processing unit, a cross-modal decoder and a multimodal target detection unit; The device comprises: A feature extraction module, used to obtain a text to be processed and an image to be processed, and to extract features from the text to be processed by the natural language processing unit to obtain features of the text to be processed, and to process the image to be processed by the image processing unit to obtain features of the image to be processed; A feature fusion module, used for inputting the to-be-processed text features and the to-be-processed image features into the cross-modal decoder, performing feature enhancement on the to-be-processed text features and the to-be-processed image features to obtain enhanced text features and enhanced image features, and performing feature fusion using a cross-attention mechanism to obtain a correlation score between the enhanced text features and enhanced image features corresponding to different image regions; A query information generating module, configured to select, based on the correlation scores between the enhanced text features and the enhanced image features corresponding to different image regions, the enhanced image features with the highest correlation with the enhanced text features, and generate query information corresponding to each selected image feature; The annotation information generation module is used to input the query information corresponding to the enhanced text feature, the enhanced image feature and each selected image feature into the multimodal target detection unit to obtain the annotation information corresponding to each query information.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal automatic annotation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the multimodal automatic annotation method as claimed in any one of claims 1 to 7 are implemented.

Images