An interactive image labeling method and system based on a multi-modal model

Abstract

The application discloses an interactive image labeling method and system based on a multi-modal model, and comprises the following steps: constructing a target detection model; obtaining a to-be-detected image and a prompt word text, performing word segmentation on the prompt word text, and obtaining a plurality of word units as different to-be-detected target categories; for each to-be-detected target category, extracting a corresponding negative category existing in a negative prompt word, and generating a word unit index table; setting a confidence threshold, inputting the to-be-detected image, the prompt word text and the word unit index table into the target detection model, and generating an intermediate detection result; setting an IoU threshold of inter-class non-maximum suppression, filtering the intermediate detection result in combination with the negative prompt word, and generating a final detection result. The application can obtain an accurate detection result without training a specific category, and can interactively reduce false detection through the negative prompt word and the description prompt word, and automatically generate accurate labeling for the image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image annotation, and particularly relates to an interactive image annotation method and system based on a multi-modal model. BACKGROUND

[0002] At present, in the field of data annotation of target detection tasks, there are many annotation tool software, such as the most widely used LabelImg and Labelme. Using such traditional annotation software, each image needs to be checked and annotated manually one by one, and the work efficiency is low. Subsequently, some more intelligent annotation software appeared, for example, AnyLabeling. In addition to the advantages of traditional annotation software, this kind of software also supports annotation using YOLO and other target detection models, but can only automatically annotate pre-trained categories. In addition, the automatic annotation process is only to use the image as the input of the detection model, which has poor detection performance in the case of large difference between sample distribution and training. SUMMARY

[0003] To solve the problems in the prior art, the application provides an interactive image annotation method and system based on a multi-modal model, which can accurately obtain detection results without collecting specific category data for training, and improve data annotation efficiency.

[0004] The application adopts the following technical solutions.

[0005] The first aspect of the application provides an interactive image annotation method based on a multi-modal model, which comprises the following steps:

[0006] S1, constructing a target detection model, wherein the target detection model is obtained by training a plurality of sets of multi-modal data through a deep neural network to minimize a total loss function, and each set of multi-modal data comprises an image and a text;

[0007] S2, obtaining an image to be detected and a prompt word text, wherein the prompt word text comprises a positive prompt word and / or a negative prompt word and / or a description prompt word; the positive prompt word contains a target category to be detected, the negative prompt word is empty or contains a positive-negative category pair, and the description prompt word contains features of the positive category and the negative category and a position description of the target to be detected in the image;

[0008] S3, performing word segmentation on the prompt word text to obtain a plurality of word units as different target categories to be detected, and for each target category to be detected, extracting a corresponding negative category existing in the negative prompt word to generate a word unit index table;

[0009] S4, set a confidence threshold, input the to-be-detected image, the prompt word text and the word index table into the target detection model, the target detection model obtains the features of the to-be-detected target class and the negative class through the description prompt word, and labels the part corresponding to the to-be-detected class on the to-be-detected image with a confidence greater than the confidence threshold as an intermediate detection;

[0010] S5, set an IoU threshold for inter-class non-maximum suppression, check each to-be-detected target class in the intermediate detection, if there is a corresponding negative class, perform inter-class non-maximum suppression on all detection boxes of the to-be-detected target class and the corresponding negative class, and after removing all detection boxes of the negative class in the output result of the non-maximum suppression, the remaining labeled detection boxes are used as the final image labeling result.

[0011] Optionally, in S1, the target detection model is trained by using multiple sets of multi-modal data through a deep neural network to minimize a total loss function, comprising:

[0012] input image and text;

[0013] extracting multi-scale original image features from the image using an image backbone network, and extracting original text features from the text using a text backbone network;

[0014] cross-modal feature fusion is performed on the multi-scale original image features and the original text features, the most relevant image features in the image features after feature fusion are selected in combination with the text features after feature fusion, and cross-modal decoding is performed to obtain the prediction box and the prediction class of the input image;

[0015] constructing a total loss function of the target detection model, and obtaining the best training parameters by minimizing the total loss function value to obtain the target detection model.

[0016] Optionally, selecting the most relevant image features in the image features after feature fusion, comprising:

[0017] querying in the image features after feature fusion, and the querying process is as follows:

[0018]

[0019] wherein I is an image feature, b x i x c in size, T is a text feature, b x t x c in size, b represents batch size, t is the current text feature length, c represents the number of channels, i represents the i-th image feature, logits bit represents the value of the i-th image feature and the t-th text feature position in the b-th batch of logits output by the target detection model, I bic and T btcrespectively represent the value of the original image feature and the original text feature at the corresponding position of the fused feature;

[0020] Calculate the maximum value of the logits of each sample output:

[0021]

[0022] where logits bj represents the value of the jth column of the bth sample of logits, and M is a two-dimensional tensor storing the maximum logits of each sample;

[0023] Obtain the query q most relevant to the input text:

[0024]

[0025] where, represents finding the set i1, i2,..., i k , such that i1, i2,..., i k is the image feature most relevant to the input text.

[0026] Optionally, the total loss function of the target detection model is composed of a bounding box regression loss and a classification loss, and is represented by the following formula:

[0027]

[0028] where α and β are the bounding box regression loss weight and the category loss weight, respectively, W d , b d , and θ are the training parameters of the target detection model, L reg is the bounding box regression loss, and L cls is the classification loss.

[0029] Optionally, the bounding box regression loss includes an L1 loss and a GIOU loss considering the overlap of bounding boxes, and the bounding box regression loss is represented by the following formula:

[0030] L reg = L1 + L GIOU

[0031] where L1 is the L1 loss, and L GIOU is the GIOU loss.

[0032] Optionally, the L1 loss is represented by the following formula:

[0033]

[0034] where,

[0035] gt ij the jth coordinate of the prediction box of the ith sample,

[0036] det ij the jth coordinate of the real box of the ith sample,

[0037] N is the number of samples.

[0038] Optionally, the GIOU loss is represented by the following formula:

[0039]

[0040] wherein gt and det represent the prediction box and the real box respectively,

[0041] C represents the minimum square box that can cover gt and det,

[0042] area(C) represents the area of C.

[0043] Optionally, the logits of each text label are predicted by calculating the dot product of each query and the text feature, and the classification loss is calculated according to each logits, and the classification loss is calculated by the following formula:

[0044]

[0045] wherein,

[0046] p i is the confidence of the correct logits predicted by the model for the ith sample,

[0047] α i is a balance factor for adjusting the weight of different categories,

[0048] γ is a regulation factor for controlling the degree of weight attenuation of the sample,

[0049] q·T j represents the dot product of the query and each text feature, q is the query, T j is the jth text feature, and n is the number of text features,

[0050] N is the number of samples.

[0051] Optionally, the positive class and / or the negative class are interactively provided in the description prompt word.

[0052] The second aspect of the application provides an image labeling system, comprising:

[0053] A construction module is configured to construct a target detection model, wherein the target detection model is trained by a deep neural network using multiple sets of multi-modal data to minimize a total loss function, and each set of the multiple sets of multi-modal data comprises an image and text;

[0054] An acquisition module is configured to acquire an image to be detected and prompt word text, wherein the prompt word text comprises positive prompt words and / or negative prompt words and / or description prompt words; the positive prompt words contain a positive category, the positive category is a target category to be detected, the negative prompt words are empty or contain a positive category-negative category pair, the negative category is a category group of the positive category, and the description prompt words contain features of the positive category and the negative category and a position description of the positive category and the negative category in the image.

[0055] An extraction module is configured to perform word segmentation on the prompt word text, and obtain a plurality of word units as different target categories to be detected; for each target category to be detected, the extraction module extracts a corresponding negative category existing in the negative prompt words to generate a word unit index table.

[0056] A first detection module is configured to set a confidence threshold, input the image to be detected, the prompt word text and the word unit index table into the target detection model, acquire features of the target category to be detected and the negative category by the description prompt words, and label a part corresponding to the target category to be detected in the image as an intermediate detection result, wherein the part has a confidence greater than the confidence threshold.

[0057] A second detection module is configured to set an IoU threshold of inter-class non-maximum suppression, check each target category to be detected in the intermediate detection result, if the target category to be detected has a corresponding negative category, perform inter-class non-maximum suppression on all detection boxes of the target category to be detected and the corresponding negative category, remove all detection boxes of the negative category from an output result of the non-maximum suppression, and obtain a final image labeling result from remaining labeled detection boxes.

[0058] Compared with the prior art, the present application has at least the following beneficial effects:

[0059] The present application comprehensively uses deep learning, artificial intelligence and human-computer interaction to solve the problem of automatic generation of image labeling, and can accurately obtain a detection result without collecting data of specific categories for training, so that the detection result can be used as actual image labeling after artificial review. The present application solves the problem of large amount of manpower required for full manual labeling, improves the efficiency of data labeling, and thus reduces the research and development cost of subsequent training of a target detection model. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative labor on the basis of these drawings. Among them:

[0061] Figure 1 A flowchart of an interactive image labeling method based on a multi-modal model is provided for the embodiments of the present application.

[0062] Figure 2 A processing flowchart of a target detection model is provided for the embodiments of the present application.

[0063] Figure 3 A schematic diagram of an inter-class non-maximum suppression process is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. The embodiments described in the present application are only some embodiments of the present application, but not all embodiments. Based on the spirit of the present application, all other embodiments obtained by those skilled in the art without any creative labor are within the protection scope of the present application.

[0065] In combination with Figure 1 As shown in the drawings, the embodiment 1 of the present application provides an interactive image labeling method based on a multi-modal model, which comprises the following steps:

[0066] S1, constructing a target detection model, the target detection model is obtained by training a deep neural network using multiple sets of data to minimize the total loss function, each set of data of the multiple sets of data comprises image data and text data.

[0067] A large-scale public data set is used to train an open set target detection model based on a deep neural network Transformer, and a model capable of detecting non-predefined categories is obtained.

[0068] In S1, training the target detection model by the deep neural network using multiple sets of multi-modal data comprises:

[0069] inputting an image and text;

[0070] extracting multi-scale original image features from the image using an image backbone network, and extracting original text features from the text using a text backbone network;

[0071] The multi-scale original image features and the original text features are cross-modal feature fused, the image features most relevant to the input text are queried and selected from the image features after the feature fusion, and the cross-modal decoding is performed in combination with the text features after the feature fusion, to obtain the prediction frame and the prediction category of the input image.

[0072] A total loss function of the target detection model is constructed, and the best training parameter is obtained by minimizing the total loss function, to obtain the target detection model.

[0073] Specifically, as shown in Figure 2 , given a (image, text) pair, multi-scale original image features are extracted using a backbone network such as Transformer, and original text features are extracted using a text backbone such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture). Then, the image and text features are cross-modal feature fused. The features more relevant to the input text are selected as the decoder query, so as to guide the target detection using the input text. Then, the text features and the query are jointly decoded in a cross-modal manner, and the decoding output is a prediction bounding box, and the text features are output as a prediction category. The query selection process is as follows:

[0074]

[0075] wherein I is an image feature, the size is b x i x c, T is a text feature, the size is b x t x c. b represents a batch size, t is a current text feature length, c represents a channel number, logits bit represents a value of the logits output by the target detection model at the bth batch, the ith image, and the tth text feature position, I bic and T btc respectively represent values of the original image features and the original text features at corresponding positions of the features after the feature fusion.

[0076] Then, the maximum value of logits with respect to each sample is calculated:

[0077]

[0078] wherein logits bj represents a value of the jth column of the bth sample of the logits. M is a two-dimensional tensor storing the maximum value of each sample.

[0079] Finally, the query most relevant to the input text is obtained:

[0080]

[0081] wherein, denotes finding a set i1, i2,..., i k such that

[0082] Optionally, the total loss function of the target detection model is composed of both the bounding box regression loss and the classification loss. The bounding box regression loss is composed of the basic L1 loss and the GIOU (Generalized Intersection over Union) loss considering the bounding box overlap:

[0083] L reg = L1 + L GIOU

[0084] wherein the L1 loss is as follows:

[0085]

[0086] wherein gt ij denotes the jth coordinate of the prediction box of the ith sample, det ij denotes the jth coordinate of the real box of the ith sample, and N is the number of samples.

[0087] The GIOU loss is as follows:

[0088]

[0089] wherein gt and det respectively denote the prediction box and the real box, C denotes the minimum square box that can contain gt and det, and area(C) denotes the area of C.

[0090] The logits of each text token are predicted by calculating the dot product of each query and the text feature, and then the classification loss L cls is calculated for each logits to classify the predicted target and the word piece:

[0091]

[0092] wherein p i is the confidence of the correct logits predicted by the model for the ith sample, a i is a balance factor for adjusting the weight of different classes, and g is a regulation factor for controlling the degree of weight decay of the sample, q·T j denotes the dot product of the query and each text feature, q is the query, T j is the jth text feature, n is the number of text features, and N is the number of samples.

[0093] The model learns the optimal parameters by minimizing the total loss:

[0094]

[0095] a, b are frame regression loss weight and class loss weight respectively. Through further training, the training parameters W d , b d , θ of the Transformer model are obtained, and then the target detection model is obtained.

[0096] S2, obtain the image to be detected and the prompt word text; the prompt word text includes positive prompt words and / or negative prompt words and / or description prompt words; wherein the positive prompt words contain the target class to be detected, and different target classes to be detected are separated by conjunctions such as punctuation marks, and, or, etc.; the negative prompt words are empty or contain positive-negative class pairs; the description prompt words contain the features of the positive class and the negative class, and the position description of the target to be detected in the image.

[0097] Wherein the negative prompt words and the description prompt words are optional. The image to be detected, the positive prompt words, the negative prompt words and the description prompt words are respectively denoted as G, P pos , P neg and P desc . The i-th character in the prompt word is denoted as P[i-1], for example, for P pos with content "acat", P[0] is "a", P[2] is "c", and P[0:3] is "a c".

[0098] S3, perform word segmentation on the prompt word text, and obtain a plurality of word units as different target classes to be detected. For each target class to be detected, extract the corresponding negative class existing in the negative prompt words. If the negative prompt words are not empty, extract the corresponding negative class existing in the negative prompt words, and generate a word unit index table.

[0099] Specifically, the word unit index table can be a dictionary D:

[0100] D={c1:(n 1,1 ,n 1,2 ...),c2:(n 2,1 ,n 2,2 ...),...,c i :(n i,1 ,n i,2 ...)}

[0101] Wherein, c i represents the i-th positive class, and n i,k represents the k-th negative class corresponding to the i-th positive class. Specifically, the negative prompt words are changed to all negative classes connected by ".". For example, for P neg"screen:window.lamp:button", and after the change, it becomes "screen.Button". The positive prompt word, the changed negative prompt word, and the description prompt word are connected with ". ", and the part-of-speech is analyzed using a word segmentation tool such as nltk and spaCy, and the part-of-speech of CC (conjunctions such as and, or, etc.), SYM (symbols such as ", ", etc.) is extracted. According to the index where these words are located, the positive and negative prompt words are divided to generate a word index table. For example, for P pos " a cat and a dog ", the word index table is [[0:5], [10:15]], corresponding to " acat " and " adog ".

[0102] S4, set a confidence threshold, input the image to be detected, the prompt word text and the word index table into the target detection model, the target detection model obtains the features of the positive and negative categories of the image to be detected through the description prompt word, and marks the part of the image to be detected corresponding to the confidence of the category to be detected greater than the confidence threshold as the intermediate detection.

[0103] Different description prompt words may obtain different detection results. When the category to be detected is a rare category, the model may not obtain ideal detection results without sufficient description of the category. At this time, multiple images and multiple targets can be detected, and the description prompt word can be modified interactively multiple times to obtain the most suitable description prompt word for this category.

[0104] S5, set the IoU threshold of inter-class non-maximum suppression, check each target category in the intermediate detection, if it has a corresponding negative category, perform inter-class non-maximum suppression on all detection boxes of the target category and the corresponding negative category. In the output result of the non-maximum suppression, after removing all detection boxes of the negative category, the remaining labeled detection boxes are taken as the final image labeling result.

[0105] In this embodiment, by setting the IoU threshold of non-maximum suppression, redundant candidate boxes are removed, the detection result is simplified, it is ensured that each target is detected only once, multiple overlapping candidate boxes are avoided to label the same target, and the accuracy and efficiency of image labeling are improved.

[0106] In combination with Figure 3 , for each key-value pair c i :(n i,1 ,n i,2 ...) in the dictionary D in S3, c i , n i,1 , n i,2...category corresponding detection together as follows inter-class NMS (Non-Maximum Suppression, non-maximum suppression) process:

[0107] Input: bboxes: a two-dimensional array containing the coordinates of the bounding boxes (x1, y1, x2, y2). scores: a one-dimensional array containing the scores of each bounding box. threshold: a float number to specify the IoU threshold for NMS, default is 0.5.

[0108] Output: a list containing the indices of the bounding boxes that are kept.

[0109] The following is a simple algorithm description:

[0110] First, sort the bounding boxes according to scores to get an index list in descending order of scores. Then initialize an empty list keep to store the indices of the bounding boxes to be kept. Then when the sorted index list is not empty, repeat the following operations: take out the bounding box index with the highest score and add it to the keep list. Calculate the intersection over union (IoU) of this bounding box with all other bounding boxes in the sorted index list. Find all bounding box indices whose IoU is less than the given threshold. Update the sorted index list by removing the bounding box indices that are suppressed by NMS. Finally, return the keep list.

[0111] Embodiment 2 of the present application provides an image annotation system running the interactive image annotation method based on the multi-modal model as described in embodiment 1, the system comprising:

[0112] A construction module for constructing a target detection model, wherein the target detection model is trained by using a plurality of sets of multi-modal data through a deep neural network to minimize a total loss function, and each set of multi-modal data includes an image and a text;

[0113] An acquisition module for acquiring an image to be detected and a prompt word text, wherein the prompt word text includes a positive prompt word and / or a negative prompt word and / or a description prompt word; the positive prompt word contains a positive category, the positive category is a target category to be detected, the negative prompt word is empty or contains a positive-negative category pair, the negative category is a category group of the positive category, and the description prompt word contains features of the positive category and the negative category and a position description of the positive category and the negative category in the image.

[0114] An extraction module for performing word segmentation on the prompt word text, obtaining a plurality of word units as different target categories to be detected, and for each target category to be detected, extracting a corresponding negative category existing in the negative prompt word to generate a word unit index table.

[0115] The first detection module is configured to set a confidence threshold, input the image to be detected, the prompt word text and the word index table into the target detection model, acquire the features of the target class to be detected and the negative class by the prompt word, and label the part corresponding to the target class to be detected on the image to be detected as the intermediate detection, wherein the confidence of the part is greater than the confidence threshold.

[0116] The second detection module is configured to set an IoU threshold of inter-class non-maximum suppression, check each target class to be detected in the intermediate detection, if the target class to be detected has a corresponding negative class, perform inter-class non-maximum suppression on all detection boxes of the target class to be detected and the corresponding negative class, remove all detection boxes of the negative class from the output result of the non-maximum suppression, and the remaining labeled detection boxes are used as the final image labeling result.

[0117] Compared with the prior art, the beneficial effects of the present application at least include:

[0118] The present application comprehensively uses deep learning, artificial intelligence and man-machine interaction to accurately obtain the detection result without collecting specific class data for training, and the detection result can be used as the image labeling for actual use after artificial review, thereby solving the problem of large amount of manpower required for full manual labeling, improving the efficiency of data labeling, and reducing the research and development cost of subsequent training of the model.

[0119] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0120] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, wherein the computer readable program instructions are used to cause a processor to implement various aspects of the present disclosure.

[0121] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0122] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0123] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0124] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. An interactive image annotation method based on a multimodal model, characterized in that: The method comprises the following steps: S1. Constructing a target detection model, wherein the target detection model is trained using a deep neural network with a goal of minimizing a total loss function using multiple sets of multimodal data, each set of the multiple sets of multimodal data including images and text; S2. Obtain an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or descriptive prompt words; wherein the positive prompt words include the category of the target to be detected, the negative prompt words are empty or include a positive category-negative category pair, and the descriptive prompt words include features of the positive category and the negative category, and a description of the location of the target to be detected in the image; S3. Segment the prompt word text, and use the obtained word units as different target categories to be detected. For each target category to be detected, extract the corresponding negative category in the negative prompt word and generate a word unit index table; S4. Setting a confidence threshold, inputting the image to be detected, the prompt word text, and the word-unit index table into the object detection model, the object detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected corresponding to the target category with a confidence greater than the confidence threshold as an intermediate detection; S5. Set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection boxes of the target category to be detected and the corresponding negative category, and remove all negative category detection boxes from the output result of non-maximum suppression. The remaining annotated detection boxes are used as the final image annotation results.

2. The interactive image annotation method based on a multimodal model according to claim 1, characterized in that: In S1, the target detection model is trained using multiple sets of multimodal data through a deep neural network with the goal of minimizing the total loss function, including: Input images and text; Use the image backbone network to extract multi-scale original image features from images, and use the text backbone network to extract original text features from texts; Perform cross-modal feature fusion on the multi-scale original image features and the original text features, select the image features most relevant to the input text from the fused image features, and perform cross-modal decoding on them in combination with the fused text features to obtain the predicted box and predicted category of the input image; Construct the total loss function of the target detection model, obtain the optimal training parameters by minimizing the total loss function value, and obtain the target detection model.

3. The interactive image annotation method based on a multimodal model according to claim 2, characterized in that: Select the image features most relevant to the input text from the image features after feature fusion, including: The query process is expressed as follows: Among them, I is the image feature, the size is b×i×c, T is the text feature, the size is b×t×c, b represents the batch size, t is the current text feature length, c represents the number of channels, i represents the i-th image feature, logits bit Represents the value of the bth batch, i-th image feature and t-th text feature position of the target detection model output logits, I bic and T btc Respectively represent the values ​​of the original image features and the original text features at the corresponding positions of the features after feature fusion; Calculate the maximum value of each sample output logits: Among them, logits bj Represents the value of the jth column of the bth sample of logits, and M is a two-dimensional tensor storing the maximum logits of each sample; Get the most relevant query q for the input text: in, Indicates finding the set i1,i2,...,i k , making i1,i2,...,i k is the image feature that is most relevant to the input text.

4. The interactive image annotation method based on a multimodal model according to claim 3, characterized in that: The total loss function of the object detection model consists of the bounding box regression loss and the classification loss, which is expressed as follows: Among them, α and β are the bounding box regression loss weight and category loss weight respectively, W d 、b d , θ are the training parameters of the target detection model, L reg is the bounding box regression loss, L cls is the classification loss.

5. The interactive image annotation method based on a multimodal model according to claim 4, characterized in that: The bounding box regression loss includes L1 loss and GIOU loss considering bounding box overlap. The bounding box regression loss is expressed as follows: L reg =L1+L GIOU Among them, L1 is L1 loss, L GIOU For GIOU loss.

6. The interactive image annotation method based on a multimodal model according to claim 5, characterized in that: The L1 loss is expressed as follows: in, gt ij represents the j-th coordinate of the prediction box of the i-th sample, det ij represents the jth coordinate of the true box of the i-th sample, N is the number of samples.

7. The interactive image annotation method based on a multimodal model according to claim 5, characterized in that: The GIOU loss is expressed as follows: Among them, gt and det represent the predicted box and the real box respectively. C represents the smallest box that can enclose gt and det, area(C) represents the area of ​​C.

8. The interactive image annotation method based on a multimodal model according to claim 4, characterized in that: The logits of each text tag are predicted by calculating the dot product of each query and the text feature, and the classification loss is calculated based on each logits. The classification loss is calculated as follows: in, p i is the confidence of the model predicting the correct logits for the i-th sample, α i is the balancing factor, used to adjust the weights of different categories, γ is a regulating factor used to control the degree of weight attenuation of the sample. q·T j Indicates the dot product between the query and each text feature, q is the query, T j is the jth text feature, n is the number of text features, N is the number of samples.

9. The interactive image annotation method based on a multimodal model according to claim 1, characterized in that: Interactively provide positive and / or negative categories in description prompts.

10. An image annotation system using the interactive image annotation method based on a multimodal model according to any one of claims 1 to 8, characterized in that: include: A construction module for constructing a target detection model, wherein the target detection model is trained using a deep neural network using multiple sets of multimodal data to minimize a total loss function, each set of the multiple sets of multimodal data including images and text; An acquisition module is configured to acquire an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or descriptive prompt words; wherein the positive prompt word includes a positive category, which is the category of the target to be detected; the negative prompt word is empty or includes a positive category-negative category pair, which is the category group described by the positive category; and the descriptive prompt word includes features of the positive category and the negative category, and a description of the location of the positive category and the negative category in the image; The extraction module is used to segment the prompt word text and use the obtained word units as different target categories to be detected. For each target category to be detected, the corresponding negative category present in the negative prompt word is extracted to generate a word unit index table; The first detection module is used to set a confidence threshold, input the image to be detected, the prompt word text, and the word unit index table into the object detection model, and the object detection model obtains the features of the target category and the negative category by describing the prompt word, and marks the parts of the image to be detected corresponding to the target category with a confidence greater than the confidence threshold as intermediate detection; The second detection module is used to set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all negative category detection frames from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

Images