Interactive image annotation method and system based on multi-modal model

Abstract

The invention discloses an interactive image annotation method and system based on a multi-modal model. The method comprises the following steps: constructing a target detection model; the method comprises the steps of obtaining a to-be-detected image and a cue word text, performing word segmentation on the cue word text to obtain a plurality of lexical elements as different to-be-detected target categories, and for each to-be-detected target category, extracting a corresponding negative category existing in a negative cue word to generate a lexical element index table; setting a confidence coefficient threshold value, inputting the to-be-detected image, the prompt word text and the lexical element index table into the target detection model, and generating intermediate detection by the target detection model; and setting an IoU threshold value of inter-class non-maximum suppression, and filtering the intermediate detection in combination with a negative cue word to generate a final detection result. According to the method, an accurate detection result can be obtained on the premise that a specific category is not trained, false detection can be interactively reduced through the negative cue word and the description cue word, and an accurate label is automatically generated for the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image annotation, and in particular, relates to an interactive image annotation method and system based on a multimodal model. Background Art

[0002] Currently, there are many labeling tool software in the field of data labeling for target detection tasks, such as the most widely used LabelImg and Labelme. When using this type of traditional labeling software, each image needs to be manually checked and labeled one by one, which is inefficient. Later, some smarter labeling software appeared, such as AnyLabeling. In addition to combining the advantages of traditional labeling software, this type of software also supports labeling using target detection models such as YOLO, but can only automatically label pre-trained categories. In addition, its automatic labeling process only uses the image as the input of the detection model, which has poor detection performance when the sample distribution is far from that during training. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the present invention provides an interactive image annotation method and system based on a multimodal model, which can accurately obtain detection results without collecting data of specific categories for training, thereby improving data annotation efficiency.

[0004] The present invention adopts the following technical solution.

[0005] A first aspect of the present invention provides an interactive image annotation method based on a multimodal model, the method comprising the following steps:

[0006] S1. Constructing a target detection model, wherein the target detection model is trained by using a deep neural network using multiple sets of multimodal data to minimize a total loss function, wherein each set of the multiple sets of multimodal data includes images and texts;

[0007] S2, obtaining an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or descriptive prompt words; wherein the positive prompt words include the category of the target to be detected, the negative prompt words are empty or include a positive category-negative category pair, and the descriptive prompt words include features of the positive category and the negative category, and a description of the location of the target to be detected in the image;

[0008] S3, segmenting the prompt word text, and obtaining several word units as different target categories to be detected. For each target category to be detected, extracting the corresponding negative category in the negative prompt word, and generating a word unit index table;

[0009] S4, setting a confidence threshold, inputting the image to be detected, the prompt word text and the word element index table into the target detection model, the target detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected whose confidence corresponding to the category to be detected is greater than the confidence threshold as an intermediate detection;

[0010] S5. Set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all detection frames of the negative category from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

[0011] Optionally, in S1, using multiple sets of multimodal data to train the target detection model through a deep neural network with the goal of minimizing the total loss function includes:

[0012] Input images and text;

[0013] Use the image backbone network to extract multi-scale original image features from images, and use the text backbone network to extract original text features from texts;

[0014] The multi-scale original image features and the original text features are cross-modally fused, the image features most relevant to the input text are selected from the image features after feature fusion, and cross-modal decoding is performed in combination with the text features after feature fusion to obtain the predicted box and predicted category of the input image;

[0015] Construct the total loss function of the target detection model, obtain the optimal training parameters by minimizing the total loss function value, and obtain the target detection model.

[0016] Optionally, the image features most relevant to the input text are selected from the image features after feature fusion, including:

[0017] The query process is expressed in the image features after feature fusion as follows:

[0018]

[0019] Among them, I is the image feature, the size is b×i×c, T is the text feature, the size is b×t×c, b represents the batch size, t is the current text feature length, c represents the number of channels, i represents the i-th image feature, logits bit Represents the value of the bth batch of target detection model output logits, the i-th image feature and the t-th text feature position, I bic and T btcRespectively represent the values ​​of the original image features and the original text features at the corresponding positions of the features after feature fusion;

[0020] Calculate the maximum value of each sample output logits:

[0021]

[0022] Among them, logits bj Represents the value of the jth column of the bth sample of logits, and M is a two-dimensional tensor storing the maximum logits of each sample;

[0023] Get the most relevant query q for the input text:

[0024]

[0025] in, Indicates finding the set i1,i2,...,i k , so that i1,i2,...,i k is the image feature that is most relevant to the input text.

[0026] Optionally, the total loss function of the object detection model is composed of the bounding box regression loss and the classification loss, which is expressed as follows:

[0027]

[0028] Among them, α and β are the border regression loss weight and category loss weight respectively, W d 、b d , θ are the training parameters of the target detection model, L reg is the bounding box regression loss, L cls is the classification loss.

[0029] Optionally, the bounding box regression loss includes L1 loss and GIOU loss considering bounding box overlap, and the bounding box regression loss is expressed as follows:

[0030] L reg =L1+L GIOU

[0031] Among them, L1 is L1 loss, L GIOU For GIOU loss.

[0032] Optionally, the L1 loss is expressed as follows:

[0033]

[0034] in,

[0035] gt ij represents the jth coordinate of the prediction box of the i-th sample,

[0036] det ij represents the jth coordinate of the true box of the i-th sample,

[0037] N is the number of samples.

[0038] Optionally, the GIOU loss is expressed as follows:

[0039]

[0040] Among them, gt and det represent the predicted box and the true box respectively.

[0041] C represents the smallest box that can enclose gt and det.

[0042] area(C) represents the area of ​​C.

[0043] Optionally, the logits of each text token are predicted by calculating the dot product of each query and the text feature, and the classification loss is calculated based on each logits. The classification loss is calculated as follows:

[0044]

[0045] in,

[0046] p i is the confidence of the model predicting the correct logits for the i-th sample,

[0047] α i is a balancing factor used to adjust the weights of different categories.

[0048] γ is an adjustment factor used to control the degree of weight attenuation of samples.

[0049] q·T j Indicates the dot product of the query and each text feature, q is the query, T j is the jth text feature, n is the number of text features,

[0050] N is the number of samples.

[0051] Optionally, positive categories and / or negative categories are interactively provided in the description prompt word.

[0052] A second aspect of the present invention provides an image annotation system, comprising:

[0053] A construction module is used to construct a target detection model, wherein the target detection model is trained by using a deep neural network to minimize a total loss function using multiple sets of multimodal data, each set of the multiple sets of multimodal data includes an image and text;

[0054] An acquisition module is used to acquire an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or description prompt words; wherein the positive prompt words include positive categories, which are the categories of the target to be detected; the negative prompt words are empty or include positive category-negative category pairs, which are the category groups described by the positive categories; and the description prompt words include features of the positive categories and the negative categories, and descriptions of the positions of the positive categories and the negative categories in the image;

[0055] The extraction module is used to segment the prompt word text, and the obtained word units are used as different target categories to be detected. For each target category to be detected, the corresponding negative category existing in the negative prompt word is extracted to generate a word unit index table.

[0056] The first detection module is used to set a confidence threshold, input the image to be detected, the prompt word text and the word element index table into the target detection model, the target detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected whose confidence is greater than the confidence threshold as an intermediate detection;

[0057] The second detection module is used to set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all negative category detection frames from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

[0058] Compared with the prior art, the beneficial effects of the present invention include at least:

[0059] The present invention focuses on the problem of automatic generation of image annotations, and comprehensively adopts deep learning, artificial intelligence, and human-computer interaction. It can accurately obtain detection results without collecting specific categories of data for training, and can be used as image annotations for actual use after manual review. It solves the problem that full manual annotation requires a lot of manpower, improves the efficiency of data annotation, and thus reduces the research and development costs of subsequent training models such as target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0061] Figure 1 A schematic diagram of a flow chart of an interactive image annotation method based on a multimodal model provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a processing flow of a target detection model provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of an inter-class non-maximum suppression process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0065] Combination Figure 1 As shown, embodiment 1 of the present invention provides an interactive image annotation method based on a multimodal model, comprising the following steps:

[0066] S1. Construct a target detection model, wherein the target detection model is trained by using a deep neural network using multiple sets of data to minimize a total loss function, and each set of the multiple sets of data includes image data and text data.

[0067] A large-scale public dataset is used to train an open collection object detection model based on a deep neural network Transformer to obtain a model that can detect undefined categories.

[0068] In S1, the target detection model is trained using multiple sets of multimodal data through a deep neural network, including:

[0069] Input images and text;

[0070] Use the image backbone network to extract multi-scale original image features from images, and use the text backbone network to extract original text features from texts;

[0071] The multi-scale original image features and the original text features are cross-modally fused, and the image features most relevant to the input text are queried and selected from the image features after feature fusion. The cross-modal decoding is combined with the text features after feature fusion to obtain the predicted box and predicted category of the input image.

[0072] Construct the total loss function of the target detection model, obtain the optimal training parameters by minimizing the total loss function, and obtain the target detection model.

[0073] Specific, combined Figure 2 As shown in the figure, given an (image, text) pair, a backbone network such as Transformer is used to extract multi-scale original image features, and a text backbone such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture) is used to extract original text features. After that, the image and text features are cross-modal feature fused. Features that are more relevant to the input text are selected as decoder queries, so that the input text is used to guide object detection. After that, the text features and queries are combined for cross-modal decoding, and the decoding output is the predicted bounding box, and the text features are output as the predicted category. The query selection process is as follows:

[0074]

[0075] Among them, I is the image feature, the size is b×i×c, T is the text feature, the size is b×t×c. b represents the batch size, t is the current text feature length, c represents the number of channels, logits bit Represents the value of the b-th batch, the i-th image, and the t-th text feature position of the target detection model output logits, I bic and T btc They respectively represent the values ​​of the original image features and the original text features at the corresponding positions of the features after feature fusion.

[0076] After that, calculate the maximum value of logits for each sample:

[0077]

[0078] Among them, logits bj Represents the value of the jth column of the bth sample of logits. M is a two-dimensional tensor that stores the maximum value of each sample.

[0079] Finally, get the most relevant queries for the input text:

[0080]

[0081] in, Indicates finding the set i1,i2,...,i k , so that

[0082] Optionally, the total loss function of the object detection model is composed of both the bounding box regression loss and the classification loss. The bounding box regression loss is composed of the basic L1 loss and the GIOU (Generalized Intersection over Union) loss that takes into account the overlap of bounding boxes:

[0083] L reg =L1+L GIOU

[0084] Among them, the L1 loss is as follows:

[0085]

[0086] Among them, gt ij represents the jth coordinate of the prediction box of the i-th sample, det ij Represents the jth coordinate of the true box of the i-th sample, and N is the number of samples.

[0087] The GIOU loss is as follows:

[0088]

[0089] Among them, gt and det represent the predicted box and the real box respectively, C represents the smallest box that can enclose gt and det, and area(C) represents the area of ​​C.

[0090] The logits of each text token are predicted by calculating the dot product of each query and the text features, and then the classification loss L is calculated for each logits cls , to predict the target and the classification of the word:

[0091]

[0092] Among them, p i is the confidence of the model predicting the correct logits for the i-th sample, α i is a balancing factor used to adjust the weights of different categories, γ is an adjustment factor used to control the degree of weight attenuation of samples, q·T j Indicates the dot product of the query and each text feature, q is the query, T j is the jth text feature, n is the number of text features, and N is the number of samples.

[0093] The model learns the optimal parameters by minimizing the total loss:

[0094]

[0095] α and β are the bounding box regression loss weight and the category loss weight respectively. Through further training, the training parameter W of the Transformer model is obtained. d 、b d , θ, and then obtain the target detection model.

[0096] S2. Obtain an image to be detected and a prompt word text; the prompt word text includes positive prompt words and / or negative prompt words and / or descriptive prompt words; wherein the positive prompt words contain the category of the target to be detected, and different categories of the target to be detected are separated by punctuation marks, and, or and other conjunctions; the negative prompt words are empty or contain a positive category-negative category pair; the descriptive prompt words contain the features of the positive category and the negative category, and a description of the location of the target to be detected in the image.

[0097] The negative prompt word and the descriptive prompt word are optional. The image to be detected, the positive prompt word, the negative prompt word and the descriptive prompt word are denoted as G, P respectively. pos , P neg and P desc The i-th character in the prompt word is recorded as P[i-1]. For example, for the content of P "acat", pos , P[0] is "a", P[2] is "c", and P[0:3] is "ac".

[0098] S3. Segment the prompt word text, and obtain several word units as different target categories to be detected. For each target category to be detected, extract the corresponding negative category in the negative prompt word. If the negative prompt word is not empty, extract the corresponding negative category in the negative prompt word and generate a word unit index table.

[0099] Specifically, the word element index table can be a dictionary D:

[0100] D={c1:(n 1,1 ,n 1,2 ...),c2:(n 2,1 ,n 2,2 ...),...,c i :(n i,1 ,n i,2 ...)}

[0101] Among them, c i represents the i-th positive category, n i,k Indicates the kth negative category corresponding to the ith positive category. Specifically, the negative prompt words are changed to all negative categories connected by “.”, for example, for P neg= "display screen:window.lamp:button", after the change, it becomes "screen.Button". Connect the positive prompt word, the changed negative prompt word and the description prompt word with ".", and use word segmentation tools such as nltk and spaCy to perform part-of-speech analysis, extract the parts of speech as CC (coordinate conjunctions, such as and, or, etc.) and SYM (symbols, such as "," and ".", etc.), segment the positive and negative prompt words according to the index where these words are located, and generate a word unit index table. For example, for P pos = "a cat and a dog", the word element index table is [[0:5], [10:15]], corresponding to "acat" and "adog".

[0102] S4. Set a confidence threshold, input the image to be detected, the prompt word text and the word element index table into the target detection model. The target detection model obtains the features of the positive category and the negative category of the image to be detected by describing the prompt word, and marks the part of the image to be detected whose confidence corresponding to the category to be detected is greater than the confidence threshold as an intermediate detection.

[0103] Different description words may produce different detection results. When the category to be detected is a rare category, the model may not be able to obtain ideal detection results without sufficient description of the category. In this case, you can detect multiple images and multiple targets, and modify the description words interactively multiple times to obtain the description words that best suit the category.

[0104] S5. Set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection boxes of the target category to be detected and the corresponding negative category. In the output result of non-maximum suppression, after removing all negative category detection boxes, the remaining annotated detection boxes are used as the final image annotation result.

[0105] In this embodiment, by setting the IoU threshold of non-maximum suppression, redundant candidate boxes are removed, the detection results are streamlined, and each target is ensured to be detected only once, avoiding multiple overlapping candidate boxes representing the same target, thereby improving the accuracy and efficiency of image annotation.

[0106] Combination Figure 3 As shown, for each key-value pair c in the dictionary D in S3 i :(n i,1 ,n i,2 ...), c i 、n i,1 、n i,2...The corresponding detection of the categories is performed together with the following inter-class NMS (Non-Maximum Suppression) process:

[0107] Input: bboxes: A 2D array containing the bounding box coordinates (x1, y1, x2, y2). scores: A 1D array containing the scores for each bounding box. threshold: A floating point number specifying the IoU threshold for NMS, defaulting to 0.5.

[0108] Output: A list containing the bounding box indices that were retained.

[0109] The following is a simple description of the algorithm:

[0110] First, sort the bounding boxes according to scores to get a list of indices sorted in descending order by score. Then initialize an empty list keep to store the bounding box indices to be kept. Then when the sorted index list is not empty, repeat the following operations: Take the bounding box index with the current highest score and add it to the keep list. Calculate the intersection over union (IoU) of this bounding box with all other bounding boxes in the sorted index list. Find all bounding box indices whose IoU is less than a given threshold. Update the sorted index list and delete the bounding box indices suppressed by NMS. Finally, return the keep list.

[0111] Embodiment 2 of the present invention provides an image annotation system, which runs the interactive image annotation method based on the multimodal model as described in Embodiment 1, and the system includes:

[0112] A construction module is used to construct a target detection model, wherein the target detection model is trained by using a deep neural network to minimize a total loss function using multiple sets of multimodal data, each set of the multiple sets of multimodal data includes an image and text;

[0113] An acquisition module is used to acquire an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or description prompt words; wherein the positive prompt words include positive categories, which are the categories of the target to be detected; the negative prompt words are empty or include positive category-negative category pairs, which are the category groups described by the positive categories; and the description prompt words include features of the positive categories and the negative categories, and descriptions of the positions of the positive categories and the negative categories in the image;

[0114] The extraction module is used to segment the prompt word text, and the obtained word units are used as different target categories to be detected. For each target category to be detected, the corresponding negative category existing in the negative prompt word is extracted to generate a word unit index table.

[0115] The first detection module is used to set a confidence threshold, input the image to be detected, the prompt word text and the word element index table into the target detection model, the target detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected whose confidence is greater than the confidence threshold as an intermediate detection;

[0116] The second detection module is used to set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all negative category detection frames from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

[0117] Compared with the prior art, the beneficial effects of the present invention include at least:

[0118] The present invention focuses on the problem of automatic generation of image annotations, and comprehensively adopts deep learning, artificial intelligence, and human-computer interaction. It can accurately obtain detection results without collecting specific categories of data for training, and can be used as image annotations for actual use after manual review. It solves the problem that full manual annotation requires a lot of manpower, improves the efficiency of data annotation, and thus reduces the research and development costs of subsequent training models such as target detection.

[0119] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0120] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0121] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0122] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0123] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An interactive image annotation method based on a multimodal model, characterized in that: The method comprises the following steps: S1. Constructing a target detection model, wherein the target detection model is trained by using a deep neural network using multiple sets of multimodal data to minimize a total loss function, wherein each set of the multiple sets of multimodal data includes images and texts; S2, obtaining an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or descriptive prompt words; wherein the positive prompt words include the category of the target to be detected, the negative prompt words are empty or include a positive category-negative category pair, and the descriptive prompt words include features of the positive category and the negative category, and a description of the location of the target to be detected in the image; S3, segmenting the prompt word text, and obtaining several word units as different target categories to be detected. For each target category to be detected, extracting the corresponding negative category in the negative prompt word, and generating a word unit index table; S4, setting a confidence threshold, inputting the image to be detected, the prompt word text and the word element index table into the target detection model, the target detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected whose confidence corresponding to the category to be detected is greater than the confidence threshold as an intermediate detection; S5. Set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all detection frames of the negative category from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

2. The interactive image annotation method based on a multimodal model according to claim 1, characterized in that: In S1, multiple sets of multimodal data are used to train the target detection model through a deep neural network with the goal of minimizing the total loss function, including: Input images and text; Use the image backbone network to extract multi-scale original image features from images, and use the text backbone network to extract original text features from texts; The multi-scale original image features and the original text features are cross-modally fused, the image features most relevant to the input text are selected from the image features after feature fusion, and cross-modal decoding is performed in combination with the text features after feature fusion to obtain the predicted box and predicted category of the input image; Construct the total loss function of the target detection model, obtain the optimal training parameters by minimizing the total loss function value, and obtain the target detection model.

3. The interactive image annotation method based on a multimodal model according to claim 2, characterized in that: Select the image features most relevant to the input text from the image features after feature fusion, including: The query process is expressed in the image features after feature fusion as follows: Among them, I is the image feature, the size is b×i×c, T is the text feature, the size is b×t×c, b represents the batch size, t is the current text feature length, c represents the number of channels, i represents the i-th image feature, logits bit Represents the value of the bth batch of target detection model output logits, the i-th image feature and the t-th text feature position, I bic and T btc Respectively represent the values ​​of the original image features and the original text features at the corresponding positions of the features after feature fusion; Calculate the maximum value of each sample output logits: Among them, logits bj Represents the value of the jth column of the bth sample of logits, and M is a two-dimensional tensor storing the maximum logits of each sample; Get the most relevant query q for the input text: in, Indicates finding the set i1,i2,...,i k , so that i1,i2,...,i k is the image feature that is most relevant to the input text.

4. The interactive image annotation method based on a multimodal model according to claim 3, characterized in that: The total loss function of the target detection model consists of the bounding box regression loss and the classification loss, which is expressed as follows: Among them, α and β are the border regression loss weight and category loss weight respectively, W d 、b d , θ are the training parameters of the target detection model, L reg is the bounding box regression loss, L cls is the classification loss.

5. The interactive image annotation method based on a multimodal model according to claim 4, characterized in that: The bounding box regression loss includes L1 loss and GIOU loss considering bounding box overlap. The bounding box regression loss is expressed as follows: L reg =L1+L GIOU Among them, L1 is L1 loss, L GIOU For GIOU loss.

6. The interactive image annotation method based on a multimodal model according to claim 5, characterized in that: The L1 loss is expressed as follows: in, gt ij represents the jth coordinate of the prediction box of the i-th sample, det ij represents the jth coordinate of the true box of the i-th sample, N is the number of samples.

7. The interactive image annotation method based on a multimodal model according to claim 5, characterized in that: The GIOU loss is expressed as follows: Among them, gt and det represent the predicted box and the true box respectively. C represents the smallest box that can enclose gt and det. area(C) represents the area of ​​C.

8. The interactive image annotation method based on a multimodal model according to claim 4, characterized in that: The logits of each text tag are predicted by calculating the dot product of each query and the text feature, and the classification loss is calculated based on each logits. The classification loss is calculated as follows: in, p i is the confidence of the model predicting the correct logits for the i-th sample, α i is the balancing factor used to adjust the weights of different categories. γ is an adjustment factor used to control the degree of weight attenuation of samples. q·T j Indicates the dot product of the query and each text feature, q is the query, T j is the jth text feature, n is the number of text features, N is the number of samples.

9. The interactive image annotation method based on a multimodal model according to claim 1, characterized in that: Interactively provide positive and / or negative categories in description prompts.

10. An image annotation system using the interactive image annotation method based on a multimodal model according to any one of claims 1 to 8, characterized in that: include: A construction module is used to construct a target detection model, wherein the target detection model is trained by using a deep neural network to minimize a total loss function using multiple sets of multimodal data, each set of the multiple sets of multimodal data includes an image and text; An acquisition module is used to acquire an image to be detected and a prompt word text, wherein the prompt word text includes positive prompt words and / or negative prompt words and / or description prompt words; wherein the positive prompt words include positive categories, which are the categories of the target to be detected; the negative prompt words are empty or include positive category-negative category pairs, which are the category groups described by the positive categories; and the description prompt words include features of the positive categories and the negative categories, and descriptions of the positions of the positive categories and the negative categories in the image; The extraction module is used to segment the prompt word text, and the obtained word units are used as different target categories to be detected. For each target category to be detected, the corresponding negative category existing in the negative prompt word is extracted to generate a word unit index table. The first detection module is used to set a confidence threshold, input the image to be detected, the prompt word text and the word element index table into the target detection model, the target detection model obtains the features of the target category to be detected and the negative category by describing the prompt word, and marks the part of the image to be detected whose confidence is greater than the confidence threshold as an intermediate detection; The second detection module is used to set the IoU threshold for inter-class non-maximum suppression, check each target category to be detected in the intermediate detection, and if its category has a corresponding negative category, perform inter-class non-maximum suppression on all detection frames of the target category to be detected and the corresponding negative category, and remove all negative category detection frames from the output result of non-maximum suppression. The remaining annotated detection frames are used as the final image annotation results.

Images