A target detection label optimization method based on image-text multi-modal

Abstract

This invention discloses a target detection label optimization method based on image-text multimodal processing, which includes a training process and an inference process for the optimization model. The training process of the optimization model includes the following steps: using the target detection model to detect targets in the images of the samples, obtaining the region images of each target and the predicted labels; inputting the region images of each target into an image encoder to obtain the image representation vector of each target; comparing the predicted label with the highest confidence for each target with the correct label; if they are the same, setting the predicted label with the second highest confidence as the incorrect label; otherwise, setting the predicted label with the highest confidence as the incorrect label; calculating the classification cross-entropy loss and the target-image-text comparison loss, and summing them to obtain the total training loss, completing the training process through backpropagation. The inference process involves inputting the template detection results into the optimization model to obtain the optimization result. This scheme is applicable to the optimization of target detection results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a target detection label optimization method based on image-text multi-modal. BACKGROUND

[0002] In recent years, target detection models based on pre-trained models have been widely used in various fields, such as video analysis, camera security, content understanding, etc. The current target detection model has basically been accurate in target candidate frame generation, but there is still a lot of room for improvement in the selection of target labels. In practical applications, on the one hand, for multi-label target detection problems, similar target labels are prone to confusion. On the other hand, target detection is prone to false detection, i.e. identifying other irrelevant targets as target labels. However, the general target detection model cannot be trained specifically for similar target labels. In addition, in the case of model deployment, for a false detection problem found, retraining the model will bring huge model iteration cost.

[0003] In view of the above difficulties, how to train the already trained target detection model to optimize similar labels and improve the robustness of the model in different use scenarios is a problem that needs to be solved. SUMMARY

[0004] The present application mainly solves the technical problems of target label confusion and misidentification existing in the prior art, and provides a target detection label optimization method based on image-text multi-modal, which can further filter the labels in the target detection result.

[0005] The present application mainly solves the above technical problems through the following technical scheme: a target detection label optimization method based on image-text multi-modal, including the training process of an optimization model and the inference process using the trained optimization model, the optimization model including a picture encoder, a text encoder and a sigmod function.

[0006] The training process of the optimization model includes the following steps:

[0007] S1, the sample of the training data set is a picture-text pair, i.e. each sample includes a picture and a correct label corresponding to the target in the picture (the correct label has only one); using a target detection model to detect the target in the picture, the result obtained includes the region picture (candidate frame) of each target and the predicted label of each target, and each target has no less than two predicted labels, which are arranged in descending order according to the confidence;

[0008] S2, inputting the region picture of each target into the picture encoder to obtain the image feature vector of each target;

[0009] S3. Compare the predicted label with the highest confidence level for each target with the correct label. If the predicted label with the highest confidence level is the same as the correct label, then set the predicted label with the second highest confidence level as the incorrect label; if the predicted label with the highest confidence level is different from the correct label, then set the predicted label with the highest confidence level as the incorrect label.

[0010] S4. Calculate the classification cross-entropy loss using the image representation vectors of each target, the correct label, and the incorrect label. Calculate the target image-text contrast loss using the image representation vectors of each target and the correct label. After superimposing, obtain the total training loss. Complete the training process through backpropagation.

[0011] The process of optimizing labels using optimization model inference is as follows: The results obtained by the object detection model include the region images of each object and the labels of each object. Each object has no less than two labels. For one object, the region image of this object is processed by an image encoder to obtain an image representation vector. Each label of this object is processed by a text encoder to obtain a text representation vector. The image representation vector of this object is multiplied by each text representation vector. The resulting product is normalized by the softmax function. The label corresponding to the highest value of the result is the final label.

[0012] Preferably, in step S4, the formula for calculating the classification cross-entropy loss is:

[0013]

[0014] In the formula, N is the total number of targets used for training. The targets used for training include target region images and target text labels. The target text labels include correct labels and incorrect labels.

[0015] y m This represents the first confidence label of the m-th target obtained by the object detection model and its match with the correct label. If they match, then y m =1, if y does not match m =0; p m This represents the probability that the label of the m-th target obtained by the object detection model is the correct label.

[0016] p m The specific process is as follows: The image representation vector v′_m obtained from the region image of the m-th target after passing through the image encoder is transformed into a linear mapping layer to map the vector dimension to the total number of labels. For example, if there are K labels, v′_m is mapped to a vector v′_m_K of dimension K. v′_m_K is then normalized using the softmax function to obtain the probability of each label. Based on the correct label of the target, the probability p of correct prediction is obtained. mFor example, if there are K tags, there will be K probabilities. If the probability that the m-th target matches the correct tag is P, then the probability of p corresponding to this target is... m =P.

[0017] Preferably, in step S4, the formula for calculating the target image-text comparison loss is:

[0018]

[0019] In the formula, l_m_pos is the text representation vector obtained by the text encoder after the correct label text of the m-th target is processed. The text encoder can generally be a conventional pre-trained text encoder such as Bert or Roberta; the calculation formula of p(v'_m,l_m_pos) is as follows:

[0020]

[0021] In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of descriptive texts of the other targets in the same batch of training, excluding the m-th target. That is, the text description k does not match the image target m, but it matches other targets in the same training sample set (batch).

[0022] The formula for calculating S is:

[0023]

[0024] In the formula, T represents transpose, and the double vertical lines represent calculating the vector length, i.e., norm². Here, v and l are variables used for illustration and have no practical meaning.

[0025] Preferably, the image encoder is a combination of a publicly disclosed pre-trained image-based neural network model (e.g., ResNet, convnext, vision transformer, etc.) and a linear mapping layer, and the text encoder is a publicly disclosed pre-trained text feature extraction model.

[0026] Preferably, the correct labels for the image-text pairs in the samples are identified by a publicly available pre-trained object detection model (e.g., GLIP) or by manual annotation.

[0027] The substantial effect of this invention is that it can optimize the results of target detection, improve the accuracy of labels, and the model of this solution is an external plug-in mode, which does not require retraining of the original target detection model, resulting in low optimization cost and high flexibility. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a model training process according to the present invention;

[0029] Figure 2 This is a schematic diagram of a model reasoning process according to the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0031] Example: This example describes a target detection label optimization method based on image and text multimodal processing, including the training process of the optimization model and the inference process using the trained optimization model. The optimization model includes an image encoder, a text encoder, and a sigmoid function.

[0032] like Figure 1 As shown, the training process of the optimized model includes the following steps:

[0033] S1. The training dataset consists of image-text pairs, where each sample includes an image and the correct label corresponding to the target in the image (there is only one correct label). The target detection model is used to detect targets in the images in the samples. The results include the region images (candidate boxes) of each target and the predicted labels of each target. Each target has at least two predicted labels, arranged in descending order of confidence.

[0034] S2. Input the region images of each target into the image encoder to obtain the image representation vector of each target;

[0035] S3. Compare the predicted label with the highest confidence level for each target with the correct label. If the predicted label with the highest confidence level is the same as the correct label, then set the predicted label with the second highest confidence level as the incorrect label; if the predicted label with the highest confidence level is different from the correct label, then set the predicted label with the highest confidence level as the incorrect label.

[0036] S4. Calculate the classification cross-entropy loss using the image representation vectors of each target, the correct label, and the incorrect label. Calculate the target image-text contrast loss using the image representation vectors of each target and the correct label. After superimposing, obtain the total training loss. Complete the training process through backpropagation.

[0037] like Figure 2 As shown, the process of optimizing labels using optimization model inference is as follows: The results obtained by the object detection model include the region images of each object and the labels of each object. Each object has no less than two labels. For one object, the region image of this object is processed by an image encoder to obtain an image representation vector. Each label of this object is processed by a text encoder to obtain a text representation vector. The image representation vector of this object is multiplied by each text representation vector. The resulting product is normalized by the softmax function. The label corresponding to the highest value of the result is the final label.

[0038] In step S4, the formula for calculating the classification cross-entropy loss is:

[0039]

[0040] In the formula, N is the total number of targets used for training. The targets used for training include target region images and target text labels. The target text labels include correct labels and incorrect labels.

[0041] y m This represents the first confidence label of the m-th target obtained by the object detection model and its match with the correct label. If they match, then y m =1, if y does not match m p is 0; m This represents the probability that the label of the m-th target obtained by the object detection model is the correct label.

[0042] p m The specific process is as follows: The image representation vector v′_m obtained from the region image of the m-th target after passing through the image encoder is transformed into a linear mapping layer to map the vector dimension to the total number of labels. For example, if there are K labels, v′_m is mapped to a vector v′_m_K of dimension K. v′_m_K is then normalized using the softmax function to obtain the probability of each label. Based on the correct label of the target, the probability p of correct prediction is obtained. m For example, if there are K tags, there will be K probabilities. If the probability that the m-th target matches the correct tag is P, then the probability of p corresponding to this target is... m =P. The linear mapping layer here has the same structure as the linear mapping layer in the image encoder, except that it has different output parameter dimensions. The purpose of the linear mapping here is to change the dimensions.

[0043] In step S4, the formula for calculating the target image-text comparison loss is:

[0044]

[0045] In the formula, l_m_pos is the text representation vector obtained by the text encoder after the correct label text of the m-th target is processed. The text encoder can generally be a conventional pre-trained text encoder such as Bert or Roberta; the calculation formula of p(v'_m,l_m_pos) is as follows:

[0046]

[0047] In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of descriptive texts of the other targets in the same batch of training, excluding the m-th target. That is, the text description k does not match the image target m, but it matches other targets in the same training sample set (batch).

[0048] The formula for calculating S is:

[0049]

[0050] In the formula, T represents transpose, and the double vertical lines represent calculating the vector length, i.e., norm². Here, v and l are variables used for illustration and have no practical meaning.

[0051] The image encoder is a combination of a publicly available pre-trained image-based neural network model (e.g., ResNet, ConvNext, Vision Transformer, etc.) and a linear mapping layer. The text encoder is a publicly available pre-trained text feature extraction model. Here, the linear mapping maps the output features of the neural network model from general features to object detection features.

[0052] The correct labels for the image-text pairs in the samples are obtained by identifying publicly available pre-trained object detection models (such as GLIP) or by manual annotation.

[0053] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0054] Although this paper uses terms such as image encoder, confidence level, and image-text contrast loss frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A target detection label optimization method based on image and text multimodal methods, characterized in that, This includes the training process of the optimized model and the inference process using the trained optimized model. The optimized model includes an image encoder, a text encoder, and a softmax function. The training process of the optimized model includes the following steps: S1. The training dataset consists of image-text pairs, meaning each sample includes an image and the correct label corresponding to the target in the image. The target detection model is used to detect targets in the images in the samples. The results include the region images of each target and the predicted labels of each target. Each target has at least two predicted labels, arranged in descending order of confidence. S2. Input the region images of each target into the image encoder to obtain the image representation vector of each target; S3. Compare the predicted label with the highest confidence level for each target with the correct label. If the predicted label with the highest confidence level is the same as the correct label, then set the predicted label with the second highest confidence level as the incorrect label; if the predicted label with the highest confidence level is different from the correct label, then set the predicted label with the highest confidence level as the incorrect label. S4. Calculate the classification cross-entropy loss using the image representation vectors of each target, the correct label, and the incorrect label. Calculate the target image-text contrast loss using the image representation vectors of each target and the correct label. After superimposing, obtain the total training loss. Complete the training process through backpropagation. The process of optimizing labels using optimization model inference is as follows: The results obtained by the target detection model include the region images of each target and the labels of each target. Each target has no less than two labels. For one target, the region image of the target is processed by an image encoder to obtain an image representation vector. Each label of the target is processed by a text encoder to obtain a text representation vector. The image representation vector of the target is multiplied by each text representation vector. The resulting product is normalized by the softmax function. The label corresponding to the highest value of the result is the final label. In step S4, the formula for calculating the target image-text comparison loss is: In the formula, l_m_pos is the text representation vector obtained by the text encoder after the correct label text of the m-th target is processed. The calculation formula of p(v'_m,l_m_pos) is as follows: In the formula, S is the similarity between the two objects in parentheses, τ is the temperature hyperparameter, and Nri is the set of descriptive texts of the other targets in the same batch of training, excluding the m-th target. That is, the text description k does not match the image target m, but it matches other targets in the same training sample set. The formula for calculating S is: In the formula, T represents transpose, and the double vertical lines represent the length of the vector, i.e., norm2.

2. The target detection label optimization method based on image and text multimodal as described in claim 1, characterized in that, In step S4, the formula for calculating the classification cross-entropy loss is: In the formula, N is the total number of targets used for training. The targets used for training include target region images and target text labels. The target text labels include correct labels and incorrect labels. y m This represents the first confidence label of the m-th target obtained by the object detection model and its match with the correct label. If they match, then y m =1, if y does not match m p is 0; m This represents the probability that the label of the i-th target obtained by the object detection model is the correct label.

3. The target detection label optimization method based on image and text multimodal as described in claim 1, characterized in that, The image encoder is a combination of a publicly disclosed pre-trained image-based neural network model and a linear mapping layer, and the text encoder is a publicly disclosed pre-trained text feature extraction model.

4. The target detection label optimization method based on image and text multimodal as described in claim 1, characterized in that, The correct labels for the image-text pairs in the samples are obtained by either a publicly available pre-trained object detection model or by manual annotation.

Images