Multi-modal open vocabulary target detection model training method and device

Abstract

The invention discloses a training method and device for a multi-modal open vocabulary target detection model, and relates to the technical field of computer vision, and the method comprises the steps: obtaining training data which comprises a training image and a corresponding real label; determining a pseudo label of a to-be-determined target in the training image based on a pre-trained image-text matching model and a category-independent region suggestion network; and training a detector by combining a confusion prototype comparative learning algorithm based on the pseudo tag and the real tag to obtain a target detection model. According to the scheme, the pre-trained region suggestion network and the image-text multi-mode model are utilized to generate the false label of the potential unknown target, then the negative influence of noise of the false label on training is relieved through confusion prototype comparative learning, and simple and efficient training of the open vocabulary target detection model is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a training method and device for a multimodal open vocabulary target detection model. Background Art

[0002] Open vocabulary object detection technology has played an increasingly important role in robotics, autonomous driving and other fields. For example, in autonomous driving scenarios, vehicles need to be able to recognize a variety of unknown traffic signs and obstacles, not just the objects defined in the training set. This ability is essential to improving the safety and adaptability of the system. With the rapid development of artificial intelligence technology, open vocabulary object detection has become an important research direction in the field of computer vision.

[0003] In the field of open vocabulary detection, the main solution is to use pre-trained multimodal models to align images and texts, combined with other learning techniques to mine unknown targets. However, there are still many problems in the implementation of the above solution, such as heavy reliance on the general recognition ability of pre-trained models and the need to deal with the noise in pseudo-labels. Summary of the invention

[0004] In view of this, this application provides the following technical solutions:

[0005] A method for training a multimodal open vocabulary object detection model, comprising:

[0006] Obtaining training data, wherein the training data includes training images and corresponding true labels;

[0007] Determine a pseudo label of a target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network;

[0008] Based on the pseudo-labels and real labels, the detector is trained in combination with a confusion prototype contrast learning algorithm to obtain a target detection model.

[0009] In a possible implementation, the determining of the pseudo-label of the target to be determined in the training image based on the pre-trained image-text matching model and the category-independent region proposal network includes:

[0010] Based on a category-independent region proposal network, all objects in the training image are located, and the objects are marked with object boxes;

[0011] Performing encoding processing on the training image and the target frame based on an image encoder to obtain encoded target features;

[0012] Inputting text of a category to be detected into a text encoder to obtain text features, wherein the category to be detected includes multiple text categories;

[0013] Similarity matching is performed on the encoded target features and the text features to obtain pseudo labels of all targets in the training image.

[0014] In a possible implementation, performing similarity matching on the encoded target feature and the text feature to obtain pseudo labels of all targets in the training image includes:

[0015] The text category with the highest similarity to the encoded target feature is set as the category of the encoded target feature, and a pseudo label of the target corresponding to the encoded target feature is obtained.

[0016] In a possible implementation, the detector includes a backbone network and a detection head. The detector is trained based on the pseudo-labels and the real labels in combination with a confused prototype contrast learning algorithm to obtain a target detection model, including:

[0017] Using a backbone network to detect and process the training image to obtain target features;

[0018] Based on the target features and the text features, a detection head is used to obtain a prediction result;

[0019] Determining a supervision type loss based on the prediction result;

[0020] Perform unsupervised learning on the target feature based on a confused prototype contrastive learning algorithm to obtain a contrastive learning loss;

[0021] Determining an overall loss based on the supervised type loss and the contrastive learning loss;

[0022] adjusting parameters of the backbone network and the detection head based on the overall loss;

[0023] The above steps are repeated until the backbone network and the detection head reach convergence conditions to obtain a target detector.

[0024] In a possible implementation, determining the supervision type loss based on the prediction result includes:

[0025] A first supervised loss is determined based on the prediction result and the pseudo label, and a second supervised loss is determined based on the prediction result and the true label, wherein the first supervised loss and the second supervised loss constitute a supervised type loss.

[0026] In a possible implementation, the detection head includes a regression layer and a classification head, and the prediction result is obtained by using the detection head based on the target feature and the text feature, including:

[0027] The target feature is processed by the regression layer to obtain a bounding box prediction result;

[0028] The target frame is processed using the classification layer of the classification head to obtain a processed target feature with the same dimension as the text feature dimension;

[0029] Using the classification head to perform similarity matching between the processing target feature and the text feature to obtain a classification prediction result;

[0030] The bounding box prediction result and the classification prediction result constitute a prediction result.

[0031] In a possible implementation, the unsupervised learning of the target feature based on the confused prototype contrastive learning algorithm to obtain the contrastive learning loss includes:

[0032] Initialize the feature pool and prototype point, each feature pool corresponds to a prototype point;

[0033] In each training process, the corresponding target features are saved into the corresponding feature pool according to the real labels and pseudo labels;

[0034] After each set batch of training is completed, the prototype points of the feature pool are updated;

[0035] During the training process, for each target feature, determine the N categories with the highest confidence in its prediction results, where N is a positive integer greater than 1. The N categories include the true category corresponding to the current target feature, and the remaining N-1 categories are the confusion categories of the current target feature;

[0036] The contrastive learning loss is determined based on feature prototype calculations corresponding to the true category and the confused category.

[0037] In a possible implementation, the updating of the prototype point of the feature pool includes:

[0038] Cluster all target features in the feature pool of each category to obtain the cluster center;

[0039] The cluster center is determined as the latest prototype point of the corresponding feature pool.

[0040] In one possible implementation, the pre-trained image-text matching model and the category-independent region proposal network are frozen models.

[0041] The present application also provides a training device for a multimodal open vocabulary object detection model, comprising:

[0042] A data acquisition module, used to obtain training data, wherein the training data includes training images and corresponding true labels;

[0043] A pseudo-label determination module, used to determine the pseudo-label of the target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network;

[0044] The model training module is used to train the detector based on the pseudo-label and the real label in combination with the confusion prototype contrast learning algorithm to obtain the target detection model.

[0045] Through the above technical scheme, it can be known that the embodiment of the present application discloses a training method and device for a multimodal open vocabulary target detection model, the method comprising: obtaining training data, the training data comprising training images and corresponding real labels; determining the pseudo-labels of the target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network; based on the pseudo-labels and real labels, the detector is trained in combination with a confused prototype contrast learning algorithm to obtain a target detection model. The above scheme uses a pre-trained region proposal network and an image-text multimodal model to generate pseudo-labels for potential unknown targets, and then uses confused prototype contrast learning to alleviate the negative impact of pseudo-label noise on training, thereby achieving simple and efficient training of an open vocabulary target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of the implementation principle architecture of the training scheme for the multimodal open vocabulary object detection model disclosed in the embodiment of the present application;

[0048] Figure 2 A flowchart of a method for training a multimodal open vocabulary object detection model disclosed in an embodiment of the present application;

[0049] Figure 3 A flowchart of determining a pseudo label of a target to be determined disclosed in an embodiment of the present application;

[0050] Figure 4 A flowchart of obtaining a target detection model disclosed in an embodiment of the present application;

[0051] Figure 5 This is a schematic diagram of the structure of the detection head disclosed in the embodiment of the present application;

[0052] Figure 6A schematic diagram of the implementation principle architecture of the target detector disclosed in the embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of a training device for a multimodal open vocabulary object detection model disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] For the purpose of reference and clarity, the descriptions, abbreviations or acronyms of the technical terms used below are summarized as follows:

[0055] Multimodality: Multimodality technology enhances the AI ​​system’s ability to understand and process complex information by fusing data and information from different senses, thereby improving performance and application scope.

[0056] Pre-trained model: A pre-trained model is a model that has been trained on a large-scale dataset. It has strong generalization capabilities and can be applied to a variety of downstream tasks.

[0057] Pseudo-label: Pseudo-label is different from real label. In semi-supervised and unsupervised learning, the prediction result of pre-trained model on unlabeled data is called pseudo-label. Training with pseudo-label and real label can improve the generalization ability and accuracy of the model. However, pseudo-label cannot be as accurate as real label and usually contains a certain degree of noise.

[0058] Object detection: Object detection studies the location and classification of objects of interest in images. It is divided into two-stage algorithms and single-stage algorithms based on the detection steps. The two-stage algorithm has high accuracy, but the reasoning speed is slightly slower; the single-stage algorithm has slightly lower accuracy, but the reasoning speed is fast. The single-stage algorithm is widely used in security, construction sites, transportation and other fields.

[0059] Open-Vocabulary Object Detection: Open-Vocabulary Object Detection is a computer vision technique that aims to identify objects that have not been seen in the training set. It uses features of known objects and unlabeled data to detect and classify new objects through common representation and matching techniques. This method improves the model's adaptability in dynamic and changing environments.

[0060] Contrastive Learning: Contrastive Learning is an unsupervised learning method that learns feature representation by comparing similar samples. It uses positive and negative sample comparison to improve the model's ability to distinguish similar and different samples. It is often used in image and text processing and can effectively improve the performance of the model on unlabeled data.

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] Figure 1 A schematic diagram of the implementation principle architecture of the training scheme for the multimodal open vocabulary object detection model disclosed in the embodiment of the present application; Figure 2 The flowchart of a method for training a multimodal open vocabulary object detection model disclosed in an embodiment of the present application is shown in FIG. Figure 1 and Figure 2 As shown, the training method of the multimodal open vocabulary object detection model may include:

[0063] Step 201: Obtain training data, where the training data includes training images and corresponding true labels.

[0064] The training data includes a plurality of training images with real labels. Figure 1 In order to facilitate understanding, the training implementation architecture of the multimodal open vocabulary target detection model is simplified, and the relevant content of the real label is not shown, but the real label is also involved in the training implementation of the entire multimodal open vocabulary target detection model, as described in detail below. In this embodiment, the training data can be used to generate pseudo labels and train the detection model, and the trained detection model is the target detection model.

[0065] Step 202: Determine a pseudo label of a target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network.

[0066] The pre-trained image-text matching model may be a CLIP (Contrastive Language-Image Pre-Training, a multimodal pre-trained neural network) model, which includes a CLIP image encoder and a CLIP text encoder. The CLIP model can perform targeted mining, detection and recognition of potential unknown targets in the training data with the category-independent region proposal network RPN to obtain pseudo labels for unknown targets.

[0067] The target to be determined may be a target of a known category in the training image, or a target of an unknown category, and the present application does not impose any fixed restrictions on this.

[0068] The specific implementation of generating pseudo labels will be described in detail in the following embodiments and will not be described in detail here.

[0069] Step 203: Based on the pseudo-labels and real labels, the detector is trained in combination with a confusion prototype contrast learning algorithm to obtain a target detection model.

[0070] After obtaining the pseudo-label, the detector can be trained based on the real label and the pseudo-label together; at the same time, considering that there must be noise in the pseudo-label, in order to reduce the negative impact of the noise of the pseudo-label on the detector training, in this application, the detector is trained in combination with the confusion prototype contrast learning algorithm during the training process.

[0071] The training scheme of the multimodal open vocabulary object detection model described in this embodiment uses a pre-trained region proposal network and an image-text multimodal model to generate pseudo labels for potential unknown targets, and then uses confused prototype contrast learning to alleviate the negative impact of pseudo label noise on training, thereby achieving simple and efficient training of the open vocabulary object detection model.

[0072] Figure 3 This is a flowchart of determining a pseudo label of a target to be determined disclosed in an embodiment of the present application. Figure 3 As shown, the determining of the pseudo label of the target to be determined in the training image based on the pre-trained image-text matching model and the category-independent region proposal network may include:

[0073] Step 301: all objects in the training image are located based on a category-independent region proposal network, and the objects are marked with object boxes.

[0074] Pseudo-labeling is used to mine potential but unlabeled targets in the training data. Two models are mainly used in the process: the category-independent region proposal network (RPN) and the pre-trained image text matching model (CLIP). CLIP consists of the CLIP image encoder and the text encoder. Among them, the parameters of the category-independent RPN, CLIP image encoder, and CLIP text encoder are frozen and do not participate in training, which can greatly reduce the resource consumption of the entire training process.

[0075] In an embodiment of the present application, a frozen category-independent RPN can first be used to locate all possible targets in the training image, and the obtained target boxes are sent to the frozen CLIP image encoder. Specifically, in order to mine potential, unlabeled targets in the data set, a category-independent RPN is used to perform region proposals. The RPN is a pre-trained model, and in order to avoid data leakage, its training data does not contain unknown category data for open vocabulary detection. RPN can locate areas in the image where targets may exist and return a set of target boxes. Specifically, the input image is represented as ,in , H and W represent the number of channels, image height and image width of the image respectively. The transformation from image to target box in RPN is recorded as , then the generation of N target boxes can be expressed as follows:

[0076] (1)

[0077] in , represents the coordinates of the center point of the i-th target box, and Represent the width and height of the i-th target box respectively.

[0078] Step 302: Encode the training image and the target frame based on an image encoder to obtain encoded target features.

[0079] The CLIP image encoder encodes the target box and the image to obtain the target features encoded by the CLIP. Specifically, the frozen CLIP image encoder has two inputs, namely the training image I and the target box Boxes provided by the RPN. Since the weights of the CLIP image encoder are frozen, only the feature transformation of its input and output is considered in this application. The feature transformation of the CLIP image encoder is denoted as , then the characteristics of the target box Boxes in the training image I It can be expressed as formula (2).

[0080] (2)

[0081] in is the encoded target feature of the i-th target box after being encoded by CLIP. Each encoded target feature is a one-dimensional vector with a length of 1024. 1024 is the default feature vector length of CLIP.

[0082] Step 303: inputting text of a category to be detected into a text encoder to obtain text features, wherein the category to be detected includes multiple text categories.

[0083] The text of the category to be detected is input into the frozen CLIP text encoder to obtain the CLIP-encoded text features. Specifically, in order to match the image features, the category of the target to be detected is embedded in the template and input into the text encoder. The text template is "a photo of {class} in scene", and the final output text features are , where C is the number of categories of all targets to be detected. Similar to the target features in S2, each text feature is also a one-dimensional vector with a length of 1024.

[0084] Step 304: performing similarity matching on the encoded target features and the text features to obtain pseudo labels of all targets in the training image.

[0085] Calculate the similarity between the target feature and the text feature, and perform similarity matching to obtain the pseudo label of the potential target. Specifically, the cosine similarity between the target feature encoded by CLIP and the text feature of each category is calculated. The category with the largest similarity is used as the category pseudo label of the current target, and the similarity is used as the confidence of the label. Cosine similarity between the i-th target feature and the j-th text feature The calculation of can be expressed as equation (3).

[0086] (3)

[0087] If the confidence is lower than the threshold thr, the pseudo label will be discarded. The confidence threshold thr is a decimal between 0 and 1.

[0088] Based on the above, step 304 may include: setting the text category with the highest similarity to the encoded target feature as the category of the encoded target feature, and obtaining a pseudo label of the target corresponding to the encoded target feature.

[0089] The above content introduces in detail the specific implementation of generating pseudo labels, which is convenient for technicians in the field to better understand and implement the technical solution of this application.

[0090] The detector in the aforementioned embodiment may include a backbone network and a detection head. Figure 4 This is a flowchart of obtaining a target detection model disclosed in the embodiment of this application. Figure 4 As shown, the detector is trained based on the pseudo-label and the real label in combination with the confusion prototype contrast learning algorithm to obtain the target detection model, which may include:

[0091] Step 401: Use a backbone network to detect and process the training image to obtain target features.

[0092] Step 402: Based on the target features and the text features, a detection head is used to obtain a prediction result.

[0093] Step 403: Determine the supervision type loss based on the prediction result.

[0094] Specifically, a first supervised loss can be determined based on the prediction result and the pseudo label, which can be understood as a weak supervised loss, and a second supervised loss can be determined based on the prediction result and the true label. The first supervised loss and the second supervised loss constitute a supervised type loss. Figure 1 Content, easy to understand, Figure 1The relevant content of determining the second supervision loss is not shown.

[0095] Step 404: performing unsupervised learning on the target feature based on a confused prototype contrastive learning algorithm to obtain a contrastive learning loss.

[0096] Specifically, the feature pool and prototype points can be initialized first, with each feature pool corresponding to a prototype point; then, during each training process, the corresponding target features are saved in the corresponding feature pool according to the true labels and pseudo labels; after each set batch of training is completed, the prototype points of the feature pool are updated; during the training process, for each target feature, the N categories with the highest confidence in its prediction results are determined, where N is a positive integer greater than 1, and the N categories include the true category corresponding to the current target feature, and the remaining N-1 categories are the confusion categories of the current target feature; the contrastive learning loss is determined based on the feature prototype calculation corresponding to the true category and the confusion category. Wherein, the updating of the prototype points of the feature pool can include: clustering all target features in the feature pool of each category to obtain the cluster center; and determining the cluster center as the latest prototype point of the corresponding feature pool.

[0097] Step 405: Determine an overall loss based on the supervised type loss and the contrastive learning loss.

[0098] The sum of the supervised type loss and the contrastive learning loss is the overall loss.

[0099] Step 406: Adjust the parameters of the backbone network and the detection head based on the overall loss.

[0100] The overall loss is back-propagated, and the parameters of the backbone network and the detection head are updated. The process of adjusting the parameters of the backbone network and the detection head based on the overall loss is a feedback control process. The trained target detection model has the ability to detect both known targets and potential unknown targets, that is, the ability of open vocabulary detection.

[0101] Step 407: Determine whether the backbone network and the detection head have reached convergence conditions, if not return to step 401, if yes, obtain the target detector.

[0102] The convergence condition may be a preset convergence condition, which may be set based on a conventional convergence condition setting method, and the present application does not impose any fixed restrictions on this.

[0103] In combination with the above content, this application introduces a conventional detection model (detector) in the implementation of model training (detector training), uses the real labels and pseudo labels of the training data for training, and obtains the supervised type loss; at the same time, the target features generated by the backbone network of the model will be confused prototype contrast learning to obtain contrast learning loss. The backbone network and the detection head are trained based on the supervised type loss and contrast learning loss, and the trained detector is the target detector.

[0104] The detector used in this application is Faster R-CNN. The structure of Faster R-CNN is very simple and classic, and it is the prototype of many new detectors. This application is introduced on Faster R-CNN, and the method of the application can also be based on this and easily extended to other detectors. The main structure and main functions of Faster R-CNN are as follows:

[0105] Backbone network: used to extract features of input images. Usually a pre-trained convolutional neural network (such as VGG16, ResNet, etc.) is used as the basic network to obtain high-level features of the image.

[0106] Region Proposal Network (RPN): RPN directly slides a small window on the feature map to generate a set of region proposals. Each window is mapped to a low-dimensional vector, which is used to generate region proposals and their scores. RPN outputs a series of region proposals with potential targets and their corresponding confidence scores.

[0107] ROI pooling layer: pools the region candidate boxes generated by RPN and represents them in a fixed-size feature map. This step enables the subsequent fully connected layer to process candidate regions of different sizes.

[0108] Detection head: It consists of a classification layer and a regression layer. The classification layer outputs the category probability distribution of each candidate box, and the regression layer outputs the bounding box adjustment parameters of each candidate box.

[0109] In order to adapt to the open vocabulary task and reduce the resource overhead of the entire system, in this application, the RPN of FasterRCNN itself is abandoned, and the candidate target boxes generated by the category-independent RPN in the first part can be directly reused. The classification layer will scale the features of the candidate boxes to 1024 dimensions and perform similarity matching with the text features of each category. The category with the highest similarity is used as the predicted category of the target. The detection head structure used in this application is as follows Figure 5 shown.

[0110] from Figure 5As can be seen in Figure 1, the detection head used in this application consists of a regression layer and a modified classification head. The regression layer is consistent with the original structure of FasterRCNN, and the target features output by the classification layer are matched with the text features encoded by CLIP for similarity to obtain the classification prediction results. The final prediction results include the bounding box prediction results and the classification prediction results.

[0111] Therefore, the prediction result of the detector is recorded as ,in represents the predicted category of the i-th target, represents the predicted bounding box of the i-th target, and K is the total number of predicted targets. The real label and pseudo label are and . Then the weakly supervised learning loss can be expressed in the form of equation (4).

[0112] (4)

[0113] in Represents commonly used classification losses, such as cross entropy loss and focal loss. Represents commonly used bounding box regression losses, such as L1 loss, IOU loss, etc. is the weakly supervised learning loss.

[0114] Since the CLIP model is a model trained on an image-text pair dataset, it is not sensitive enough to target positioning, resulting in noise in the pseudo-labels. Specifically, the pseudo-labels generated by CLIP are misclassified; moreover, for some inaccurately positioned target frames, CLIP may assign them a higher confidence, while some more accurately positioned target frames may have a lower confidence in their pseudo-labels. Such pseudo-label noise will directly limit the ability of the detection model. Therefore, the present application proposes a method for contrastive learning of confused prototypes, which alleviates the negative impact of noise on model training by performing targeted contrastive learning in easily confused category prototypes.

[0115] In this way, by using contrastive learning, the differences between easily confused category features are increased, and the discrimination of easily confused categories is improved. At the same time, due to the possibility of misclassification in pseudo labels, this application does not directly use the target features of each category as positive and negative samples, but uses the cluster feature centers of each category. This can not only reduce the amount of calculation of contrast loss, but also alleviate the negative impact of noise on contrastive learning.

[0116] Based on the above content, combined with Figure 5As shown, the detection head includes a regression layer and a classification head. The prediction result obtained by using the detection head based on the target feature and the text feature may include: using the regression layer to process the target feature to obtain a border prediction result; using the classification layer of the classification head to process the target frame to obtain a processed target feature with the same dimension as the text feature dimension; using the classification head to perform similarity matching between the processed target feature and the text feature to obtain a classification prediction result; the border prediction result and the classification prediction result constitute a prediction result.

[0117] Specifically, the process of obfuscated prototype contrast learning can include:

[0118] Initialize feature pool and prototype point: Create an empty feature pool for each category to store target features; initialize a prototype point for each feature pool;

[0119] Update feature pool: Each time training, the generated target features are saved into the corresponding feature pool according to the real label and pseudo label. In order to avoid excessive memory usage, each feature pool only saves a certain number of the latest features. Features are first-in-first-out in a queue form;

[0120] Update feature prototype: After each training of K batches of data, update the feature prototype. The updating method is: run the K-means algorithm on the feature pool of each category, and the number of cluster centers is 1. When the cluster center no longer changes, set the cluster center as the feature prototype of the feature pool;

[0121] Calculate the confusion prototype contrast loss: During the training process, for each target feature, read its prediction result, obtain the topk categories before its confidence, remove its true category, and get the topk-1 categories. This topk-1 category is the category that is most easily confused by the current target. The feature prototype of the true category is used as the positive sample k+, and the feature prototypes of the remaining topk-1 confused categories are used as negative samples. , the feature of the target is used as the query sample q, and the object learning loss is calculated. Taking Info-NCE loss as an example, the confusion prototype contrast learning loss can be expressed in the form of equation (5).

[0122] (5)

[0123] in It is a similarity calculation function, such as cosine similarity. is the temperature coefficient. In order to confuse the prototype contrastive learning loss, other forms of contrastive learning losses such as Triplet Loss and NT-Xent Loss can also be used to calculate the confused prototype loss.

[0124] Finally, this application superimposes the supervised type loss and contrastive learning loss to train the backbone network and detection head of the detection model. The overall loss can be expressed as equation (6).

[0125] (6)

[0126] in, is the loss weight coefficient, and its value is between 0 and 1.

[0127] In summary, the present application has the following advantages: First, the present application first uses the pre-trained image-text matching model (CLIP) and the class-agnostic region proposal network (Class-Agnostic RPN) to conduct targeted mining, detection and identification of potential unknown targets in the training data to obtain pseudo-labels of unknown targets. Then use the real labels and pseudo-labels to train a task-specific target detector; second, in the process of training the target detector, cluster analysis and contrastive learning are used to alleviate the negative impact of some noise in the pseudo-label; third, confusing prototype contrastive learning is introduced to separate the feature distribution of easily confused categories in a targeted manner and improve the classification accuracy of the model; fourth, the parameters of the CLIP image encoder, CLIP text encoder and class-agnostic RPN used in the present application are all frozen, and no additional auxiliary data or training model needs to be generated. Only a conventional-level target detector (including the backbone network and the detection head) needs to be trained, which minimizes the cost of training resources.

[0128] In addition, it should be noted that the target detection and contrastive learning of the present application scheme involve the setting of some hyperparameters, and the values ​​of these parameters will directly affect the effect of open vocabulary detection.

[0129] In the pseudo-label generation process, the confidence threshold directly affects the quality of the pseudo-label. If the confidence is too high, the number of pseudo-labels will be too small and there will be more missed detections. If the confidence is too low, there will be too many false detections. Therefore, in the implementation, the average number of pseudo-labels for each image can be counted periodically. If the average number of pseudo-labels is lower than 0.5, the threshold is lowered. If the average number of pseudo-labels is higher than 1, the threshold is increased. The amplitude of each adjustment is 0.25, and the initial confidence threshold is 0.7.

[0130] In contrastive learning, the frequency K of running K-means can be set to 200 iterations, 8 images each time. If the number of features in a feature pool is less than 40, wait for the next clustering. If the frequency is too high, the system efficiency will be reduced; if the frequency is too low, the prototype obtained is not accurate enough; the maximum capacity of the feature pool is 1000, and 1000 features are enough to calculate a representative feature prototype. The topk categories that are easily confused are set to 6. Most categories have only 2 to 3 easily confused categories. 6 is a more conservative level. At the same time, categories with extremely low confidence will be filtered out to avoid meaningless contrastive learning.

[0131] In the overall loss of target detection, the loss coefficient is the coefficient that balances the loss of weakly supervised learning and contrastive learning. Set to 0.9.

[0132] After training the object detector, it can be put into use for open vocabulary detection. In the application of open vocabulary detection, only the object detector backbone network, the object detector detection head and the CLIP text encoder are required. The text features generated by the CLIP text encoder and the target features output by the backbone network are simultaneously input into the detection head to obtain the final prediction result, such as Figure 6 shown.

[0133] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0134] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.

[0135] Figure 7 This is a schematic diagram of the structure of a training device for a multimodal open vocabulary object detection model disclosed in an embodiment of the present application. Figure 7 As shown, the training device 70 of the multimodal open vocabulary object detection model may include:

[0136] The data acquisition module 701 is used to obtain training data, where the training data includes training images and corresponding true labels.

[0137] The pseudo-label determination module 702 is used to determine the pseudo-label of the target to be determined in the training image based on the pre-trained image-text matching model and the category-independent region proposal network.

[0138] The model training module 703 is used to train the detector based on the pseudo-labels and the real labels in combination with the confusion prototype contrast learning algorithm to obtain the target detection model.

[0139] The training scheme of the multimodal open vocabulary object detection model described in this embodiment uses a pre-trained region proposal network and an image-text multimodal model to generate pseudo labels for potential unknown targets, and then uses confused prototype contrast learning to alleviate the negative impact of pseudo label noise on training, thereby achieving simple and efficient training of the open vocabulary object detection model.

[0140] The specific implementation of the training device of the multimodal open vocabulary object detection model and the modules it contains can be found in the corresponding parts of the method embodiment, which will not be repeated here.

[0141] The training device for any one of the multimodal open vocabulary target detection models described in the above embodiments includes a processor and a memory. The data acquisition module, pseudo-label determination module, model training module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to implement corresponding functions.

[0142] The processor includes a kernel, which retrieves the corresponding program module from the memory. One or more kernels can be set, and the processing of the access data can be realized by adjusting the kernel parameters.

[0143] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0144] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the training method of the multimodal open vocabulary target detection model mentioned above.

[0145] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer, wherein the computer program contains software code, and after being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the training method of the multimodal open vocabulary object detection model described above.

[0146] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0147] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0148] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0149] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a multimodal open vocabulary object detection model, characterized in that: include: Obtaining training data, wherein the training data includes training images and corresponding true labels; Determine a pseudo label of a target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network; Based on the pseudo-label and the real label, the detector is trained in combination with a confusion prototype contrast learning algorithm to obtain a target detection model.

2. The method for training a multimodal open vocabulary object detection model according to claim 1, characterized in that: The method of determining a pseudo label of a target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network includes: Based on a category-independent region proposal network, all objects in the training image are located, and the objects are marked with object boxes; Performing encoding processing on the training image and the target frame based on an image encoder to obtain encoded target features; Inputting text of a category to be detected into a text encoder to obtain text features, wherein the category to be detected includes multiple text categories; Similarity matching is performed on the encoded target features and the text features to obtain pseudo labels of all targets in the training image.

3. The method for training a multimodal open vocabulary object detection model according to claim 2, characterized in that: The performing similarity matching on the encoded target feature and the text feature to obtain pseudo labels of all targets in the training image includes: The text category with the highest similarity to the encoded target feature is set as the category of the encoded target feature, and a pseudo label of the target corresponding to the encoded target feature is obtained.

4. The method for training a multimodal open vocabulary object detection model according to claim 2, characterized in that: The detector includes a backbone network and a detection head. The detector is trained based on the pseudo-label and the real label in combination with a confused prototype contrast learning algorithm to obtain a target detection model, including: Using a backbone network to detect and process the training image to obtain target features; Based on the target features and the text features, a detection head is used to obtain a prediction result; Determining a supervision type loss based on the prediction result; Perform unsupervised learning on the target feature based on a confused prototype contrastive learning algorithm to obtain a contrastive learning loss; Determining an overall loss based on the supervised type loss and the contrastive learning loss; adjusting parameters of the backbone network and the detection head based on the overall loss; The above steps are repeated until the backbone network and the detection head reach convergence conditions to obtain a target detector.

5. The method for training a multimodal open vocabulary object detection model according to claim 4, characterized in that: The determining of the supervision type loss based on the prediction result comprises: A first supervised loss is determined based on the prediction result and the pseudo label, and a second supervised loss is determined based on the prediction result and the true label, wherein the first supervised loss and the second supervised loss constitute a supervised type loss.

6. The method for training a multimodal open vocabulary object detection model according to claim 4, characterized in that: The detection head includes a regression layer and a classification head. The prediction result is obtained by using the detection head based on the target feature and the text feature, including: The target feature is processed by the regression layer to obtain a bounding box prediction result; The target frame is processed using the classification layer of the classification head to obtain a processed target feature with the same dimension as the text feature dimension; Using the classification head to perform similarity matching between the processing target feature and the text feature to obtain a classification prediction result; The bounding box prediction result and the classification prediction result constitute a prediction result.

7. The method for training a multimodal open vocabulary object detection model according to claim 4, characterized in that: The unsupervised learning of the target feature based on the confused prototype contrastive learning algorithm to obtain the contrastive learning loss includes: Initialize the feature pool and prototype point, each feature pool corresponds to a prototype point; In each training process, the corresponding target features are saved into the corresponding feature pool according to the real labels and pseudo labels; After each set batch of training is completed, the prototype points of the feature pool are updated; During the training process, for each target feature, determine the N categories with the highest confidence in its prediction results, where N is a positive integer greater than 1. The N categories include the true category corresponding to the current target feature, and the remaining N-1 categories are the confusion categories of the current target feature; The contrastive learning loss is determined based on feature prototype calculations corresponding to the true category and the confused category.

8. The method for training a multimodal open vocabulary object detection model according to claim 7, characterized in that: The prototype point of the updating feature pool includes: Cluster all target features in the feature pool of each category to obtain the cluster center; The cluster center is determined as the latest prototype point of the corresponding feature pool.

9. The method for training a multimodal open vocabulary object detection model according to claim 1, characterized in that: The pre-trained image-text matching model and the category-independent region proposal network are frozen models.

10. A training device for a multimodal open vocabulary object detection model, characterized in that: include: A data acquisition module, used to obtain training data, wherein the training data includes training images and corresponding true labels; A pseudo-label determination module, used to determine the pseudo-label of the target to be determined in the training image based on a pre-trained image-text matching model and a category-independent region proposal network; The model training module is used to train the detector based on the pseudo-label and the real label in combination with the confusion prototype contrast learning algorithm to obtain the target detection model.

Images