A target detection model training method, a detection method, a device and a medium

Abstract

The application relates to a training method and a detection method of a target detection model, equipment and a medium, and the method comprises the following steps: obtaining a test set, inputting the test set into a first model to obtain a potential target sample picture of each test image, obtaining a positive sample set and a negative sample set according to the potential target sample picture of each test image, training a second model by using the positive sample set and the negative sample set, inputting the positive sample set into the trained second model to obtain a positive sample feature library, inputting the negative sample set into the trained second model to obtain a negative sample feature library, inputting the potential target sample picture into the trained second model to obtain a feature of the potential target sample picture, inputting the feature of the potential target sample picture into a third model to obtain a predicted category of the potential target sample picture, adjusting parameters of the third model according to the predicted category and an actual category of the potential target sample picture to obtain a trained third model, and the recall rate and the precision of the few-sample target detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of visual inspection, and more particularly to a training method, detection method, device, and medium for an object detection model. Background Technology

[0002] Deep learning detectors have achieved remarkable success in detecting common traffic targets such as cars, pedestrians, and cyclists. However, many detectors often fail to detect novel targets that were not seen or rarely seen during training (such as irregular roadblocks and dogs crossing the street), which are called few-shot targets. Since there are a large number of similar objects in the detection of few-shot targets, improving detection accuracy is a key challenge in few-shot target detection.

[0003] Current methods for detecting targets with few samples rely on data augmentation to supplement the sample data and train conventional neural networks, such as the YOLO series, to increase the generalization of the detection model. Although this can solve the problem of identifying targets with few samples to some extent, this method suffers from low recall and low detection accuracy. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a training method, detection method, device and medium for a target detection model, which can improve the recall and accuracy of target detection with few samples.

[0005] On one hand, the present invention provides a method for training an object detection model, the object detection model including a first model, a second model, and a third model, the method comprising the following steps:

[0006] Obtain a test set, input the test set into the first model to obtain the potential target sample image for each test image, and obtain a positive sample set and a negative sample set based on the potential target sample image for each test image; the test set includes N test images;

[0007] Data augmentation is performed on the positive sample set and the negative sample set to obtain a multimodal training sample set, and the second model is trained using the multimodal training sample set;

[0008] The second model is trained using the positive and negative sample sets;

[0009] The potential target sample image is input into the trained second model to obtain the features of the potential target sample image;

[0010] The features of the potential target sample image are input into the third model to obtain the predicted category of the potential target sample image. The parameters of the third model are adjusted according to the predicted category and the true category of the potential target sample image to obtain the trained third model.

[0011] Optionally, the step of inputting the test set into the first model to obtain a potential target sample image for each test image, and obtaining a positive sample set and a negative sample set based on the potential target sample image for each test image, specifically includes:

[0012] The N test images are input into the first model to obtain several potential target sample boxes for each test image. Based on the several potential target sample boxes for each test image, the potential target sample image for each test image is obtained.

[0013] Calculate the first overlap rate between the potential target sample image and the labeled target sample box for each test image, and obtain the positive sample set and the negative sample set based on the first overlap rate; the positive sample set includes target sample images, and the negative sample set includes non-target sample images.

[0014] Optionally, obtaining the potential target sample image for each test image based on several potential target sample boxes for each test image specifically includes:

[0015] Calculate the second overlap rate of several potential target bounding boxes for each test image;

[0016] The potential target sample boxes with the second overlap rate greater than the preset value are integrated to obtain potential target sample images for each test image; wherein the potential target sample boxes that are integrated belong to the same test image.

[0017] Optionally, training the second model using the positive and negative sample sets specifically includes:

[0018] Image fusion is performed on potential target sample images in the positive sample set, and text fusion is performed on target sample text in the positive sample set.

[0019] Based on the fused potential target sample image and the fused target sample text, sample image-text pairs are obtained;

[0020] The sample image text pairs are combined with the negative sample set to obtain a multimodal training sample set;

[0021] The second model is trained using the multimodal training sample set.

[0022] Optionally, the step of inputting the features of the potential target sample image into the third model to obtain the predicted category of the potential target sample image, and adjusting the parameters of the third model according to the predicted category and the true category of the potential target sample image to obtain the trained third model specifically includes:

[0023] The features of the potential target sample image are input into the third model, which calculates the similarity between the features of the potential target sample image and the first sample in the positive sample feature library, and calculates the similarity between the features of the potential target sample image and the second sample in the negative sample feature library.

[0024] The predicted category of the potential target sample image is obtained based on the similarity between the first sample and the second sample. The parameters of the third model are adjusted based on the predicted category and the true category of the potential target sample image. The process of inputting the features of the potential target sample image into the third model is repeated until the predicted category meets the preset conditions, thus obtaining the trained third model.

[0025] On the other hand, the present invention provides a target detection method, comprising the following steps:

[0026] The image to be detected is acquired and input into the first model to obtain the potential target image;

[0027] The potential target image is input into the trained second model to obtain the image features of the potential target image; the second model is trained using a target detection model training method.

[0028] The image features are input into a trained third model to obtain the detection category of the image to be detected; the third model is trained using a target detection model training method.

[0029] Optionally, the step of inputting the image features into the trained third model to obtain the detection category of the image to be detected specifically includes:

[0030] The image features are input into the trained third model, and the image features are compared with the text feature library to obtain the first similarity and the first attribute of the image features;

[0031] The image features are compared with a positive sample feature library and with a negative text feature library to obtain a second similarity and a second attribute of the image features;

[0032] If both the first attribute and the second attribute are positive, check whether the first similarity or the second similarity is greater than a preset value. If the first similarity or the second similarity is greater than the preset value, the detection category of the image to be detected is obtained as a positive sample corresponding to the second similarity.

[0033] Optionally, comparing the image features with a positive sample feature library and comparing the image features with a negative text feature library to obtain a second similarity and a second attribute of the image features specifically includes:

[0034] The image features are compared with the positive sample feature library to obtain the third similarity.

[0035] The image features are compared with the negative text feature library to obtain the fourth similarity score;

[0036] If the third similarity is greater than the fourth similarity, the third similarity is used as the second similarity, and the second attribute of the image feature is positive.

[0037] If the third similarity is less than or equal to the fourth similarity, the fourth similarity is used as the second similarity, and the second attribute of the image feature is negative.

[0038] On the other hand, the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the training method or object detection method of the object detection model described above.

[0039] On the other hand, the present invention provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the training method or object detection method of the object detection model described above.

[0040] Implementing this invention provides the following beneficial effects: This invention provides an object detection model comprising a first model, a second model, and a third model. The training method for this object detection model includes the following steps: obtaining a test set; inputting the test set into the first model to obtain potential target sample images for each test image; obtaining a positive sample set and a negative sample set based on the potential target sample images for each test image; the test set includes N test images; training the second model using the positive and negative sample sets; inputting the positive sample set into the trained second model to obtain a positive sample feature library; inputting the negative sample set into the trained second model to obtain a negative sample feature library, thus enabling the detection model to have both positive and negative sample feature libraries, increasing the recall rate of the detection model; inputting potential target sample images into the trained second model to obtain the features of the potential target sample images; inputting the features of the potential target sample images into the third model to obtain the predicted category of the potential target sample images; adjusting the parameters of the third model based on the predicted category and the true category of the potential target sample images to obtain the trained third model, which can improve the detection accuracy of the detection model. Attached Figure Description

[0041] Figure 1 This is a flowchart of a training method for an object detection model provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a target detection model provided by the present invention;

[0043] Figure 3 This invention provides a flowchart for obtaining a positive sample feature library and a negative sample feature library;

[0044] Figure 4 This invention provides a flowchart for obtaining a positive sample set and a negative sample set;

[0045] Figure 5 This invention provides a flowchart for obtaining sample image text pairs;

[0046] Figure 6 This is a flowchart of adjusting the parameters of a third model provided by the present invention;

[0047] Figure 7 This is a flowchart of a target detection method provided by the present invention;

[0048] Figure 8 This is another flowchart of a target detection method provided by the present invention;

[0049] Figure 9 This is a structural diagram of a training system for an object detection model provided by the present invention;

[0050] Figure 10 This is a structural diagram of a target detection system provided by the present invention;

[0051] Figure 11 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0053] To better understand the technical solution of this invention, the following technical terms are explained.

[0054] Data augmentation is a technique in deep learning that increases the amount of training data by processing existing data to generate new training data, thus expanding the original dataset. Data augmentation tools transform existing data into new, unique samples by manipulating its parameters. Data augmentation can be performed on image, text, audio, and video inputs. There are two types of data augmentation: offline data augmentation (augmented images are stored on a drive and then combined with real data before training the model) and online data augmentation (data augmentation is applied to randomly selected images and used to train the original data).

[0055] Model generalization: Generalization refers to the ability of a model to make accurate predictions when applied to new data after it has been trained. A model is often trained too well on the training data, i.e., overfitting, so that it cannot generalize.

[0056] Recall: In relation to the original sample, it means the probability that a sample that is actually positive will be predicted as positive.

[0057] Multimodal large models: Multimodal large models are a type of general-purpose large model that can process multiple different types of data simultaneously, such as text, images, audio, or video. This type of model is better able to capture the relationships between various data types and helps improve the accuracy and efficiency of large models.

[0058] Large visual models: Large visual models refer to neural network models with tens of millions or more parameters. These models are widely used in the field of computer vision and can be used for tasks such as image classification, object detection, and semantic segmentation.

[0059] Model fine-tuning training: Given a pre-trained model, fine-tuning is performed on the model. Compared to training a model from scratch, fine-tuning training saves a lot of computational resources and time, improving computational efficiency and even accuracy.

[0060] Dual-tower structure: Generally, it is a representation feature structure with two different modalities. A common example is the image-text dual-tower structure represented by the CLIP model. In this invention, the CLIP model is used for image classification. It mainly calculates the similarity between the image and the text through the image tower and the text tower of the CLIP model to obtain the feature vector of the image to be detected, including image features and text features.

[0061] In some embodiments, such as Figure 1 and Figure 2 As shown, Figure 1 This is a flowchart of a training method for an object detection model. Figure 2 This is a schematic diagram of the structure of an object detection model. The present invention provides a training method for an object detection model, which includes a first model, a second model, and a third model. The training method includes the following steps:

[0062] S1100. Obtain the test set. Input the test set into the first model to obtain the potential target sample image of each test image. Obtain the positive sample set and negative sample set based on the potential target sample image of each test image.

[0063] The test set includes, but is not limited to, N test images, text corresponding to the target and text not corresponding to the target in the N test images.

[0064] The test image can be either a picture or a video frame.

[0065] The first model is a general segmentation model, including but not limited to the SegmentAnything Model (SAM), Fast SegmentAnything Model (fast SAM), and Mobile SegmentAnything Model (mobileSAM). The input to this segmentation model is an image or video frame, and the output is the unclassified bounding box of potential objects obtained through object detection.

[0066] Specifically, N test images are obtained from the test set, along with the text corresponding to the target and the text not corresponding to the target in the N test images. The N test images are then input into the first model in sequence, and the generalization ability of the first model is used to extract the potential instance target sample image, i.e., the potential target sample image, for each test image.

[0067] By comparing the potential target sample images with the labeled target sample images in the training set, the potential target sample images of the test images are divided into positive samples and negative samples, thus obtaining the positive sample set and the negative sample set.

[0068] By leveraging the generalization ability of large models and utilizing the differences between positive and negative sample sets, samples can be distinguished, thereby increasing the sample data.

[0069] S1200: Train the second model using the positive and negative sample sets.

[0070] The positive sample set includes, but is not limited to, several positive sample images and the text corresponding to the positive sample images; the negative sample set includes, but is not limited to, negative sample images and the text corresponding to the negative sample images.

[0071] The second model is a multimodal model, such as the Contrastive Language-Image Pre-training (CLIP) model and the fine-tuning multimodal model. The CLIP model is a dual-tower model with two feature towers: one for text and one for image.

[0072] Specifically, data augmentation is performed on the positive and negative sample sets, including text augmentation and image augmentation, to obtain a training set for the second model. The training set of the second model is then used to fine-tune the training of the second model.

[0073] By using step S1200, only fine-tuning of the model is required, which can reduce the training time of the model, improve the generalization of the second model, and reduce the possibility of model overfitting and inaccurate detection.

[0074] S1300. Input the positive sample set into the trained second model to obtain the positive sample feature library, and input the negative sample set into the trained second model to obtain the negative sample feature library.

[0075] Specifically, such as Figure 3 As shown, Figure 3 This is a flowchart for obtaining positive and negative sample feature libraries. Images from the positive and negative sample sets are sequentially input into the image tower of the second model. The corresponding text for each image in the positive and negative sample sets is input into the text tower of the second model. Using the trained image tower of the second model, image features of each target image in the positive sample set are extracted (i.e., the positive sample features in step S1200). Text features are extracted using the text tower of the second model to establish the positive sample feature library. Similarly, image features of each non-target image in the negative sample set are extracted using the trained image tower of the second model (i.e., the negative sample features in step S1200). The corresponding text features are extracted using the text tower of the second model to establish the negative sample feature library.

[0076] S1400: Input the potential target sample image into the trained second model to obtain the features of the potential target sample image.

[0077] Specifically, similar to step S1300, the potential target sample images detected by the first model are input into the trained second model, and the second model extracts the features of the potential target sample images.

[0078] S1500. Input the features of the potential target sample image into the third model to obtain the predicted category of the potential target sample image. Adjust the parameters of the third model according to the predicted category and the true category of the potential target sample image to obtain the trained third model.

[0079] The third model is the output decision model, which is used to obtain the targets and their categories in the test image based on the image features detected by the second model.

[0080] Specifically, the features of the potential target sample image are input into the third model. The third model calculates the similarity between the features of the potential target sample image and the positive sample feature library, the similarity between the features of the potential target sample image and the negative sample feature library, and the similarity between the features of the potential target sample image and the text feature library. Based on the three similarities, the category of the potential target sample image is predicted. The parameters of the third model are continuously adjusted according to the accuracy of the predicted category, that is, the distance between the predicted category and the actual category, until the accuracy of the predicted category meets certain conditions, thus completing the training of the third model.

[0081] The object detection model trained using steps S1100-S1500 can be applied to the cold start phase of projects in scenarios such as intelligent transportation and autonomous driving. Especially when training with only a very small number of target image samples, it can achieve high-precision visual detection results with few samples in anomaly detection scenarios involving complex road traffic environments such as manhole covers, roadblocks, and strollers.

[0082] Among them, the intelligent transportation scenario includes sub-scenarios such as security transportation, vehicle detection, pedestrian and passage detection, which are used for AI intelligent detection and identification of targets with few samples; the autonomous driving scenario includes sub-scenarios such as logistics and delivery, shared mobility, public transportation, ports and terminals, and smart mines, which are used for AI intelligent detection of abnormal targets; the data project scenario includes various multimodal (visual, text, etc.) projects that lack large amounts of data, such as data feedback in long-tail projects to accelerate project implementation.

[0083] In some embodiments, such as Figure 4 As shown, Figure 4 It is a flowchart for obtaining positive and negative sample sets. Figure 4 Taking a test image from the test set as an example, step S1100 inputs the test set into the first model to obtain the potential target sample image for each test image. Based on the potential target sample image for each test image, a positive sample set and a negative sample set are obtained, specifically including:

[0084] S1110. Input N test images into the first model to obtain several potential target sample boxes for each test image. Based on the several potential target sample boxes for each test image, obtain the potential target sample image for each test image.

[0085] Specifically, N test images are sequentially input into the first model. The first model obtains multiple potential target sample boxes for each test image through unsupervised learning. The multiple potential target sample boxes of the same test image are then integrated using a filtering and fusion module to obtain the potential target sample image for each test image.

[0086] For example, Figure 4 The manhole cover, roadblock, pavilion, and fallen leaves are all potential target sample images obtained by segmenting the image to be tested using the first model.

[0087] S1120. Calculate the first overlap rate between the potential target sample image and the labeled target sample box for each test image, and obtain the positive sample set and the negative sample set based on the first overlap rate; the positive sample set includes target sample images, and the negative sample set includes non-target sample images.

[0088] The positive sample set includes, but is not limited to, several positive sample images (target sample images) and the text corresponding to the positive sample images; the negative sample set includes, but is not limited to, negative sample images (non-target sample images) and the text corresponding to the negative sample images.

[0089] Similar to the potential target sample images, the labeled target sample bounding boxes can be obtained by passing images containing the target in the training set GT through the first model, and by labeling the potential target sample images detected by the first model.

[0090] Specifically, the Intersection over Union (IOU) calculation method is used to calculate the overlap ratio (IOU) between the potential target sample image and the labeled target sample box in each test image. When the IOU is greater than a preset value, which can be set according to the detection accuracy (e.g., a preset value of 0.75), the potential target box in the image is considered to have matched the labeled target sample box, and the potential target sample image is used as the target sample obtained through segmentation. When the IOU is less than or equal to the preset value, which can be set according to the detection accuracy (e.g., a preset value of 0.75), the potential target sample image is used as a non-target sample. In the positive sample set, the target sample is a positive sample, and in the negative sample set, the non-target sample is a negative sample.

[0091] For example, Figure 3 In the study, manhole covers and roadblocks were classified as positive samples, while pavilions and fallen leaves were classified as negative samples.

[0092] In some embodiments, step S1110 obtains a potential target sample image for each test image based on a plurality of potential target sample boxes for each test image, specifically including:

[0093] S1111 Calculate the second overlap rate of several potential target sample boxes for each test image.

[0094] Each test image may include at least one potential target bounding box.

[0095] Specifically, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the calculation of target bounding box overlap rate. It calculates the ratio of the intersection and union of the areas of every two potential target sample boxes in the same test image, and uses this ratio as the second overlap rate.

[0096] S1112. The potential target sample boxes with a second overlap rate greater than the preset value are integrated to obtain the potential target sample image of each test image.

[0097] Among them, the potential target sample boxes for target integration belong to the same test image.

[0098] Specifically, when the IOU value between two potential target sample boxes is greater than ths, the value of ths can be set according to the detection accuracy. For example, if ths = 0.75, then the two potential target sample boxes are considered to be the same target. The two potential target sample boxes are then integrated to obtain a potential target sample image.

[0099] In some embodiments, step S1200 trains a second model using a positive sample set and a negative sample set, specifically including:

[0100] S1210. Perform image fusion on the potential target sample images in the positive sample set, and perform text fusion on the target sample text in the positive sample set.

[0101] Specifically, such as Figure 5 As shown, Figure 5 This is a flowchart for obtaining text-image pairs from sample images. The diagram uses data fusion from a positive sample set as an example. Due to the extremely small amount of data, this invention proposes combining a very small number of text-target image pairs with a large number of text-non-target image pairs. Specifically, the text-target image pairs are obtained by fusing the target sample images from the positive sample set in step S1100. For example, merging image 1 and image 2, or scaling two cat images to the same resolution before merging them to obtain a new image. The text corresponding to the target sample images is also fused; for example, text 1 corresponding to image 1 is merged with text 2 corresponding to image 2.

[0102] S1220. Obtain sample image-text pairs based on the fused potential target sample images and the fused target sample texts.

[0103] Specifically, the fused image and text from step S1210 are combined to obtain sample image-text pairs, for example... Figure 5 Lieutenant General Figure 1 and Figure 2The fusion process yields a merged image from the two images. The fusion of text 1 and text 2 results in a merged text. The merged text and the merged image together form a sample image-text pair.

[0104] S1230. Combine the sample image text pairs with the negative sample set to obtain a multimodal training sample set.

[0105] Specifically, the target samples in the sample image-text pairs are then fused with the non-target samples in the negative sample set to obtain a multimodal training sample set. The fusion method is similar to that in step S1210. By augmenting the positive and negative sample sets, the size of the dataset can be increased.

[0106] S1240. Train the second model using the multimodal training sample set.

[0107] Specifically, the text from the multimodal training sample set in step S1230 is fed into the text tower to extract text features, and the images are fed into the image tower to extract image features. To align the image features with the text features, the CLIP-based image target recognition task is transformed into an image feature retrieval task that is most similar to the text features. The target in the image is identified, and the second model is adjusted based on the identification, thus completing the training of the second model.

[0108] In some embodiments, such as Figure 6 As shown, Figure 6 This is a flowchart for adjusting the parameters of a third model. Features of a potential target sample image are input into the third model to obtain the predicted category of the potential target sample image. The parameters of the third model are then adjusted based on the predicted category and the true category of the potential target sample image to obtain a trained third model. Specifically, it includes:

[0109] S1510. Input the features of the potential target sample image into the third model. The third model calculates the similarity between the features of the potential target sample image and the first sample in the positive sample feature library, and calculates the similarity between the features of the potential target sample image and the second sample in the negative sample feature library.

[0110] Specifically, during training, the third sample similarity is first calculated between the features of the potential target sample images obtained by the second model and the text features in the positive and negative sample feature library. The cosine similarity distance is used, as shown in formula (1), to obtain the third sample similarity.

[0111] Sim_score = (a1 * I_f + a2 * T_f) / (a1 + a2) (1)

[0112] Where Sim_score is the specific similarity score, a1 and a2 are hyperparameters, I_f is the feature of the potential target sample image, and T_f is the text feature.

[0113] The higher the similarity of the third sample, the more similar the features of the potential target sample image are to the corresponding text features. If the similarity is less than a threshold, such as 0.5, the potential target sample image is considered unreliable and is a negative sample, i.e., a non-target sample image. The positive sample is used as the initial category obtained in this detection. If the similarity is greater than or equal to a threshold, such as 0.5, the potential target sample image is considered unreliable and is a positive sample, i.e., a target sample image. The negative sample is used as the initial category obtained in this detection.

[0114] The features of the potential target sample image are compared with the target sample features in the positive sample library and the non-target sample features in the negative sample library, respectively. The similarity distance is calculated for each comparison to obtain the first sample similarity between the features of the potential target sample image and the positive sample feature library and the second sample similarity between the features of the potential target sample image and the negative sample feature library. The method for calculating the similarity distance is similar to the method for calculating the third sample similarity.

[0115] S1520. Based on the similarity between the first sample and the second sample, the predicted category of the potential target sample image is obtained. The parameters of the third model are adjusted according to the predicted category and the true category of the potential target sample image. The process of inputting the features of the potential target sample image into the third model is repeated until the predicted category meets the preset conditions, and the trained third model is obtained.

[0116] The preset conditions can be, but are not limited to, the detection accuracy meeting a preset value, meeting the learning rate requirement, and reaching a preset number of training iterations.

[0117] Specifically, if the similarity between the features of a potential target sample image and the first sample matching the positive sample feature library is greater than the similarity between the features of a second sample matching the negative sample feature library, the potential target sample image is initially considered to be a positive sample, and the positive sample is used as the second initial category of the potential target sample image.

[0118] If the similarity of the first sample matching the positive sample feature library is less than or equal to the similarity of the second sample matching the negative sample feature library, then the potential target sample image is considered a negative sample, and the negative sample is used as the second initial category of the potential target sample image.

[0119] The similarity values ​​of the first and second samples are compared, and the maximum value between the first and second sample similarities is taken as the initial sample similarity.

[0120] The first initial category is obtained by comparing the features of the potential target sample image with the text features, and the second initial category is obtained by comparing the features of the potential target sample image with the positive and negative sample database. A second comparison is performed. If either the first initial category or the other first initial category is a negative sample, the predicted category of the image is a negative sample, and the image is not output and is directly filtered. If both the first initial category and the other first initial category are positive samples, the highest value between the third sample similarity and the initial sample similarity is taken, and it is determined whether the highest value is greater than a threshold, for example, 0.5. If the highest value is greater than the threshold, the predicted category of the potential target sample image is the positive sample corresponding to the initial sample similarity; otherwise, the predicted category of the potential target sample image is the negative sample corresponding to the initial sample similarity and is filtered out.

[0121] The accuracy of the third model's category recognition for the features of the potential target sample image is confirmed by comparing the predicted category with the actual category of the potential target sample image. If the predicted category is similar to the actual category of the potential target sample image, the third model after the last adjustment is considered a well-trained third model.

[0122] If the predicted category is not similar to the actual category of the potential target sample image, adjust the parameters of the third model, such as the hyperparameters of the third model, input a new potential target sample image, and return to execute steps S1510 and S1520 until the predicted category is similar to the actual category of the potential target sample image, thus obtaining the trained third model.

[0123] Through steps S1510-S1520, the potential target sample images are compared with the positive sample feature library and the negative sample feature library, which improves the recognition accuracy of targets with few samples from the perspective of retrieval.

[0124] In some embodiments, such as Figure 7 As shown, Figure 7 This is a flowchart of a target detection method. The present invention provides a target detection method comprising the following steps:

[0125] S2100. Obtain the image to be detected, input the image to be detected into the first model, and obtain the potential target image.

[0126] The image to be detected can be either a picture or a video frame.

[0127] Specifically, the process of passing the image to be detected through the first model and the second model to detect the potential target image is similar to the process of passing the test image through the first model to obtain the potential sample target image in steps S1111-S1112, and will not be described again here.

[0128] S2200: Input the potential target image into the trained second model to obtain the image features of the potential target image.

[0129] The second model was trained using the same method described above for the object detection model.

[0130] Specifically, the potential target image is processed through the second model trained in steps S1210-S1240 to obtain the image features of the potential target image. The type of the second model is the same as described above.

[0131] S2300: Input the image features into the trained third model to obtain the detection category of the image to be detected.

[0132] The third model was trained using the same method described above for the object detection model.

[0133] Specifically, the image features are input into the third model trained through steps S1510-S1520. The third model compares the image features with the positive sample feature library, the negative sample feature library, and their corresponding text to obtain the detection category of the image to be detected.

[0134] First, the potential target object image is obtained through the first model. Then, the segmented image is fed into the trained second model to obtain the image feature vector. Finally, the image feature vector to be matched is compared with the pre-extracted text feature vector, as well as the features of the positive sample feature library and the negative sample feature library. The three similarities of the image features to be matched are calculated. Then, the category to be detected is inferred through a high-precision decision-level fusion method, thus completing the few-shot visual detection task.

[0135] This invention innovatively constructs a general, high-precision few-sample visual detection framework with minimal training cost by using a first model, a second model, multi-task cascading (a third model / output decision model), and feature library retrieval techniques. It can improve the visual recognition accuracy of few-sample targets and the recall rate of the model with very few target image training samples. It can be applied to the cold start problem of projects and application scenarios where it is impossible to collect a large amount of training data.

[0136] In some embodiments, such as Figure 8 As shown, Figure 8 This is another flowchart of an object detection method. Image features are input into a trained third model to obtain the detection category of the image to be detected. Specifically, it includes:

[0137] S2310. Input the image features into the trained third model, compare the image features with the text feature library, and obtain the first similarity and the first attribute of the image features.

[0138] Specifically, calculating the first similarity is similar to calculating the third sample similarity in step S1510, and the process of confirming the first attribute is similar to the process of obtaining the first initial category, so it will not be described again here.

[0139] S2320. Compare the image features with the positive sample feature library and the image features with the negative text feature library to obtain the second similarity and the second attribute of the image features.

[0140] Specifically, the process of obtaining the second similarity is similar to the process of obtaining the initial sample similarity in step S1520, and the process of obtaining the second attribute of the image feature is similar to the process of obtaining the first initial category in step S1520, which will not be repeated here.

[0141] S2330. If both the first and second attributes are positive, confirm whether the first similarity or the second similarity is greater than a preset value. If the first similarity or the second similarity is greater than the preset value, the detection category of the image to be detected is obtained as a positive sample corresponding to the second similarity.

[0142] Specifically, the process of obtaining the detection category of the image to be detected is similar to the process of obtaining the predicted category in step S1520, and will not be repeated here.

[0143] In some embodiments, step S2320 compares the image features with a positive sample feature library and with a negative text feature library to obtain a second similarity and a second attribute of the image features, specifically including:

[0144] S2321. Compare the image features with the positive sample feature library to obtain the third similarity.

[0145] Specifically, after the image features are input into the third model, the third model compares the image features with the features in the positive sample feature library, calculates the cosine similarity distance, and obtains the third similarity. The calculation process of the third similarity is similar to the calculation process of the first sample similarity in step S1510, and will not be repeated here.

[0146] S2322. Compare the image features with the negative text feature library to obtain the fourth similarity score.

[0147] Specifically, after the image features are input into the third model, the third model also compares the image features with the features in the negative sample feature library, calculates the cosine similarity distance, and obtains the fourth similarity. The calculation process of the fourth similarity is similar to the calculation process of the second sample similarity in step S1510, and will not be repeated here.

[0148] S2323a. If the third similarity is greater than the fourth similarity, the third similarity is used as the second similarity, and the second attribute of the image feature is positive.

[0149] S2323b. If the third similarity is less than or equal to the fourth similarity, the fourth similarity is used as the second similarity, and the second attribute of the image feature is negative.

[0150] Specifically, steps S2323a and S2323b are the process of confirming the second attribute and the second similarity. Specifically, the similarity obtained by comparing the image features with the positive sample feature library and the negative sample feature library is compared respectively. That is, the magnitude of the third similarity and the fourth similarity is compared. The maximum value of the third similarity and the fourth similarity is taken as the second similarity, and the positive sample corresponding to the maximum value is taken as the second attribute of the image to be detected. This is similar to the process of obtaining the second initial category in step S1520, and will not be described again here.

[0151] In some embodiments, such as Figure 9 As shown, Figure 9 This invention provides a structural diagram of a training system for an object detection model, comprising a first segmentation module, a first training module, a feature library establishment module, a second segmentation module, and a second training module.

[0152] The first segmentation module is used to acquire the test set, input the test set into the first model, obtain the potential target sample image for each test image, and obtain the positive sample set and negative sample set based on the potential target sample image for each test image; the test set includes N test images;

[0153] The first training module is used to train the second model using the positive and negative sample sets;

[0154] The feature library building module is used to input the positive sample set into the trained second model to obtain the positive sample feature library, and to input the negative sample set into the trained second model to obtain the negative sample feature library;

[0155] The second segmentation module is used to input the potential target sample image into the trained second model to obtain the features of the potential target sample image.

[0156] The second training module is used to input the features of the potential target sample image into the third model to obtain the predicted category of the potential target sample image. Based on the predicted category and the true category of the potential target sample image, the parameters of the third model are adjusted to obtain the trained third model.

[0157] The specific implementation method is the same as the training method of the object detection model described above, and will not be repeated here.

[0158] In some embodiments, the first segmentation module is specifically used to input N test images into the first model to obtain several potential target sample boxes for each test image, and to obtain potential target sample images for each test image based on the several potential target sample boxes for each test image.

[0159] The first segmentation module calculates the first overlap rate between the potential target sample image and the labeled target sample box for each test image, and obtains a positive sample set and a negative sample set based on the first overlap rate; the positive sample set includes target sample images, and the negative sample set includes non-target sample images.

[0160] In some embodiments, the first segmentation module is further configured to calculate a second overlap rate of several potential target sample boxes for each test image.

[0161] Potential target sample boxes with a second overlap rate greater than a preset value are integrated to obtain potential target sample images for each test image; potential target sample boxes that are integrated belong to the same test image.

[0162] In some embodiments, the first training module is specifically used to perform image fusion on potential target sample images in the positive sample set and text fusion on target sample text in the positive sample set.

[0163] Based on the fused potential target sample image and the fused target sample text, sample image-text pairs are obtained;

[0164] The sample image text pairs are combined with the negative sample set to obtain the multimodal training sample set;

[0165] The second model is trained using a multimodal training sample set.

[0166] In some embodiments, the second training module is specifically used to input the features of the potential target sample image into the third model, and the third model calculates the similarity between the features of the potential target sample image and the first sample in the positive sample feature library, and calculates the similarity between the features of the potential target sample image and the second sample in the negative sample feature library.

[0167] The predicted category of the potential target sample image is obtained based on the similarity between the first and second samples. The parameters of the third model are adjusted based on the predicted category and the true category of the potential target sample image. The process of inputting the features of the potential target sample image into the third model is repeated until the predicted category meets the preset conditions, resulting in a trained third model.

[0168] The specific implementation method is the same as the training method of the object detection model described above, and will not be repeated here.

[0169] In some embodiments, such as Figure 10 As shown, Figure 10 This is a structural diagram of a target detection system. The present invention provides a target detection system comprising a first module, a second module, and a third module, wherein...

[0170] The first module is used to acquire the image to be detected and input the image to be detected into the first model to obtain a potential target image;

[0171] The second module is used to input the potential target image into the trained second model to obtain the image features of the potential target image.

[0172] The third module is used to input the image features into the trained third model to obtain the detection category of the image to be detected.

[0173] Specifically, the first model detects potential target images in the image to be detected, the second model detects the features of the potential target images, and finally the third model compares the features of the potential target images with the positive sample feature library and the negative sample feature library to output the predicted category. The process of each model of a target detection method obtaining the predicted category based on the image to be detected is the same, and will not be repeated here.

[0174] In some embodiments, the third module is specifically used to input image features into a trained third model, compare the image features with a text feature library, and obtain a first similarity and a first attribute of the image features;

[0175] The image features are compared with the positive sample feature library and the image features are compared with the negative text feature library to obtain the second similarity and the second attribute of the image features.

[0176] If both the first and second attributes are positive, check whether the first similarity or the second similarity is greater than a preset value. If the first similarity or the second similarity is greater than the preset value, the detection category of the image to be detected is the positive sample corresponding to the second similarity.

[0177] In some embodiments, the third module is further configured to compare the image features with a positive sample feature library to obtain a third similarity.

[0178] The image features are compared with the negative text feature library to obtain the fourth similarity score.

[0179] If the third similarity is greater than the fourth similarity, the third similarity is used as the second similarity, and the second attribute of the image feature is positive.

[0180] If the third similarity is less than or equal to the fourth similarity, the fourth similarity is used as the second similarity, and the second attribute of the image feature is negative.

[0181] The specific implementation method is the same as that of the target detection method described above, and will not be repeated here.

[0182] In some embodiments, such as Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of an electronic device provided by the present invention. The present invention also provides an electronic device, which includes a processor 110 and a memory 111. The memory 111 stores a computer program, and the processor 110 executes the computer program to implement any of the methods described in the above method embodiments.

[0183] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. The memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include remote memory located remotely relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0184] Furthermore, embodiments of this application also disclose a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the methods described above.

[0185] The present invention also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform any of the methods described in the above method embodiments.

[0186] Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0187] It is understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0188] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A training method for an object detection model, characterized in that, The target detection model includes a first model, a second model, and a third model, and the method includes the following steps: Obtain a test set, input the test set into the first model, and obtain the potential target sample image for each test image. Obtain a positive sample set and a negative sample set based on the potential target sample image for each test image. The test set includes N test images, and the first model is a large segmentation model. Training the second model using the positive and negative sample sets specifically includes: Image fusion is performed on potential target sample images in the positive sample set, and text fusion is performed on target sample text in the positive sample set. Based on the fused potential target sample image and the fused target sample text, sample image-text pairs are obtained; The sample image text pairs are combined with the negative sample set to obtain a multimodal training sample set; The second model is trained using the multimodal training sample set, wherein the second model is a multimodal model; The positive sample set is input into the trained second model to obtain a positive sample feature library, and the negative sample set is input into the trained second model to obtain a negative sample feature library. The potential target sample image is input into the trained second model to obtain the features of the potential target sample image; The features of the potential target sample image are input into the third model to obtain the predicted category of the potential target sample image. The parameters of the third model are adjusted according to the predicted category and the true category of the potential target sample image to obtain the trained third model, specifically including: The features of the potential target sample image are input into the third model, which calculates the similarity between the features of the potential target sample image and the first sample in the positive sample feature library, and calculates the similarity between the features of the potential target sample image and the second sample in the negative sample feature library. The predicted category of the potential target sample image is obtained based on the similarity between the first sample and the second sample. The parameters of the third model are adjusted based on the predicted category and the true category of the potential target sample image. The process of inputting the features of the potential target sample image into the third model is repeated until the predicted category meets the preset conditions, resulting in a trained third model. The third model is an output decision model.

2. The method according to claim 1, characterized in that, The step of inputting the test set into the first model to obtain a potential target sample image for each test image, and obtaining a positive sample set and a negative sample set based on the potential target sample image for each test image, specifically includes: The N test images are input into the first model to obtain several potential target sample boxes for each test image. Based on the several potential target sample boxes for each test image, the potential target sample image for each test image is obtained. Calculate the first overlap rate between the potential target sample image and the labeled target sample box for each test image, and obtain the positive sample set and the negative sample set based on the first overlap rate; the positive sample set includes target sample images, and the negative sample set includes non-target sample images.

3. The method according to claim 2, characterized in that, The step of obtaining a potential target sample image for each test image based on several potential target sample boxes specifically includes: Calculate the second overlap rate of several potential target bounding boxes for each test image; The potential target sample boxes with the second overlap rate greater than the preset value are integrated to obtain potential target sample images for each test image; wherein the potential target sample boxes that are integrated belong to the same test image.

4. A target detection method, characterized in that, Includes the following steps: The image to be detected is acquired and input into the first model to obtain the potential target image; The potential target image is input into the trained second model to obtain the image features of the potential target image; the second model is trained by the method described in any one of claims 1-3. The image features are input into the trained third model to obtain the detection category of the image to be detected; the third model is trained by the method described in any one of claims 1-3.

5. The detection method according to claim 4, characterized in that, The step of inputting the image features into the trained third model to obtain the detection category of the image to be detected specifically includes: The image features are input into the trained third model, and the image features are compared with the text feature library to obtain the first similarity and the first attribute of the image features; The image features are compared with a positive sample feature library and with a negative sample feature library to obtain a second similarity and a second attribute of the image features; If both the first attribute and the second attribute are positive, check whether the first similarity or the second similarity is greater than a preset value. If the first similarity or the second similarity is greater than the preset value, the detection category of the image to be detected is obtained as a positive sample corresponding to the second similarity.

6. The detection method according to claim 5, characterized in that, The step of comparing the image features with a positive sample feature library and comparing the image features with a negative text feature library to obtain a second similarity and a second attribute of the image features specifically includes: The image features are compared with the positive sample feature library to obtain the third similarity. The image features are compared with the negative text feature library to obtain the fourth similarity score; If the third similarity is greater than the fourth similarity, the third similarity is used as the second similarity, and the second attribute of the image feature is positive. If the third similarity is less than or equal to the fourth similarity, the fourth similarity is used as the second similarity, and the second attribute of the image feature is negative.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It contains a processor-executable program, which, when executed by a processor, is used to perform the method as described in any one of claims 1-6.

Images