A Small-Sample Object Detection Method Based on Feature Relationship Migration

Abstract

The present invention provides a training method for a small-sample object detection model based on feature relationship migration. The method includes: S1. Obtain a training set and a support set, and use the training set to train a pre-trained two-stage model to obtain an initialized object detection basic model and a region of interest adjustment module; S2. Use the training set and the support set to perform multiple iterative trainings on a training model composed of the initialized object detection basic model, the region of interest adjustment module, the feature extraction network, the relationship attention module, the class classification module, and the detection box position regression adjustment module until convergence to obtain a final small-sample object detection model composed of the object detection basic model, the region of interest adjustment module, the feature extraction network, the detection box position regression adjustment module, and the class classification module. The present invention improves the object positioning ability of the small-sample object detection model, and can also solve the problem of learning and forgetting of the base class while having the ability to recognize small-sample classes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision. Specifically, it relates to the field of object detection in computer vision. More specifically, it relates to a training method for a small-sample object detection model based on feature relationship transfer and a small-sample object detection method based on the trained model. Background Art

[0002] Computer vision is an important field in artificial intelligence (AI), which is used for a computer to obtain meaningful information from images, videos or other visual inputs, and provide suggestions or reference information for downstream tasks based on this information. Computer vision involves multiple task branches, such as object detection, image classification, etc., which are of great significance for downstream tasks in artificial intelligence. Among them, the object detection task is an important branch in the field of computer vision. This task requires finding all the objects of interest in an image and determining their corresponding categories and positions. A relatively common object detection task is to train a model with a high accuracy on a dataset with a large amount of annotation resources for object recognition. However, annotation resources are relatively scarce, and the annotation work requires extremely high costs. Moreover, samples of some categories (such as rare animal faces) are also relatively difficult to obtain. Therefore, in practical applications, the relatively common one is the detection task of small-sample categories. To achieve the detection task of small-sample categories, a common approach is to combine the information of the source domain dataset (also called the training set) of the category with relatively sufficient training samples and the target domain dataset (also called the support set) of the category with only a small number of training samples (small-sample category) to achieve the object detection of the small-sample type.

[0003] Currently, most of the detection tasks of small-sample categories are based on the deep neural network framework. For example, the "Implementation Method and Device of an Object Detector Based on Limited Samples" disclosed in the Chinese patent application document with the patent publication number CN108229658A is a detection method for small-sample categories based on the regularization transfer learning method. The main technical means of this method is to establish an object detector (object detection model) based on a neural network, and design a regularization-based transfer learning method to obtain the knowledge transfer from the source domain dataset (training set) to the target domain dataset (support set) and the background suppression regularization term to train this object detector. This type of method only transfers knowledge at the classification result level, and does not mine the relationship between the feature attributes of the samples in the training set and the support set in a fine-grained manner, which affects the category classification ability and the object position detection ability of the object detection model.

[0004] For another example, the "Object Detection Method under Limited Samples" disclosed in the Chinese patent application document with the patent publication number CN112364747A is a small-sample category detection method based on graph structure modeling. The main technical means of this method are as follows: Use the model trained on the source domain (the categories on the source domain are called base classes) to screen the prediction candidate boxes in the picture samples in the support set (the categories in the support set are called new classes, and the samples in the support set are called new class samples), obtain the candidate boxes containing objects (the candidate boxes containing objects are candidate regions or regions of interest), and perform convolution processing on them. The convolution features corresponding to the obtained candidate regions form a graph structure to train the class labels of each candidate region. This method only constructs the relationship between the features and classification results of the base classes and new classes to detect the category of each candidate region. However, due to the lack of direct enhancement of the features of a small number of samples in the new class, the generalization detection ability of the model obtained by this method for the new class is not good.

[0005] In summary, in the prior art, only transfer training is performed on the object detection model at the classification result level, and the relationship between the feature attributes of the samples in the source domain training set and the target domain support set is not mined. As a result, the trained object detection model does not fully utilize the relationship between the features of the new class and the base class, leading to the problem that the trained object detection model has a learning and forgetting problem for the base class categories and an inaccurate positioning problem in the object detection of small-sample categories (new classes). Summary of the Invention

[0006] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a training method for a small-sample object detection model based on feature relationship transfer and a small-sample object detection method based on the trained model.

[0007] According to a first aspect of the present invention, there is provided a training method for a few-shot object detection model based on feature relationship transfer, the method comprising: S1. Obtain a training set and a support set, and use the training set to train a pre-trained two-stage model to obtain an initialized object detection basic model and a region of interest adjustment module; wherein, the training set includes a plurality of base classes, each base class having a plurality of samples with base class labels, and the base class labels include the class label of the base class and the target detection box position label; the support set includes a plurality of new classes different from the base class categories, each new class having samples that meet the requirements of the few-shot task and have new class labels, and the new class labels include the class label of the new class, the target detection box position label, and the background label; S2. Use the training set and the support set to perform multiple iterative trainings on the training model composed of the initialized object detection basic model, the initialized region of interest adjustment module, the feature extraction network, the relationship attention module, the class classification module, and the detection box position regression adjustment module until convergence to obtain a final few-shot object detection model composed of the object detection basic model, the region of interest adjustment module, the feature extraction network, the detection box position regression adjustment module, and the class classification module.

[0008] In some embodiments of the present invention, each iterative training of the present invention includes: S21. Use the initialized object detection basic model to extract the original regions of interest of each sample in the training set and the support set respectively; S22. Use the region of interest adjustment module to adjust the original regions of interest of each sample in the training set and the support set extracted based on step S21; S23. Use the feature extraction network to extract the original features of the adjusted regions of interest of each sample in the training set and the support set after step S22; S24. Use the relationship attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set; S25. Use the detection box position regression adjustment module to output the predicted target detection box position of the sample based on the original features of each sample in the support set, and calculate the target detection box position regression loss based on the target detection box position labels and the predicted target detection box positions of all samples in the support set; S26. Use the class classification module to perform class classification on the sample based on the enhanced features of each sample in the support set to obtain the predicted classification result, and calculate the class classification loss based on the predicted classification results and the class labels of all samples in the support set; S27. Use the target detection box position regression loss and the class classification loss to update the parameters of the training model.

[0009] In some embodiments of the present invention, the training model further includes a background classification module. Each iteration training of the present invention further includes: S26': Using the background classification module to output the background classification prediction result of the sample based on the original features of each sample in the support set, and calculating the background classification loss based on the background classification prediction results of all samples in the support set and the background labels; S27': Using the target detection box position regression loss, the background classification loss, and the category classification loss to update the parameters of the training model.

[0010] In some embodiments of the present invention, the present invention uses the target detection box position regression loss to update the parameters of the detection box position regression adjustment module; uses the category classification loss and the background classification loss to update the parameters of the category classification module, the feature extraction network, and the region of interest adjustment module.

[0011] In some embodiments of the present invention, the pre-trained two-stage model is any one of the following networks: RCNN, FastRCNN, FasterRCNN, SPPNet, FPN.

[0012] In some embodiments of the present invention, in step S24, the original features of each sample in the support set are enhanced through the following steps: S241: Calculating the attention matrix of the sample in the support set for all samples in the training set based on the features of all samples in the training set extracted in step S21 and the features of the current sample in the support set; S242: Enhancing the original features of the sample based on the attention matrix and features of the current sample obtained in step S241 through the following formula:

[0013] F_qknew = F_qk + Rk

[0014] where F_qknew is the enhanced feature of the k-th sample in the support set, F_qk is the original feature of the k-th sample in the support set, and Rk is the attention matrix of the k-th sample in the support set for all samples in the training set.

[0015] In some embodiments of the present invention, the present invention uses the following formula to calculate the attention matrix of each sample in the support set for all samples in the training set:

[0016] Rk = softmax(Uk) * F_s = {softmax(Ukv) * F_sv}

[0017] Uk = {Ukv}

[0018] F_s = {F_sv}

[0019]

[0020]

[0021] Among them, Rk is the attention matrix of the k-th sample in the support set to all samples in the training set, Uk is the similarity matrix of the k-th sample in the support set to all samples in the training set, softmax(Uk) normalizes each element in the similarity matrix respectively, F_s is the set of features of all samples in the training set, F_SV is the feature of the v-th sample in the training set, F_qk is the feature of the k-th sample in the support set, Ukv is the v-th element in Uk, Uki is the similarity between the k-th sample in the support set and the i-th base class sample, and n is the number of all base class samples in the training set.

[0022] In some embodiments of the present invention, the following formula is used to calculate the background classification loss of the support set:

[0023]

[0024] Among them, bglabel is the background label of the new class, K_fb represents the set of original features of all samples in the support set input to the background classification module, and sigmoid(K_fb) is the background classification result corresponding to the feature set K_fb.

[0025] In some embodiments of the present invention, the following formula is used to calculate the class classification loss of the support set:

[0026] LF = objlabel * log(Softmax(Kc)) = objlabel * log(Softmax({Kc i}));

[0027] Among them, objlabel is the class label of the new class, Kc i represents the probability that all samples of each class in the support set are classified as the i-th class, and Softmax(Kc_i) is the normalized classification probability that all samples of each class in the support set are classified as the i-th class.

[0028] In some embodiments of the present invention, the following formula is used to calculate the regression loss in step S25:

[0029]

[0030] Among them, bi jx is the x-axis coordinate of the j-th vertex in the predicted target detection frame position after the original feature of the i-th sample in the support set is adjusted by the detection frame position regression adjustment module, bi jy is the y-axis coordinate in the predicted target detection frame position after the original feature of the i-th sample in the support set is adjusted by the detection frame position regression adjustment module, Di jxis the x - axis coordinate of the j - th vertex in the target detection box position label of the i - th sample in the support set, Di jy is the y - axis coordinate of the j - th vertex in the target detection box position label of the i - th sample in the support set.

[0031] According to the second aspect of the present invention, there is provided a target detection method, which includes: F1. Obtain an input image; F2. Use the small - sample target detection model obtained by the method described in the first aspect of the present invention to perform target detection on the input image obtained in step F1 to obtain the target positions and target classifications in the image.

[0032] Compared with the prior art, the advantages of the present invention are as follows: For the problem of small - sample category detection (such as target categories like objects, animal face detection, etc.), the present invention introduces a detection box position regression adjustment module as a regression branch to predict the target detection box position, and calculates the target detection box position regression loss of all samples in the support set based on the target detection box position labels and predicted target detection box positions of all samples in the support set, and then adjusts the parameters of the detection box position regression adjustment module based on the regression loss, thereby improving the target localization ability in the small - sample target detection task; and also introduces a relationship attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set. The category classification module performs category classification prediction based on the enhanced features of each sample in the support set, and calculates the category classification loss of the support set based on the predicted classifications and category labels of all samples in the support set, and then adjusts the parameters of the training model based on the category classification loss, so that the final small - sample target detection model can have the ability to recognize small - sample categories while retaining the learning knowledge of the base classes, solving the problem of learning forgetting of the base classes existing in the existing models. Description of the Drawings

[0033] The following further describes the embodiments of the present invention with reference to the drawings, where:

[0034] Figure 1 is a schematic flow chart of a training method for a small - sample target detection model based on feature relationship transfer according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a training method for a small - sample target detection model based on feature relationship transfer according to an embodiment of the present invention. Detailed Embodiments

[0036] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further details the present invention through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0037] As mentioned in the background art, in the prior art, only the target detection model is transfer-trained at the classification result level, and the relationship between the feature attributes of the samples in the training set and the support set is not mined, so that the converged target detection model does not make full use of the relationship between the new class and the base class features, resulting in the problem that the converged target detection model has the problem of learning and forgetting the base class and the problem of insufficient localization ability in the target detection of small sample classes.

[0038] To solve the above problems, the solution provided by the present invention is to introduce an auxiliary training task (such as a target regression detection task, a background classification task, etc.) to transfer-train the training model in addition to the basic target detection task in the process of obtaining a small sample target detection model by transfer-training the target detection model, thereby improving the target localization ability of the small sample target detection model, making it have generalization ability for small sample classes while avoiding learning and forgetting of the base class.

[0039] Generally speaking, the present invention first provides a training method for a small sample target detection model based on feature relationship transfer, as Figure 1 shown, the method includes: S1. Obtain a training set and a support set, and use the training set to train a pre-trained two-stage model (that is, a two-stage model that has been trained in the prior art) to obtain an initialized target detection basic model and a region of interest adjustment module; S2. Use the training set and the support set to train the training model constructed based on the initialized target detection basic model and the region of interest adjustment module for multiple iterative trainings until convergence to obtain a final small sample target detection model.

[0040] According to an embodiment of the present invention, step S2 includes: using a training set and a support set to perform multiple iterative trainings on a training model composed of an initialized object detection basic model, an initialized region of interest adjustment module, a feature extraction network, a relational attention module, a class classification module, and a detection box position regression adjustment module (i.e., constructing a training model) until convergence to obtain a final few-shot object detection model composed of an object detection basic model, a region of interest adjustment module, a feature extraction network, a detection box position regression adjustment module, and a class classification module, and using an object detection box position regression loss and a class classification loss to update the parameters of the training model. Among them, the present invention improves the object localization ability in the few-shot object detection task by introducing a detection box position regression adjustment module as a regression branch to predict the object detection box position, calculating the object detection box position regression loss of all samples in the support set based on the object detection box position labels and the predicted object detection box positions of all samples in the support set, and then adjusting the parameters of the detection box position regression adjustment module based on the regression loss. The present invention also introduces a relational attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set. The class classification module performs class classification prediction based on the enhanced features of each sample in the support set, calculates the class classification loss of the support set based on the predicted classifications and class labels of all samples in the support set, and then adjusts the parameters of the training model based on the class classification loss, so that the final few-shot object detection model can have the ability to recognize few-shot classes while retaining the learning knowledge of the base classes, solving the problem of learning forgetting of the base classes in the existing model. In addition, in order to further improve the recognition ability of the final few-shot object detection model for few-shot classes, the present invention also introduces a background classification module into the training model, and uses it to output the background classification prediction results of the samples based on the original features of each sample in the support set to obtain the background classification predictions of all samples in the support set, calculates the background classification loss of the support set based on the background classification predictions of all samples and the background labels, and uses it to update the model parameters, enhancing the background perception ability of the final few-shot object detection model, thereby improving the recognition ability of the model for few-shot class objects.

[0041] According to an embodiment of the present invention, the object detection basic model and the region of interest adjustment module in the present invention adopt a pre-trained two-stage model. According to an embodiment of the present invention, the object detection basic model is any one of the following models: RCNN, FastRCNN, FasterRCNN, SPPNet, FPN. These models can all achieve basic object detection and region of interest adjustment. For the convenience of understanding, in the present invention, the part used to implement the object detection function is called the object detection basic model, and the part used to implement the region of interest adjustment is called the region of interest adjustment module. After initializing the training of the two-stage model with the training set, an initialized object detection basic model and a region of interest adjustment module can be obtained. During the transfer training process, the present invention solidifies the parameters of the object detection basic model initialized by the training set and iteratively updates the parameters of the region of interest adjustment function module. According to an embodiment of the present invention, the feature extraction network adopts a CNN network. According to an embodiment of the present invention, both the class classification module and the background classification module adopt fully connected layers. Since the network structures and functions adopted by the training model are basically known to those skilled in the art, the present invention does not elaborate too much on the specific functions of each sub-module and sub-network in the training model, and only introduces the present invention from the level of the training method of the small sample object detection model.

[0042] To better understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0043] Figure 2The figure shows a schematic diagram of a training method for a few-shot object detection model based on feature relationship transfer. The area within the dashed box represents the constructed training model and the data flow in the model. The training model includes an object detection basic model, a region of interest adjustment module, a feature extraction network, a relationship attention module, a class classification module, and a detection box position regression adjustment module. Among them, the few-shot object detection model includes an object detection basic model, a region of interest adjustment module, a feature extraction network, a detection box position regression adjustment module, and a class classification module. The arrows between the modules represent the specific data information transmitted between the modules. During the training process of the constructed training model using the training set and the support set, the training set and the support set are jointly used as the input of the model and are iteratively trained multiple times until convergence. Among them, the training set includes multiple base classes, and each base class has multiple samples with base class labels (the samples in the training set are also called base class samples). The base class labels include the class label of the base class and the object detection box position label. The support set includes multiple new classes different from the base class categories, and each new class has samples that meet the requirements of the few-shot task and have new class labels (the samples in the support set are also called new class samples). The new class labels include the class label of the new class, the object detection box position label, and the background label. According to an embodiment of the present invention, each iterative training in the present invention includes: S21. Use the initialized object detection basic model to extract the original regions of interest of each sample in the training set and the support set respectively; S22. Use the region of interest adjustment module to adjust the original regions of interest of each sample in the training set and the support set extracted based on step S21; S23. Use the feature extraction network to extract the original features of the adjusted regions of interest of each sample in the training set and the support set after being adjusted in step S22 (attached Figure 1 The original features of the support set samples in are a set composed of the original features of the adjusted regions of interest of all samples in the support set; the original features of the training set samples are a set composed of the original features of the adjusted regions of interest of all samples in the training set); S24. Use the relationship attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set (attached Figure 1The enhanced support set features in it are the set composed of the enhanced features of all samples in the support set); S25. Use the detection box position regression adjustment module to output the predicted target detection box position of the sample based on the original features of each sample in the support set, and calculate the target detection box position regression loss based on the target detection box position labels and the predicted target detection box positions of all samples in the support set; S26. Use the class classification module to perform class classification on the sample based on the enhanced features of each sample in the support set to obtain the predicted classification result, and calculate the class classification loss based on the predicted classification results and the class labels of all samples in the support set; S27. Use the target detection box position regression loss and the class classification loss to update the parameters of the training model. According to an embodiment of the present invention, it further includes a background classification module to enhance the background perception ability of the final small-sample object detection model and improve the recognition ability of small-sample classes. According to an embodiment of the present invention, in each iterative training, on the basis of steps S21 to S26, each iterative training further includes: S26'. Use the background classification module to output the background classification prediction result of the sample based on the original features of each sample in the support set, and calculate the background classification loss based on the background classification prediction results and the background labels of all samples in the support set; S27'. Use the target detection box position regression loss, the background classification loss, and the class classification loss to update the parameters of the training model.

[0044] Since the steps described in steps S21 to S23 are methods known to those skilled in the art, the present invention will not be described in detail. The following will explain other steps in detail to better understand the present invention.

[0045] In step S24, use the relational attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set. According to an embodiment of the present invention, the original features of each sample in the support set are enhanced through the following steps: S241. Calculate the attention matrix of the sample in the support set for all samples in the training set based on the features of all samples in the training set extracted in step S21 and the features of the current sample in the support set; S242. Enhance the original features of the sample through the following formula based on the attention matrix and features of the current sample obtained in step S241:

[0046] F_qknew = F_qk + Rk

[0047] where F_qknew is the enhanced feature of the k-th sample in the support set, F_qk is the original feature of the k-th sample in the support set, and Rk is the attention matrix of the k-th sample in the support set for all samples in the training set.

[0048] According to an embodiment of the present invention, the present invention uses the following formula to calculate the attention matrix of each sample in the support set for all samples in the training set:

[0049] Rk = softmax(Uk) * F_s = {softmax(Ukv) * F_sv}

[0050] Uk = {Ukv}

[0051] F_s = {F_sv}

[0052]

[0053]

[0054] Wherein, Rk is the attention matrix of the k-th sample in the support set for all samples in the training set, Uk is the similarity matrix of the k-th sample in the support set for all samples in the training set, softmax(Uk) is to normalize each element in the similarity matrix respectively, F_s is the set of features of all samples in the training set, F_sv is the feature of the v-th sample in the training set, F_qk is the feature of the k-th sample in the support set, Ukv is the v-th element in Uk, Uki is the similarity between the k-th sample in the support set and the i-th base class sample, and n is the number of all base class samples in the training set.

[0055] In step S25, the detection box position regression adjustment module is used to output the predicted target detection box position of the sample based on the original features of each sample in the support set, and calculate the target detection box position regression loss based on the target detection box position labels and the predicted target detection box positions of all samples in the support set. The target detection box position regression process is an independent regression branch task, which is used to perform target position regression processing based on the original features of the support set to achieve the purpose of more accurate target position localization. According to an embodiment of the present invention, the following formula is used to calculate the regression loss:

[0056]

[0057] Wherein, bi jx is the x-axis coordinate of the j-th vertex in the predicted target detection box position after the original features of the i-th sample in the support set are adjusted by the detection box position regression adjustment module, and bi jy is the y-axis coordinate in the predicted target detection box position after the original features of the i-th sample in the support set are adjusted by the detection box position regression adjustment module, Di jx is the x-axis coordinate of the j-th vertex in the target detection box position label of the i-th sample in the support set, and Di jy is the y-axis coordinate of the j-th vertex in the target detection box position label of the i-th sample in the support set.

[0058] In step S26, the category classification module is used to perform category classification on the samples based on the enhanced features of each sample in the support set to obtain the predicted classification results, and the category classification loss is calculated based on the predicted classification results of all samples in the support set and the category labels. According to an embodiment of the present invention, the category classification loss of the support set is calculated using the following formula:

[0059] LF = objlabel * log(Softmax(Kc)) = objlabel * log(Softmax({Kc i}));

[0060] where objlabel is the category label of the new class, and Kc i represents the probability that all samples of each category in the support set are classified as the i-th class, and Softmax(Kc_i) is the normalized classification probability that all samples of each category in the support set are classified as the i-th class.

[0061] Among them, in order to improve the classification accuracy, in the classification task, an additional background classification task is introduced to use the background classification module to perform background classification prediction on the samples in the support set based on their original features, and the background classification loss is calculated based on the background classification prediction results of all new class samples. According to an embodiment of the present invention, the background classification loss of the support set is calculated using the following formula:

[0062]

[0063] where bglabel is the background label of the new class, K_fb represents the set of original features of all samples in the support set input to the background classification module, and sigmoid(K_fb) is the background classification prediction result corresponding to the feature set K_fb.

[0064] In step S27, the parameters of the training model are updated using the target detection box position regression loss and the category classification loss. Among them, in the case where the background classification task is introduced, the parameters of the training model are updated using the target detection box position regression loss, the background classification loss, and the category classification loss. According to an embodiment of the present invention, the parameters of the detection box position regression adjustment module are updated using the target detection box position regression loss, and the parameters of the category classification module, the feature extraction network, and the region of interest adjustment module are updated using the category classification loss and the background classification loss.

[0065] To better illustrate the technical effects of the present invention, the following experiments are carried out for verification.

[0066] During the experiment, first, the final small-sample object detection model is obtained through a training method for a small-sample object detection model provided by the present invention. Then, a comparison test evaluation is carried out on the final small-sample object detection model and the object detection model trained by the existing method by selecting an experimental dataset.

[0067] First, select the commonly used object detection image dataset pascal voc, divide a part of the categories (for example, 10 categories) as the training set S, and take a small number of samples (the number of small samples is set according to the required number of small samples) from each of the other part of the categories (for example, 2 categories) as the support set Q. At the same time, other samples with the same categories as those in the support set Q will be used as the validation set V. Secondly, input the training set S into the trained two-stage model (preferably a two-stage model pre-trained by the Faster RCNN network in this experiment) to obtain an initialized object detection basic model and a region of interest adjustment module. In this experiment, the Faster RCNN network was trained for 15 rounds with the training set S to obtain an initialized object detection basic model and a region of interest adjustment module. The initial learning rate from the 1st round to the 10th round was 0.02, and the learning rate decayed to 0.1 of its previous round learning rate in the 11th round and the 13th round respectively. Input the training set S and the support set Q into the training model (such as Figure 2 the corresponding training model in it) and train the training model to convergence by using the method corresponding to the training method of the small-sample object detection model provided by the present invention. In this experiment, the specific data for convergence are: training for 300 rounds reached convergence, the initial learning rate was 0.005, and the learning rate decayed to 0.1 of the previous one in the 250th round and the 275th round. Finally, select the small-sample object detection model in the converged training model as the final small-sample object detection model.

[0068] Then, the final few-shot object detection model and the object detection model trained by the existing method are tested as follows: Randomly select images from the image dataset pascal voc and input them into the final few-shot object detection model to obtain the object location and object classification of the image. If there is no object in the image, the output is the background class. Specifically, in the commonly used object detection image dataset pascal voc, randomly select 15 classes of samples (set by the official) as the base class samples and 5 classes of samples (set by the official) as the new class samples to form an experimental dataset together, and conduct an evaluation test on the detection results of the final few-shot object detection model. The evaluation test uses the commonly used test metric mAP (mean Average Precision, that is, the average correct rate of all classes) to obtain the test results. In this experiment, seven groups of control experiments are selected to calculate the base class average precision and the new class average precision. The specific calculation results are shown in Table 1. Among them, the first group of experiments (represented by "3 new class samples" in Table 1) selects all classes in the experimental dataset for testing, and takes 3 samples from each class to obtain data and calculate the base class average precision and the new class average precision; the second group of experiments (represented by "10 new class samples" in Table 1) selects all classes in the experimental dataset for testing, and takes 10 samples different from the first group from each class to obtain data and calculate the base class average precision and the new class average precision. In this experiment, five groups of control experiments are also used to calculate the new class average precision. The specific calculation results are shown in Table 2. Among them, the third group of experiments selects all new classes in the experimental dataset for testing, and takes 1 sample from each class to obtain data and calculate the new class average precision; the fourth group of experiments selects all new classes in the experimental dataset for testing, and takes 2 samples from each class to obtain data and calculate the new class average precision. The fifth group of experiments selects all new classes in the experimental dataset for testing, and takes 3 samples from each class to obtain data and calculate the new class average precision. The sixth group of experiments selects all new classes in the experimental dataset for testing, and takes 5 samples from each class to obtain data and calculate the new class average precision. The seventh group of experiments selects all new classes in the experimental dataset for testing, and takes 10 samples from each class to obtain data and calculate the new class average precision. The samples in each group are not the same as the samples used in the other groups. In Table 2, the columns corresponding to 1, 2, 3, 5, and 10 represent the third group of experiments, the fourth group of experiments, the fifth group of experiments, the sixth group of experiments, and the seventh group of experiments in turn; in the column "Method" in Table 2, each row represents an object detection model trained by a method. It should be noted that the calculation process of the average precision is well-known in the art and will not be elaborated here; in Table 1 and Table 2, FRTN represents the final few-shot object detection model obtained by the training method provided by the present invention, and the other English abbreviations, such as LSTD, Meta-YOLO, etc., represent the models trained by the corresponding existing methods, which will not be elaborated here too much.

[0069] As can be seen from Table 1, the average precision of the base classes of the final few-shot object detection model obtained by the present invention in the first group of experiments is 75.0, and the average precision of the new classes is 61.4; the average precision of the base classes of the final few-shot object detection model obtained by the present invention in the second group of experiments is 79.5, and the average precision of the new classes is 71.3. This shows that the final few-shot object detection model obtained by the training method of the few-shot object detection model provided by the present invention can maintain the average precision of the base classes, that is, reduce the problem of learning forgetting of the base classes (learning forgetting refers to the decline in the classification and detection ability of the base classes during the learning process).

[0070] Table 1

[0071]

[0072] Table 2

[0073]

[0074] As can be seen from Table 2, the average precision of the new classes of the final few-shot object detection model obtained by the present invention in the seventh group of experiments is 71.3, which is greater than the average precision of the new classes of the models obtained by other existing methods. This shows that the training method of the few-shot object detection model provided by the present invention improves the object localization ability of the final few-shot object detection model obtained by the method of the present invention through the improvement of the classification accuracy of the new class detection.

[0075] In summary, for the detection problem of few-shot classes (such as object, animal face detection, etc.), the present invention introduces a detection box position regression adjustment module as a regression branch to predict the position of the object detection box, and calculates the object detection box position regression loss of all samples in the support set based on the object detection box position labels of all samples in the support set and the predicted object detection box position. Then, based on the regression loss, the parameters of the detection box position regression adjustment module are adjusted, thereby improving the object localization ability in the few-shot object detection task. The present invention also introduces a relational attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set. The class classification module performs class classification prediction based on the enhanced features of each sample in the support set, and calculates the class classification loss of the support set based on the predicted classification and class labels of all samples in the support set. Then, based on the class classification loss, the parameters of the training model are adjusted, so that the final few-shot object detection model can have the ability to recognize few-shot classes while retaining the learning knowledge of the base classes, solving the problem of learning forgetting of the base classes in the existing model and improving the perception ability of the final few-shot object detection model for the background, thereby improving the recognition ability of the model for few-shot class objects.

[0076] It should be noted that although the above steps are described in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required functions can be achieved.

[0077] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0078] The computer-readable storage medium may be a tangible device that retains and stores instructions for use by an instruction execution device. The computer-readable storage medium may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in a groove having instructions stored thereon, and any suitable combination of the foregoing.

[0079] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art in the field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skill in the art in the field to understand the embodiments disclosed herein.

Claims

1. A training method for a small-sample object detection model based on feature relationship migration, characterized in that The method includes: S1. Obtain a training set and a support set, and use the training set to train a pre-trained two-stage model to obtain an initialized object detection base model and a region of interest adjustment module; wherein, the training set includes multiple base classes, each base class has multiple samples with base class labels, and the base class labels include the class label of the base class and the target detection box position label; the support set includes multiple new classes different from the base class categories, each new class has samples that meet the requirements of the few-shot task and have new class labels, and the new class labels include the class label of the new class, the target detection box position label, and the background label; S2. Use the training set and the support set to perform multiple iterative trainings on the training model composed of the initialized object detection base model, the initialized region of interest adjustment module, the feature extraction network, the relational attention module, the class classification module, and the detection box position regression adjustment module until convergence to obtain a final few-shot object detection model composed of the object detection base model, the region of interest adjustment module, the feature extraction network, the detection box position regression adjustment module, and the class classification module, where each iterative training includes: S21. Use the initialized object detection base model to extract the original regions of interest of each sample in the training set and the support set respectively; S22. Use the region of interest adjustment module to adjust the original regions of interest of each sample in the training set and the support set extracted based on step S21; S23. Use the feature extraction network to extract the original features of the adjusted regions of interest of each sample in the training set and the support set after being adjusted in step S22; S24. Use the relational attention module to enhance the original features of each sample in the support set based on the original features of each sample in the support set and the original features of each sample in the training set to obtain the enhanced features of each sample in the support set; S25. Use the detection box position regression adjustment module to output the predicted target detection box position of the sample based on the original features of each sample in the support set, and calculate the target detection box position regression loss based on the target detection box position labels and the predicted target detection box positions of all samples in the support set; S26. Use the class classification module to perform class classification on the sample based on the enhanced features of each sample in the support set to obtain the predicted classification result, and calculate the class classification loss based on the predicted classification results and the class labels of all samples in the support set; S27. Use the target detection box position regression loss and the class classification loss to update the parameters of the training model.

2. The method according to claim 1, wherein the training model further comprises a background classification module, characterized in that, Each iterative training further includes: S26'. Use the background classification module to output the background classification prediction result of the sample based on the original features of each sample in the support set, and calculate the background classification loss based on the background classification prediction results and the background labels of all samples in the support set; S27'. Use the target detection box position regression loss, the background classification loss, and the class classification loss to update the parameters of the training model.

3. The method according to claim 2, characterized in that Use the target detection box position regression loss to update the parameters of the detection box position regression adjustment module; use the class classification loss and the background classification loss to update the parameters of the class classification module, the feature extraction network, and the region of interest adjustment module.

4. The method according to claim 3, characterized in that The pre-trained two-stage model is any one of the following networks: RCNN, FastRCNN, FasterRCNN, SPPNet, FPN.

5. The method according to claim 4, characterized in that In step S24, the original features of each sample in the support set are enhanced through the following steps: S241. Calculate the attention matrix of the k-th sample in the support set for all samples in the training set based on the features of all samples in the training set extracted in step S21 and the features of the current sample in the support set; S242. Enhance the original features of the sample through the following formula based on the attention matrix and features of the current sample obtained in step S241: F_qknew = F_qk + Rk where F_qknew is the enhanced feature of the k-th sample in the support set, F_qk is the original feature of the k-th sample in the support set, and Rk is the attention matrix of the k-th sample in the support set for all samples in the training set.

6. The method according to claim 5, characterized in that, The following formula is used to calculate the attention matrix of each sample in the support set for all samples in the training set: Rk = softmax(Uk) * F_s = {softmax(Ukv) * F_sv} Uk = {Ukv} F_s = {F_sv} where Rk is the attention matrix of the k-th sample in the support set for all samples in the training set, Uk is the similarity matrix of the k-th sample in the support set for all samples in the training set, softmax(Uk) normalizes each element in the similarity matrix respectively, F_s is the set of features of all samples in the training set, F_sv is the feature of the v-th sample in the training set, F_qk is the feature of the k-th sample in the support set, Ukv is the v-th element in Uk, Uki is the similarity between the k-th sample in the support set and the i-th base class sample, and n is the number of all base class samples in the training set.

7. The method according to claim 6, characterized in that, The following formula is used to calculate the background classification loss of the support set: LQ = bglabel * log(sigmoid(K_fb)); where bglabel is the background label of the new class, K_fb represents the set of original features of all samples in the support set input to the background classification module, and sigmoid(K_fb) is the background classification result corresponding to the feature set K_fb.

8. The method according to claim 7, wherein The following formula is used to calculate the class classification loss of the support set: LF = objlabel * log(Softmax(Kc)) = objlabel * log(Softmax({Kc i})); where objlabel is the class label of the new class, Kc i represents the probability that all samples of each class in the support set are classified as the i-th class, and Softmax(Kc_i) is the normalized classification probability that all samples of each class in the support set are classified as the i-th class.

9. The method according to claim 3, characterized in that, In step S25, the following formula is used to calculate the regression loss: where, bi jx is the x-axis coordinate of the j-th vertex in the predicted target detection box position adjusted by the detection box position regression adjustment module for the original feature of the i-th sample in the support set, and bi jy is the y-axis coordinate in the predicted target detection box position adjusted by the detection box position regression adjustment module for the original feature of the i-th sample in the support set, and Di jx is the x-axis coordinate of the j-th vertex in the target detection box position label of the i-th sample in the support set, and Di jy is the y-axis coordinate of the j-th vertex in the target detection box position label of the i-th sample in the support set.

10. A target detection method, characterized in that, The detection method includes: F1. Obtain an input image; F2. Use the small sample object detection model obtained by the method according to any one of claims 1 to 9 to perform object detection on the input image obtained in step F1 to obtain the object position and object classification in the image.

11. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 9, 10.

12. An electronic device, characterized in that, Including: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the electronic device to implement the steps of the method according to any one of claims 1 to 9, 10.

Images