Two-stage generalized small sample target detection method and system based on self-supervision

Abstract

The invention provides a two-stage generalized small sample target detection method and system based on self-supervision, and relates to the technical field of deep learning, and the method comprises the steps: defining a potential knowledge container to store all potential knowledge, and defining a prototype container to store prototypes of various candidate boxes; dividing the training data set into a base class set and a new class set; the self-supervised category domain carries out pre-training and fine tuning on the detection model based on the potential knowledge of the base category set and the new category set; the self-supervised detection frame field carries out pre-training and fine tuning on the detection model based on the potential knowledge of the base class set and the new class set; and inputting a to-be-detected image to the detection model, obtaining features through the feature extractor, performing potential knowledge search and segmentation on the to-be-detected image based on the potential knowledge container to obtain a relationship between the to-be-detected image and each potential knowledge, and obtaining a candidate box category. According to the method, detection model training is carried out in the category domain and the detection frame domain by using self-supervised learning based on potential knowledge, and the method has good target detection capability for both a base class and a new class.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a two-stage generalized small sample target detection method and system based on self-supervision. Background Art

[0002] Object detection has long been one of the core tasks in the field of computer vision. With the emergence of neural networks, conventional deep learning-based object detection tasks have been fully developed and have achieved remarkable results. The premise for them to achieve good performance is to use a large amount of labeled data, but many categories in real life do not have enough data. This situation leads to unsatisfactory performance of conventional object detection in tasks with extremely uneven data distribution. Therefore, many scholars have conducted research on this issue.

[0003] As a method of building accurate machine learning models with a small number of training sets, the task of few-shot object detection (FSOD) has attracted widespread attention in current research due to its good application prospects. The related methods are mainly divided into meta-learning-based methods and fine-tuning-based methods. The meta-learning-based method quickly adapts to new tasks in the case of few samples by constructing meta-tasks, while the fine-tuning-based method first pre-trains on a base class set with a large number of samples, then freezes the feature extraction parameters and uses a small number of samples for fine-tuning. However, in the task of few-shot object detection (GFSOD) for detecting both seen and unseen classes, these two types of methods only show excellent performance on new classes, while performance on base classes degrades. The limitation is that they do not take into account the emergence of new potential knowledge as new classes are learned, and they also do not take into account the problem that the overall performance of the model will decline due to the forgetting of base classes as new classes are learned. Summary of the invention

[0004] In view of the above problems, the present invention provides a two-stage generalized small sample target detection method and system based on self-supervision, which aims to overcome the problem of forgetting the base class when learning a new class in the generalized small sample learning scenario, and effectively solve the problem of difficult target detection in the scenario of rare cases. By using potential knowledge and self-supervised learning to preserve the knowledge learned from the base class as much as possible, the knowledge of the base class can be well preserved after learning the knowledge of the new class, so as to achieve good target detection capabilities for both the base class and the new class.

[0005] To achieve the above object, the present invention provides a two-stage generalized small sample target detection method based on self-supervision, comprising:

[0006] Constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0007] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0008] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0009] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base category set and the new category set respectively;

[0010] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain a final detection model;

[0011] Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected according to the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0012] Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

[0013] As a further improvement of the present invention, the two-stage detection model adopts a two-stage Faster RCNN network model.

[0014] As a further improvement of the present invention, the self-supervised category domain pre-trains the two-stage detection model based on the potential knowledge of the base class set, including:

[0015] Performing color transformation and translation transformation on the training samples in the base class set respectively, to obtain two groups of base class expansion samples, the two groups of base class expansion samples correspond to each other one by one;

[0016] For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain ROI features one by one;

[0017] After the ROI feature is reduced in dimension by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container to calculate the cosine similarity score with each potential knowledge, and the potential knowledge container stores all the potential knowledge of the base class set;

[0018] The self-supervised task constrains the ROI features through the cosine similarity loss, and calculates the class probability constraint category through the prototype classifier module to obtain a pre-trained two-stage detection model.

[0019] As a further improvement of the present invention, the self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new class set, including: fine-tuning the pre-trained two-stage detection model based on the new class set;

[0020] Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one;

[0021] For each pair of corresponding samples in the two groups of new class expansion samples, the weight-sharing feature extractor is used to obtain ROI features one by one;

[0022] After the ROI feature is dimensionally reduced by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container to calculate the cosine similarity score with each potential knowledge, and the potential knowledge container stores all potential knowledge of the base class set and the new class set;

[0023] The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

[0024] As a further improvement of the present invention, the self-supervised detection box domain pre-trains the two-stage detection model based on the base class set, including:

[0025] Perform color transformation and translation transformation on the base class samples respectively to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one by one;

[0026] For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features;

[0027] The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0028] The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0029] The two sets of regression frames are used to extract corresponding frame features using a weight-shared embedding network, the two sets of frame features are compared and learned, and the candidate frames in the prototype container are momentum updated.

[0030] As a further improvement of the present invention, the self-supervised detection box domain fine-tunes the two-stage detection model based on the new class set to obtain a final detection model; including:

[0031] Perform color transformation and translation transformation on the new class set samples respectively, and obtain two groups of new class expansion samples. The two groups of base class expansion samples correspond one to one.

[0032] For each pair of corresponding samples in the two groups of new class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features;

[0033] The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0034] The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0035] The two sets of regression frames are used to extract corresponding frame features using a weight-shared embedding network, the two sets of frame features are compared and learned, and the candidate frames in the prototype container are momentum updated.

[0036] As a further improvement of the present invention, the potential knowledge container is defined to store all potential knowledge, and the prototype container is defined to store the prototypes of various candidate boxes to be detected; including:

[0037] Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge;

[0038] Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate box, and M represents the number of potential knowledge corresponding to the candidate box;

[0039] When the two-stage detection model is pre-trained based on the base class set in the self-supervised category domain, C=|C base |+1; when the self-supervised category domain fine-tunes the two-stage detection model based on the new category set, C=|C base |+|C novel |+1, C base Represents the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

[0040] As a further improvement of the present invention, the image to be detected is input into the final detection model, ROI features are obtained by a feature extractor, potential knowledge search is performed on the image to be detected based on the potential knowledge container after dimensionality reduction by a fully connected layer, and features of the image to be detected are segmented according to the searched potential knowledge to obtain the relationship between the features of the image to be detected and each potential knowledge; including:

[0041] The ROI feature is reduced in dimension by a fully connected layer to be consistent with the potential knowledge dimension in the potential knowledge container, thereby obtaining a feature;

[0042] The similarity score between the reduced-dimensional feature and each potential knowledge in the potential knowledge container is calculated by cosine similarity.

[0043] As a further improvement of the present invention, the image to be detected is classified by the prototype classifier module based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, to obtain the target detection result; including:

[0044] Perform cosine similarity judgment between the calculated reduced-dimensional features and the similarity scores between each potential knowledge container and the prototypes of each candidate box in the prototype container;

[0045] After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

[0046] The present invention also provides a two-stage generalized small sample target detection system based on self-supervision, including: a potential knowledge classifier construction module, a self-supervised category domain pre-training and fine-tuning module, a self-supervised detection frame domain pre-training and fine-tuning module, an image detection module and a detection result acquisition module;

[0047] The latent knowledge classifier building module is used to:

[0048] Constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0049] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0050] The self-supervised category domain pre-training and fine-tuning module is used to:

[0051] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0052] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively;

[0053] The self-supervised detection box domain pre-training and fine-tuning module is used to:

[0054] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain a final detection model;

[0055] The image detection module is used to:

[0056] Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected according to the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0057] The detection result acquisition module is used to:

[0058] Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] To address the problem that connections cannot be established between candidate box categories in generalized small sample target detection, the present invention proposes the use of latent knowledge. Through feature extraction, similar candidate classes are given similar latent knowledge representations, thereby constructing connections between candidate box categories and categories through latent knowledge. Therefore, when learning new classes after model pre-training, the knowledge of base classes can be connected with the new classes, thereby achieving accurate target detection of the new classes.

[0061] To address the problem of base class forgetting in generalized small sample target detection, the present invention proposes a self-supervised learning method to provide more constraints for latent feature construction, which to some extent alleviates the negative impact of model fine-tuning and potential knowledge in new classes on the base class, thereby improving the overall performance of the detection model.

[0062] The present invention uses self-supervised learning for training in the category domain and the detection box domain respectively, so as to alleviate the decline in the performance of the model base class; first, self-supervised learning is used in base class training to provide more constraints and perspectives for the potential feature construction module, and the problem of training collapse is avoided by stopping the gradient backpropagation. Then, self-supervised learning is used in new class training to constrain the features learned by the model in base class training by constructing proxy tasks. With the help of self-supervised learning, the method proposed in the present invention can ensure to the greatest extent that the knowledge learned on the base class is not forgotten, so that the model has good feature extraction capabilities for both the base class and the new class, and then has good target detection capabilities for both the base class and the new class. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A general architecture diagram of a two-stage generalized small sample target detection method based on self-supervision disclosed in an embodiment of the present invention;

[0064] Figure 2 A model framework diagram of a detection model base class training phase disclosed in an embodiment of the present invention;

[0065] Figure 3 A model framework diagram of a new class training phase of a detection model disclosed in an embodiment of the present invention;

[0066] Figure 4 A model framework diagram of self-supervised detection box domain training disclosed in an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of a two-stage generalized small sample target detection system based on self-supervision disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0070] like Figure 1As shown, the present invention uses potential knowledge and self-supervised learning to avoid forgetting the knowledge of the base class during the detection model training process, thereby better improving the performance of generalized small sample target detection. Among them, in the pre-training stage of the detection model, the present invention uses potential features to define each category, and describes each category in the form of attribute groups; the process of constructing potential features is relatively complicated, and the present invention introduces self-supervised learning as an auxiliary task to improve the efficiency of constructing potential features. In the specific process, it includes color transformation and translation cutting of the training set, expanding the data set and creating self-supervised proxy tasks, enhancing the training constraints of the model, and preventing training collapse by stopping gradient backpropagation. In the fine-tuning stage of the detection model, the number of features in the potential feature container is first increased so that the newly added features can capture the potential knowledge in the new category. However, the increase of potential features may cause catastrophic forgetting of the base class and affect the overall performance of the detection model; for this reason, self-supervised constraints are used to retain the representative potential features learned in the pre-training stage, and these features are further optimized in the process of learning new categories through dynamic update strategies. This method effectively reduces the negative impact of model fine-tuning and new potential features on the base class, thereby improving the overall performance of the model.

[0071] Specifically, the present invention provides a two-stage generalized small sample target detection method based on self-supervision, comprising:

[0072] S1, constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0073] in,

[0074] As a way to connect classes, attributes have some problems. First, it is difficult to design a complete set of discriminative attributes for the object detection task by manually annotating attributes, because it is difficult to reflect all human-defined attributes through images, and it is also difficult to determine whether the annotated attributes are sufficient to meet the detection task; second, manually annotating attributes requires a lot of manpower and material resources.

[0075] The present invention considers that there is latent knowledge in feature space that can be used to segment and classify features, just as attributes in visual space can segment and classify classes. Unlike attributes, latent knowledge is learned autonomously and does not need to be labeled separately. This method can avoid the difficulty and cost of manually labeling attributes and improve the classification effect.

[0076] To this end, the present invention proposes a latent knowledge classifier (LK-Classifier), which is divided into a latent feature module and a prototype classifier module, and aims to classify candidate boxes using latent knowledge.

[0077] S2. Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0078] Among them, include:

[0079] Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge;

[0080] Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate box, and M represents the number of potential knowledge corresponding to the candidate box;

[0081] When the two-stage detection model is pre-trained on the base class set in the self-supervised category domain, C = |C base |+1; when fine-tuning the two-stage detection model based on the new set of categories in the self-supervised category domain, C=|C base |+|C novel |+1, C base Represents the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

[0082] S3, divide the training data set into a base class set with a large number of samples and a new class set with a small number of samples;

[0083] in,

[0084] In the pre-training phase, the two-stage detection model is pre-trained using only the base class set, and in the fine-tuning phase the model is fine-tuned using all classes.

[0085] S4, the self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the latent knowledge of the base category set and the new category set respectively;

[0086] in,

[0087] The two-stage detection model adopts a two-stage Faster RCNN network model, and its network structure ultimately consists of five parts: feature extractor, regressor, latent feature module, prototype classifier module, and self-supervised auxiliary branch module.

[0088] Furthermore,

[0089] The self-supervised category domain pre-trains a two-stage detection model based on the latent knowledge of the base category set, including:

[0090] Perform color transformation and translation transformation on the training samples in the base class set to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one by one;

[0091] For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain ROI features one by one;

[0092] After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all the potential knowledge of the base class set.

[0093] The self-supervised task constrains the ROI features through cosine similarity loss, and calculates the class probability constraint category through the prototype classifier module to obtain the pre-trained two-stage detection model.

[0094] The self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new category set, including: fine-tuning the pre-trained two-stage detection model based on the new category set;

[0095] Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one;

[0096] For each pair of corresponding samples in the two groups of new class expansion samples, the ROI features are obtained by using the weight-sharing feature extractor one by one;

[0097] After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all the potential knowledge of the base class set and the new class set.

[0098] The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

[0099] Specifically,

[0100] like Figure 2 As shown in the figure, during the pre-training stage of the detection model, the training samples in the base class set are first color-transformed and translated to expand the data scale. Then, the two expanded samples are passed through a weight-shared feature extractor to obtain the ROI pooled features, and the similarity score is calculated through latent feature search after the fully connected layer FC. Then, the self-supervised task constrains the latent features by using the cosine similarity loss to help better extract visual features. Finally, the classification task calculates the class probability constraint category through the prototype classifier module.

[0101] The loss function of the self-supervised category domain in the pre-training stage is as follows:

[0102]

[0103] like Figure 3 As shown in the figure, the detection model is fine-tuned for the new category. The color transformation and translation of the image are consistent with the pre-training. The latent feature search module increases the number of features of the new category and the latent features learned in the base class stage in the self-supervision module constrain the new class training. Self-supervision constraints are used to prevent the model from forgetting the base class. The notification of the base class latent features in the self-supervision module is slowly updated through gradient updates, and the feature learning module in the new class training branch is optimized. Improve the overall classification accuracy.

[0104] The latent feature consistency loss formula is as follows:

[0105]

[0106] The loss of the self-supervised category domain in the fine-tuning stage is as follows:

[0107]

[0108] S5, the self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain the final detection model;

[0109] in,

[0110] The self-supervised detection box domain pre-trains the two-stage detection model based on the base class set, including:

[0111] Perform color transformation and translation transformation on the base class samples respectively to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one by one;

[0112] For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features;

[0113] The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0114] The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0115] The two sets of regression boxes are used to extract the corresponding box features using a weight-shared embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0116] The self-supervised detection box domain fine-tunes the two-stage detection model based on the new set of classes to obtain the final detection model; including:

[0117] Perform color transformation and translation transformation on the new class set samples respectively, and obtain two groups of new class expansion samples. The two groups of base class expansion samples correspond one to one.

[0118] For each pair of corresponding samples in the two groups of new class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features;

[0119] The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0120] The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0121] The two sets of regression boxes are used to extract the corresponding box features using a weight-shared embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0122] Specifically,

[0123] The pre-training process of the self-supervised detection box domain module and the subsequent fine-tuning process are basically the same.

[0124] like Figure 4 As shown in the figure, firstly, the sample image X is subjected to color transformation and translation transformation to obtain enhanced sample images X1 and X2. Then, the enhanced sample image is passed through the weight-sharing feature extractor to obtain the ROI pooling feature, and the corresponding prediction box is obtained through the fully connected layer FC and the detection head composed of the fully connected layer and ReLU. Finally, the real detection box label P corresponding to the translation transformation is obtained. i For comparison, the target allocation constraint operation of sample matching is performed. For the detection box without target, the corresponding background label will be given.

[0125] At the same time, the features of the two sets of predicted box selections are mapped to another visual space through a weight-shared embedding network, and then the features are updated with momentum through contrastive learning. The regression head of the box selection features follows the architecture in the MOCO v2 network, that is, a 2-layer multi-layer perception MLP with ReLU as the regression head. Contrastive learning ensures that predictions for the same target should have similar features, while predictions for different targets should have greater differences. In order to avoid gradient collapse, this paper uses momentum update to ensure fast and efficient training. The contrastive loss is as follows:

[0126]

[0127] Where τ is a temperature hyperparameter that controls the difficulty of the contrastive learning task, and is set to 0.5 in the present invention.

[0128] S6, input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction by the fully connected layer, segment the features of the image to be detected according to the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0129] Among them, include:

[0130] The ROI features are reduced in dimension by the fully connected layer to be consistent with the potential knowledge dimension in the potential knowledge container to obtain the features;

[0131] The similarity score between the reduced-dimensional features and each potential knowledge in the potential knowledge container is calculated by cosine similarity.

[0132] Specifically,

[0133] When an input image x is given, first, the feature extractor is used to obtain the ROI feature F(x)∈R Z×D , where Z and D are the number and dimension of ROI features respectively. Then, the feature F(x) is reduced in dimension, because the dimension should be consistent with the dimension of potential knowledge, and too large a dimension of potential knowledge will make it difficult to learn potential knowledge. Therefore, the present invention uses a fully connected layer to reduce the dimension of F(x) to obtain G(x)∈R Z×m :

[0134] G(x) i =W c *F(x) i +b c

[0135] Where W c and b c is the parameter of the fully connected layer. Then for the feature G(x) i Perform potential knowledge search and identify its (feature G(x)) according to the potential knowledge i ) is segmented to obtain its (feature G(x) i ) and the relationship between each potential knowledge.

[0136] The present invention uses cosine similarity to calculate G(x) i With each potential knowledge Q j The similarity score U between i = {U i,1 ,U i,2 ,…,U i,N ,}。 i,j represents G(x) i With Qj The similarity score between .

[0137]

[0138] S7. Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified through the prototype classifier module to obtain the target detection result.

[0139] Among them, include:

[0140] The calculated reduced-dimensional features are used to determine the similarity scores between the potential knowledge containers and the prototypes of the candidate boxes in the prototype container using cosine similarity.

[0141] After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

[0142] Specifically,

[0143] Comprehensively consider the true probability distribution of all potential knowledge of each class and U i The actual probability distribution in U is obtained by using the prototype classifier i Specifically, U i And all class prototypes P are repeated again

[0144] Chord similarity judgment, the result is passed through a softmax layer to obtain the final category probability, as shown in the following formula:

[0145]

[0146] S i,k Denote G(x) i is the probability of the kth class, α and β are the scaling factors for amplifying the gradient, and the present invention sets them both to 20 in the experiment.

[0147] like Figure 5 As shown, the present invention also provides a two-stage generalized small sample target detection system based on self-supervision, including: a potential knowledge classifier construction module, a self-supervised category domain pre-training and fine-tuning module, a self-supervised detection box domain pre-training and fine-tuning module, an image detection module and a detection result acquisition module;

[0148] Latent knowledge classifier building blocks for:

[0149] Constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0150] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0151] Self-supervised category domain pre-training and fine-tuning module for:

[0152] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0153] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively;

[0154] Self-supervised detection box domain pre-training and fine-tuning module for:

[0155] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain the final detection model;

[0156] Image detection module for:

[0157] Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected based on the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0158] The test result acquisition module is used to:

[0159] Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the image to be detected is classified through the prototype classifier module to obtain the target detection result.

[0160] Advantages of the present invention:

[0161] To address the problem that connections cannot be established between candidate box categories in generalized small sample target detection, the present invention proposes the use of latent knowledge. Through feature extraction, similar candidate classes are given similar latent knowledge representations, thereby constructing connections between candidate box categories and categories through latent knowledge. Therefore, when learning new classes after model pre-training, the knowledge of base classes can be connected with the new classes, thereby achieving accurate target detection of the new classes.

[0162] To address the problem of base class forgetting in generalized small sample target detection, the present invention proposes a self-supervised learning method to provide more constraints for latent feature construction, which to some extent alleviates the negative impact of model fine-tuning and potential knowledge in new classes on the base class, thereby improving the overall performance of the detection model.

[0163] The present invention uses self-supervised learning for training in the category domain and the detection box domain respectively, so as to alleviate the decline in the performance of the model base class; first, self-supervised learning is used in base class training to provide more constraints and perspectives for the potential feature construction module, and the problem of training collapse is avoided by stopping the gradient backpropagation. Then, self-supervised learning is used in new class training to constrain the features learned by the model in base class training by constructing proxy tasks. With the help of self-supervised learning, the method proposed in the present invention can ensure to the greatest extent that the knowledge learned on the base class is not forgotten, so that the model has good feature extraction capabilities for both the base class and the new class, and then has good target detection capabilities for both the base class and the new class.

[0164] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A two-stage generalized small sample target detection method based on self-supervision, characterized in that: include: Constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module; Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected; The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples; The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base category set and the new category set respectively; The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain a final detection model; Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected according to the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge; Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

2. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The two-stage detection model adopts a two-stage Faster RCNN network model.

3. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised category domain pre-trains a two-stage detection model based on the latent knowledge of the base category set, including: Performing color transformation and translation transformation on the training samples in the base class set respectively, to obtain two groups of base class expansion samples, the two groups of base class expansion samples correspond to each other one by one; For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain ROI features one by one; After the ROI feature is reduced in dimension by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container to calculate the cosine similarity score with each potential knowledge, and the potential knowledge container stores all the potential knowledge of the base class set; The self-supervised task constrains the ROI features through the cosine similarity loss, and calculates the class probability constraint category through the prototype classifier module to obtain a pre-trained two-stage detection model.

4. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new class set, including: fine-tuning the pre-trained two-stage detection model based on the new class set; Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one; For each pair of corresponding samples in the two groups of new class expansion samples, the ROI features are obtained by using the weight-sharing feature extractor one by one; After the ROI feature is dimensionally reduced by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container to calculate the cosine similarity score with each potential knowledge, and the potential knowledge container stores all potential knowledge of the base class set and the new class set; The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

5. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised detection box domain pre-trains the two-stage detection model based on the base class set, including: Perform color transformation and translation transformation on the base class samples respectively to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one by one; For each pair of corresponding samples in the two groups of base class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features; The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames; The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching; The two sets of regression frames are used to extract corresponding frame features using a weight-shared embedding network, the two sets of frame features are compared and learned, and the candidate frames in the prototype container are momentum updated.

6. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised detection box domain fine-tunes the two-stage detection model based on the new class set to obtain a final detection model; include: Perform color transformation and translation transformation on the new class set samples respectively, and obtain two groups of new class expansion samples. The two groups of base class expansion samples correspond one to one. For each pair of corresponding samples in the two groups of new class expansion samples, a weight-sharing feature extractor is used to obtain two groups of ROI features; The two sets of ROI features are reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames; The two sets of regression frames are compared with the real detection frames after color transformation and the real detection frames after parallel transformation to perform target allocation constraints for sample matching; The two sets of regression frames are used to extract corresponding frame features using a weight-shared embedding network, the two sets of frame features are compared and learned, and the candidate frames in the prototype container are momentum updated.

7. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The potential knowledge container is defined to store all potential knowledge, and the prototype container is defined to store the prototypes of various candidate boxes to be detected; including: Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge; Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate box, and M represents the number of potential knowledge corresponding to the candidate box; When the two-stage detection model is pre-trained based on the base class set in the self-supervised category domain, C=|C base |+1; when the self-supervised category domain fine-tunes the two-stage detection model based on the new category set, C=|C base |+|C novel |+1, C base Represents the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

8. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The method comprises: inputting the image to be detected into the final detection model, obtaining ROI features through a feature extractor, performing potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through a fully connected layer, segmenting the features of the image to be detected according to the searched potential knowledge, and obtaining the relationship between the features of the image to be detected and each potential knowledge; comprising: The ROI feature is reduced in dimension by a fully connected layer to be consistent with the potential knowledge dimension in the potential knowledge container, thereby obtaining a feature; The similarity score between the reduced-dimensional feature and each potential knowledge in the potential knowledge container is calculated by cosine similarity.

9. The two-stage generalized small sample target detection method based on self-supervision according to claim 1, characterized in that: The method of classifying the image to be detected by the prototype classifier module based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, and obtaining the target detection result includes: Perform cosine similarity judgment between the calculated reduced-dimensional features and the similarity scores between each potential knowledge container and the prototypes of each candidate box in the prototype container; After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

10. A two-stage generalized small sample target detection system based on self-supervision, implementing the two-stage generalized small sample target detection method based on self-supervision as claimed in any one of claims 1 to 9, characterized in that: include: Latent knowledge classifier construction module, self-supervised category domain pre-training and fine-tuning module, self-supervised detection box domain pre-training and fine-tuning module, image detection module and detection result acquisition module; The latent knowledge classifier building module is used to: Constructing a latent knowledge classifier, including a latent feature module and a prototype classifier module; Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected; The self-supervised category domain pre-training and fine-tuning module is used to: The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples; The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively; The self-supervised detection box domain pre-training and fine-tuning module is used to: The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain a final detection model; The image detection module is used to: Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected according to the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge; The detection result acquisition module is used to: Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

Images