A two-stage generalized small sample target detection method and system based on self-supervision

Abstract

The application provides a two-stage generalized small sample target detection method and system based on self-supervision, relates to the technical field of deep learning, and comprises the following steps: defining a potential knowledge container to store all potential knowledge, and defining a prototype container to store prototypes of various candidate boxes; dividing a training data set into a base class set and a new class set; pre-training and fine-tuning a detection model based on the potential knowledge of the base class set and the new class set in a self-supervised category domain; pre-training and fine-tuning the detection model based on the potential knowledge of the base class set and the new class set in a self-supervised detection box domain; inputting a to-be-detected image into the detection model, obtaining features through a feature extractor, searching and segmenting the to-be-detected image based on the potential knowledge container, obtaining the relationship between each potential knowledge, and obtaining the category of the candidate box. The application uses self-supervised learning to train the detection model in the category domain and the detection box domain based on the potential knowledge, and has good target detection capability for the base class and the new class.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a two-stage generalized small sample target detection method and system based on self-supervision. Background Art

[0002] Object detection has long been a core task in computer vision. With the advent of neural networks, conventional deep learning-based object detection has rapidly developed and achieved remarkable results. However, achieving good performance requires the use of large amounts of labeled data, yet in reality, this data is insufficient for many categories. This situation results in unsatisfactory performance for conventional object detection in tasks with extremely uneven data distribution. Consequently, many researchers have conducted research on this issue.

[0003] The few-shot object detection (FSOD) task, as a method for building accurate machine learning models using a relatively small training set, has attracted widespread attention in current research due to its promising application prospects. Related methods are mainly categorized as meta-learning-based and fine-tuning-based. Meta-learning-based methods construct meta-tasks to enable rapid adaptation to new tasks with few samples, while fine-tuning-based methods first pre-train on a base set with a large number of samples, then freeze the feature extraction parameters and fine-tune using a small number of samples. However, in the GFSOD task of detecting both seen and unseen classes, both types of methods demonstrate excellent performance only on new classes, while performance degrades on base classes. Their limitations lie in their failure to account for the emergence of new latent knowledge as new classes are learned, and in their failure to consider the degradation of overall model performance due to forgetting about base classes as new classes are learned. Summary of the Invention

[0004] To address the above issues, the present invention provides a two-stage self-supervised generalized small-sample target detection method and system. This method aims to overcome the problem of forgetting the base class when learning a new class in generalized small-sample learning scenarios, and effectively address the difficulty of target detection in rare case scenarios. By leveraging latent knowledge and self-supervised learning to preserve as much knowledge learned from the base class as possible, the knowledge of the base class can be well preserved after learning the knowledge of the new class, achieving excellent target detection capabilities for both the base class and the new class.

[0005] To achieve the above objectives, the present invention provides a two-stage generalized small-sample target detection method based on self-supervision, comprising:

[0006] Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0007] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0008] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0009] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the latent knowledge of the base category set and the new category set respectively;

[0010] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain the final detection model;

[0011] Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected based on the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0012] Based on the true probability distribution of all potential knowledge of various candidate frames in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

[0013] As a further improvement of the present invention, the two-stage detection model adopts a two-stage Faster RCNN network model.

[0014] As a further improvement of the present invention, the self-supervised category domain pre-trains the two-stage detection model based on the latent knowledge of the base class set, including:

[0015] Performing color transformation and translation transformation on the training samples in the base class set respectively to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one to one;

[0016] For each pair of samples in the two groups of base class expansion samples, the ROI features are obtained using the weight-sharing feature extractor one by one;

[0017] After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set.

[0018] The self-supervised task constrains the ROI features through the cosine similarity loss, and calculates the class probability constraint category through the prototype classifier module to obtain a pre-trained two-stage detection model.

[0019] As a further improvement of the present invention, the self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new class set, including: fine-tuning the pre-trained two-stage detection model based on the new class set;

[0020] Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one;

[0021] For each pair of samples in the two groups of new class expansion samples, the ROI features are obtained using the weight-sharing feature extractor one by one;

[0022] After the ROI features are reduced in dimension by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container, and a cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set and the new class set.

[0023] The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

[0024] As a further improvement of the present invention, the self-supervised detection box domain pre-trains the two-stage detection model based on the base class set, including:

[0025] Perform color transformation and translation transformation on the base class samples to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond one to one;

[0026] For each pair of corresponding samples in the two groups of base class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor;

[0027] The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0028] The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0029] The two sets of regression boxes are used to extract corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0030] As a further improvement of the present invention, the self-supervised detection box domain fine-tunes the two-stage detection model based on the new set of classes to obtain a final detection model; including:

[0031] Perform color transformation and translation transformation on the new class set samples respectively to obtain two groups of new class expansion samples, and the two groups of base class expansion samples correspond one to one;

[0032] For each pair of samples in the two groups of new class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor;

[0033] The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0034] The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0035] The two sets of regression boxes are used to extract corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0036] As a further improvement of the present invention, the potential knowledge container is defined to store all potential knowledge, and the prototype container is defined to store the prototypes of various candidate boxes to be detected; including:

[0037] Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge;

[0038] Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate frame, and M represents the number of potential knowledge corresponding to the candidate frame;

[0039] When the two-stage detection model is pre-trained based on the base class set in the self-supervised category domain, C=|C base |+1; when the self-supervised classification domain fine-tunes the two-stage detection model based on the new class set, C=|C base |+|C novel |+1, C base Indicates the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

[0040] As a further improvement of the present invention, the method includes inputting an image to be detected into the final detection model, obtaining ROI features through a feature extractor, performing a potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through a fully connected layer, segmenting the features of the image to be detected based on the searched potential knowledge, and obtaining a relationship between the features of the image to be detected and each potential knowledge; including:

[0041] The ROI feature is reduced in dimension by a fully connected layer to be consistent with the potential knowledge dimension in the potential knowledge container to obtain a feature;

[0042] The similarity score between the reduced-dimensional feature and each potential knowledge in the potential knowledge container is calculated by cosine similarity.

[0043] As a further improvement of the present invention, the method of classifying the image to be detected by the prototype classifier module based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and the potential knowledge to obtain the target detection result includes:

[0044] Perform cosine similarity judgment on the similarity scores between the calculated dimensionality reduction features and each potential knowledge container and the prototypes of each candidate box in the prototype container;

[0045] After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

[0046] The present invention also provides a two-stage generalized small sample target detection system based on self-supervision, comprising: a potential knowledge classifier construction module, a self-supervised category domain pre-training and fine-tuning module, a self-supervised detection box domain pre-training and fine-tuning module, an image detection module, and a detection result acquisition module;

[0047] The latent knowledge classifier building module is used to:

[0048] Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0049] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0050] The self-supervised category domain pre-training and fine-tuning module is used to:

[0051] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0052] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively;

[0053] The self-supervised detection box domain pre-training and fine-tuning module is used to:

[0054] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain the final detection model;

[0055] The image detection module is used to:

[0056] Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected based on the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge;

[0057] The detection result acquisition module is used to:

[0058] Based on the true probability distribution of all potential knowledge of various candidate frames in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] To address the problem that connections cannot be established between candidate box categories in generalized small sample target detection, the present invention proposes the use of latent knowledge. Through feature extraction, similar candidate classes are given similar latent knowledge representations, thereby constructing connections between candidate box categories and categories through latent knowledge. Therefore, when learning new classes after model pre-training, the knowledge of the base class and the new class can be connected, thereby achieving accurate target detection of the new class.

[0061] To address the problem of base class forgetting in generalized small-sample target detection, the present invention proposes a self-supervised learning method to provide more constraints for latent feature construction, which to a certain extent alleviates the negative impact of model fine-tuning and latent knowledge in new classes on the base class, thereby improving the overall performance of the detection model.

[0062] The present invention uses self-supervised learning for training in the category domain and the detection box domain respectively, so as to alleviate the decline in the performance of the model base class. First, self-supervised learning is used in base class training to provide more constraints and perspectives for the potential feature construction module, and the problem of training collapse is avoided by stopping the gradient backpropagation. Then, self-supervised learning is used in new class training to constrain the features learned by the model in base class training by constructing proxy tasks. With the help of self-supervised learning, the method proposed in the present invention can ensure to the greatest extent that the knowledge learned on the base class is not forgotten, so that the model has good feature extraction capabilities for both the base class and the new class, and thus has good target detection capabilities for both the base class and the new class. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a diagram showing the overall architecture of a two-stage generalized small-sample target detection method based on self-supervision disclosed in one embodiment of the present invention;

[0064] Figure 2 A model framework diagram of the detection model base class training phase disclosed in one embodiment of the present invention;

[0065] Figure 3 This is a model framework diagram for the new class training phase of the detection model disclosed in one embodiment of the present invention;

[0066] Figure 4 This is a model framework diagram for self-supervised detection box domain training disclosed in an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of a two-stage generalized small-sample target detection system based on self-supervision disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0069] The present invention is described in further detail below with reference to the accompanying drawings:

[0070] like Figure 1As shown, the present invention utilizes latent knowledge and self-supervised learning during detection model training to prevent the loss of base class knowledge, thereby improving the performance of generalized small-sample object detection. During the detection model pre-training phase, the present invention uses latent features to define each class, describing each class using attribute groups. The process of constructing latent features is relatively complex, so the present invention introduces self-supervised learning as an auxiliary task to improve the efficiency of latent feature construction. Specifically, the process includes color transformation and translational slicing of the training set, dataset expansion, and the creation of self-supervised proxy tasks to enforce model training constraints and prevent training crashes by stopping gradient backpropagation. During the detection model fine-tuning phase, the number of features in the latent feature container is first increased to enable the newly added features to capture the latent knowledge of the new class. However, the addition of latent features can cause catastrophic forgetting of the base class, impacting the overall performance of the detection model. Therefore, self-supervised constraints are employed to retain the representative latent features learned during the pre-training phase, while a dynamic update strategy is used to further optimize these features during the learning of new classes. This approach effectively mitigates the negative impact of model fine-tuning and the addition of new latent features on the base class, thereby improving the overall performance of the model.

[0071] Specifically, the present invention provides a two-stage generalized small sample target detection method based on self-supervision, comprising:

[0072] S1. Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0073] in,

[0074] Attributes, as a means of connecting classes, present several challenges. First, manually labeling attributes makes it difficult to design a complete set of discriminative attributes for object detection. This is because it is difficult to capture all human-defined attributes in an image, and it is also difficult to determine whether the labeled attributes are sufficient for the detection task. Second, manually labeling attributes requires significant manpower and resources.

[0075] This paper proposes that latent knowledge exists in feature space that can be used to segment and classify features, just as attributes in visual space can segment and classify classes. Unlike attributes, latent knowledge is learned autonomously and does not require separate labeling. This approach avoids the difficulty and cost of manual attribute labeling and improves classification performance.

[0076] To this end, the present invention proposes a latent knowledge classifier (LK-Classifier), which is divided into a latent feature module and a prototype classifier module, with the purpose of using latent knowledge to classify candidate boxes.

[0077] S2. Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0078] Among them, include:

[0079] Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge;

[0080] Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate frame, and M represents the number of potential knowledge corresponding to the candidate frame;

[0081] When pre-training the two-stage detection model based on the base class set in the self-supervised category domain, C = |C base |+1; when fine-tuning the two-stage detection model based on the new set of categories in the self-supervised category domain, C=|C base |+|C novel |+1, C base Indicates the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

[0082] S3, divide the training data set into a base class set with a large number of samples and a new class set with a small number of samples;

[0083] in,

[0084] In the pre-training phase, the two-stage detection model is pre-trained using only the base class set, and in the fine-tuning phase, the model is fine-tuned using all classes.

[0085] S4, the self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the latent knowledge of the base category set and the new category set respectively;

[0086] in,

[0087] The two-stage detection model adopts a two-stage Faster RCNN network model, and its network structure ultimately consists of five parts: feature extractor, regressor, latent feature module, prototype classifier module and self-supervised auxiliary branch module.

[0088] Further,

[0089] The self-supervised category domain pre-trains a two-stage detection model based on the latent knowledge of the base class set, including:

[0090] Perform color transformation and translation transformation on the training samples in the base class set to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond one to one;

[0091] For each pair of samples in the two groups of base class expansion samples, the ROI features are obtained using the weight-sharing feature extractor one by one;

[0092] After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set.

[0093] The self-supervised task constrains ROI features through cosine similarity loss, and calculates class probabilities to constrain categories through the prototype classifier module, resulting in a pre-trained two-stage detection model.

[0094] The self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new category set, including: fine-tuning the pre-trained two-stage detection model based on the new category set;

[0095] Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one;

[0096] For each pair of samples in the two groups of new class expansion samples, the weight-sharing feature extractor is used to obtain ROI features one by one;

[0097] After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set and the new class set.

[0098] The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

[0099] Specifically,

[0100] like Figure 2 As shown in the figure, during the pre-training phase of the detection model, the training samples in the base class set are first subjected to color transformation and translation shearing to expand the data size. The two expanded samples are then passed through a weight-sharing feature extractor to obtain ROI-pooled features. Similarity scores are calculated through latent feature search after the fully connected layer (FC). The self-supervised task then constrains the latent features using a cosine similarity loss to help better extract visual features. Finally, the classification task calculates class probabilities and constrains the categories using a prototype classifier module.

[0101] The loss function of the self-supervised category domain in the pre-training stage is as follows:

[0102]

[0103] like Figure 3 As shown, the detection model is fine-tuned for the new category. The image color transformation and translation are consistent with pre-training. The latent feature search module increases the number of features for the new category and newly added classes. In the self-supervised module, latent features learned during the base class phase are also used to constrain the training of the new category. Self-supervised constraints are used to prevent the model from forgetting the base class. The notification of the base class latent features in the self-supervised module is slowly updated through gradient updates, and the feature learning module in the new class training branch is optimized, thereby improving overall classification accuracy.

[0104] The latent feature consistency loss formula is as follows:

[0105]

[0106] The loss of the self-supervised category domain in the fine-tuning stage is as follows:

[0107]

[0108] S5. The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain the final detection model;

[0109] in,

[0110] The self-supervised detection box domain pre-trains a two-stage detection model based on a base class set, including:

[0111] Perform color transformation and translation transformation on the base class samples to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond one to one;

[0112] For each pair of corresponding samples in the two groups of base class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor;

[0113] The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0114] The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0115] The two sets of regression boxes are used to extract the corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0116] The self-supervised detection box domain fine-tunes the two-stage detection model based on the new set of classes to obtain the final detection model; this includes:

[0117] Perform color transformation and translation transformation on the new class set samples respectively to obtain two groups of new class expansion samples, and the two groups of base class expansion samples correspond one to one;

[0118] For each pair of samples in the two groups of new class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor;

[0119] The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames;

[0120] The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching;

[0121] The two sets of regression boxes are used to extract the corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

[0122] Specifically,

[0123] The pre-training process and subsequent fine-tuning process of the self-supervised detection box region module are basically the same.

[0124] like Figure 4 As shown in Figure 1, we first perform color transformation and translation transformation on the sample image X to obtain enhanced sample images X1 and X2. Then, the enhanced sample image is passed through the weight-sharing feature extractor to obtain the ROI pooling feature, and the corresponding prediction box is obtained through the fully connected layer FC and the detection head composed of the fully connected layer and ReLU. Finally, the corresponding real detection box label P after the translation transformation is obtained. i For comparison, the target assignment constraint operation of sample matching is performed. For detection boxes without targets, the corresponding background labels will be used.

[0125] At the same time, the features of the two sets of predicted bounding boxes are mapped to another visual space through a weight-shared embedding network, and then momentum updates are performed on the features through contrastive learning. The regression head of the bounding box features follows the architecture of the MOCO v2 network, that is, a two-layer multi-layer perceptron (MLP) with ReLU as the regression head. Contrastive learning ensures that predictions for the same object should have similar features, while predictions for different objects should have greater differences. To avoid gradient collapse, this paper uses momentum updates to ensure fast and efficient training. The contrastive loss is as follows:

[0126]

[0127] Wherein tau is a temperature hyperparameter that controls the difficulty of the contrastive learning task, which is set to 0.5 in the present application.

[0128] S6, input the to-be-detected image into the obtained final detection model, obtain ROI features through the feature extractor, perform latent knowledge search on the to-be-detected image based on the latent knowledge container after dimension reduction through the full connection layer, segment the features of the to-be-detected image according to the searched latent knowledge, and obtain the relationship between the features of the to-be-detected image and each latent knowledge.

[0129] Wherein, comprising:

[0130] The ROI features are reduced in dimension through the full connection layer to be consistent with the dimensions of the latent knowledge in the latent knowledge container, and the features are obtained.

[0131] The similarity scores between the reduced features and each latent knowledge in the latent knowledge container are calculated through the cosine similarity.

[0132] Specifically,

[0133] When a given input image x is given, first, the ROI features F(x) are obtained using the feature extractor, and the features F(x) are reduced in dimension through the full connection layer to be consistent with the dimensions of the latent knowledge in the latent knowledge container, and the features G(x) are obtained. Z×D Wherein Z and D are the number and dimension of the ROI features respectively. Then, the features F(x) are reduced in dimension, because the dimension needs to be consistent with the dimension of the latent knowledge, and the dimension of the latent knowledge is too large to cause difficulty in learning the latent knowledge. Therefore, the full connection layer is used to reduce the dimension of F(x) to obtain G(x) R Z×m :

[0134] G(x) i =W c *F(x) i +b c

[0135] Wherein W c and b c are parameters of the full connection layer. Then, the latent knowledge is searched for the features G(x) i , the features G(x) i are segmented according to the latent knowledge, and the relationship between the features G(x) i and each latent knowledge is obtained.

[0136] The cosine similarity is used to calculate the similarity scores U i ={U i,1 ,U i,2 ,…,U i,N ,} between G(x) i,j and each latent knowledge Q i . U j represents the similarity between G(x) i and Qj The similarity score between .

[0137]

[0138] S7. Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and each potential knowledge, the image to be detected is classified through the prototype classifier module to obtain the target detection result.

[0139] Among them, including:

[0140] The cosine similarity between the calculated dimensionality reduction features and the similarity scores between each potential knowledge container and the prototypes of each candidate box in the prototype container is judged;

[0141] After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

[0142] Specifically,

[0143] Comprehensively consider the true probability distribution of all potential knowledge of each class and U i The actual probability distribution in U is obtained by using the prototype classifier i Specifically, U i and all class prototypes P again

[0144] Chord similarity judgment, the result is passed through a softmax layer to obtain the final category probability, as shown in the following formula:

[0145]

[0146] S i,k Denotes G(x) i is the probability of the kth class, α and β are the scaling factors of the amplified gradient, and the present invention sets them both to 20 in the experiment.

[0147] like Figure 5 As shown, the present invention also provides a two-stage generalized small sample target detection system based on self-supervision, including: a potential knowledge classifier construction module, a self-supervised category domain pre-training and fine-tuning module, a self-supervised detection box domain pre-training and fine-tuning module, an image detection module and a detection result acquisition module;

[0148] Latent Knowledge Classifier building blocks for:

[0149] Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module;

[0150] Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected;

[0151] Self-supervised category domain pre-training and fine-tuning module for:

[0152] The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples;

[0153] The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively;

[0154] Self-supervised detection box domain pre-training and fine-tuning module for:

[0155] The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain the final detection model;

[0156] Image detection module for:

[0157] The image to be detected is input into the final detection model, and the ROI features are obtained through the feature extractor. After dimensionality reduction by the fully connected layer, the potential knowledge search is performed on the image to be detected based on the potential knowledge container. The features of the image to be detected are segmented based on the searched potential knowledge, and the relationship between the features of the image to be detected and each potential knowledge is obtained.

[0158] The test result acquisition module is used to:

[0159] Based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the prototype classifier module is used to classify the image to be detected and obtain the target detection result.

[0160] Advantages of the present invention:

[0161] To address the problem that connections cannot be established between candidate box categories in generalized small sample target detection, the present invention proposes the use of latent knowledge. Through feature extraction, similar candidate classes are given similar latent knowledge representations, thereby constructing connections between candidate box categories and categories through latent knowledge. Therefore, when learning new classes after model pre-training, the knowledge of the base class and the new class can be connected, thereby achieving accurate target detection of the new class.

[0162] To address the problem of base class forgetting in generalized small-sample target detection, the present invention proposes a self-supervised learning method to provide more constraints for latent feature construction, which to a certain extent alleviates the negative impact of model fine-tuning and latent knowledge in new classes on the base class, thereby improving the overall performance of the detection model.

[0163] The present invention uses self-supervised learning for training in the category domain and the detection box domain respectively, so as to alleviate the decline in the performance of the model base class. First, self-supervised learning is used in base class training to provide more constraints and perspectives for the potential feature construction module, and the problem of training collapse is avoided by stopping the gradient backpropagation. Then, self-supervised learning is used in new class training to constrain the features learned by the model in base class training by constructing proxy tasks. With the help of self-supervised learning, the method proposed in the present invention can ensure to the greatest extent that the knowledge learned on the base class is not forgotten, so that the model has good feature extraction capabilities for both the base class and the new class, and thus has good target detection capabilities for both the base class and the new class.

[0164] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A two-stage generalized small-sample target detection method based on self-supervision, characterized by: include: Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module; Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected; The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples; The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the latent knowledge of the base category set and the new category set respectively; The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the potential knowledge of the base class set and the new class set to obtain the final detection model; Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected based on the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge; Based on the true probability distribution of all potential knowledge of various candidate frames in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

2. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The two-stage detection model adopts a two-stage Faster RCNN network model.

3. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised category domain pre-trains a two-stage detection model based on the latent knowledge of the base class set, including: Performing color transformation and translation transformation on the training samples in the base class set respectively to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond to each other one to one; For each pair of samples in the two groups of base class expansion samples, the ROI features are obtained using the weight-sharing feature extractor one by one; After the ROI features are reduced in dimension by the fully connected layer, potential knowledge search is performed based on the potential knowledge container, and the cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set. The self-supervised task constrains the ROI features through the cosine similarity loss, and calculates the class probability constraint category through the prototype classifier module to obtain a pre-trained two-stage detection model.

4. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised category domain fine-tunes the two-stage detection model based on the latent knowledge of the new class set, including: fine-tuning the pre-trained two-stage detection model based on the new class set; Among them, the samples in the new class set are subjected to color transformation and translation transformation respectively, and two groups of new class expansion samples are obtained, and the two groups of new class expansion samples correspond to each other one by one; For each pair of samples in the two groups of new class expansion samples, the weight-sharing feature extractor is used to obtain ROI features one by one; After the ROI features are reduced in dimension by the fully connected layer, a potential knowledge search is performed based on the potential knowledge container, and a cosine similarity score with each potential knowledge is calculated. The potential knowledge container stores all potential knowledge of the base class set and the new class set. The self-supervised task constrains the ROI features through cosine similarity loss, gradient updates the base class potential knowledge in the self-supervised module, and calculates the class probability constraint category through the prototype classifier module to obtain a fine-tuned two-stage detection model.

5. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised detection box domain pre-trains the two-stage detection model based on the base class set, including: Perform color transformation and translation transformation on the base class samples to obtain two groups of base class expansion samples, and the two groups of base class expansion samples correspond one to one; For each pair of corresponding samples in the two groups of base class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor; The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames; The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching; The two sets of regression boxes are used to extract corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

6. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The self-supervised detection box domain fine-tunes the two-stage detection model based on the new set of classes to obtain the final detection model; include: Perform color transformation and translation transformation on the new class set samples respectively to obtain two groups of new class expansion samples, and the two groups of base class expansion samples correspond one to one; For each pair of samples in the two groups of new class expansion samples, two sets of ROI features are obtained using a weight-sharing feature extractor; The two sets of ROI features are respectively reduced in dimension by the fully connected layer, and then the detection head composed of the fully connected layer and ReLU obtains two sets of regression frames; The two sets of regression frames are compared with the true detection frames after color transformation and the true detection frames after parallel transformation to perform target allocation constraints for sample matching; The two sets of regression boxes are used to extract corresponding box features using a weight-sharing embedding network, the two sets of box features are compared and learned, and the candidate boxes in the prototype container are updated with momentum.

7. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The potential knowledge container is defined to store all potential knowledge, and the prototype container is defined to store the prototypes of various candidate boxes to be detected; including: Define a potential knowledge container Q = {Q i ∈R m , i=1,…,N}, N and m represent the number and dimension of potential knowledge respectively, Q i represents potential knowledge, and R represents the dimension of potential knowledge; Defining a prototype container C represents the number of candidate box categories that the detection model needs to detect. represents the score of the βth potential knowledge in the αth candidate frame, and M represents the number of potential knowledge corresponding to the candidate frame; When the two-stage detection model is pre-trained based on the base class set in the self-supervised category domain, C=|C base |+1; when the self-supervised classification domain fine-tunes the two-stage detection model based on the new class set, C=|C base |+|C novel |+1, C base Indicates the number of candidate boxes in the base class set, C novel Indicates the number of candidate boxes in the new class set.

8. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The method comprises: inputting an image to be detected into the final detection model, obtaining ROI features through a feature extractor, performing a potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through a fully connected layer, segmenting the features of the image to be detected based on the searched potential knowledge, and obtaining a relationship between the features of the image to be detected and each potential knowledge; The ROI feature is reduced in dimension by a fully connected layer to be consistent with the potential knowledge dimension in the potential knowledge container to obtain a feature; The similarity score between the reduced-dimensional feature and each potential knowledge in the potential knowledge container is calculated by cosine similarity.

9. The two-stage generalized small-sample target detection method based on self-supervision according to claim 1, characterized in that: The method of classifying the image to be detected by the prototype classifier module based on the true probability distribution of all potential knowledge of various candidate boxes in the prototype container and the relationship between the features of the image to be detected and the potential knowledge to obtain the target detection result includes: Perform cosine similarity judgment on the similarity scores between the calculated dimensionality reduction features and each potential knowledge container and the prototypes of each candidate box in the prototype container; After the softmax layer, the probability that the image to be detected belongs to each candidate box category is obtained, and the candidate box with the largest probability is taken as the target detection result of the image to be detected.

10. A two-stage generalized small sample target detection system based on self-supervision, implementing the two-stage generalized small sample target detection method based on self-supervision according to any one of claims 1 to 9, characterized in that: include: Latent knowledge classifier construction module, self-supervised category domain pre-training and fine-tuning module, self-supervised detection box domain pre-training and fine-tuning module, image detection module and detection result acquisition module; The latent knowledge classifier building module is used to: Construct a latent knowledge classifier, including a latent feature module and a prototype classifier module; Define a potential knowledge container to store all potential knowledge, and define a prototype container to store the prototypes of various candidate boxes to be detected; The self-supervised category domain pre-training and fine-tuning module is used to: The training data set is divided into a base class set with a large number of samples and a new class set with a small number of samples; The self-supervised category domain pre-trains and fine-tunes the two-stage detection model based on the base category set and the new category set respectively; The self-supervised detection box domain pre-training and fine-tuning module is used to: The self-supervised detection box domain pre-trains and fine-tunes the two-stage detection model based on the base class set and the new class set to obtain the final detection model; The image detection module is used to: Input the image to be detected into the final detection model, obtain ROI features through the feature extractor, perform potential knowledge search on the image to be detected based on the potential knowledge container after dimensionality reduction through the fully connected layer, segment the features of the image to be detected based on the searched potential knowledge, and obtain the relationship between the features of the image to be detected and each potential knowledge; The detection result acquisition module is used to: Based on the true probability distribution of all potential knowledge of various candidate frames in the prototype container and the relationship between the features of the image to be detected and the potential knowledge, the image to be detected is classified by the prototype classifier module to obtain the target detection result.

Images