Remote sensing image few-sample target detection method combined with double-branch evolution module

Abstract

The invention discloses a remote sensing image few-sample target detection method combined with a double-branch evolution module, and the method comprises the steps: a basic training stage: training a target detector through employing a large amount of base class data with labels, so as to obtain a pre-trained model parameter; in the few-sample fine tuning stage, an original feature extractor structure is evolved into parallel base class and new class branches, and the problem of disastrous forgetting of the model to base class knowledge is effectively relieved through a freezing mechanism. The category separation region-of-interest detector is used for detecting a base category sample and a new category sample respectively, and can adapt to features of different target categories. And the distillation loss is used for guiding a new type of detection head, so that the detection precision is improved while the model complexity is reduced. According to the method, the new class detection precision is improved, and meanwhile, the problem of disastrous forgetting of a remote sensing image few-sample target detection method on base class knowledge can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and more specifically to using a few-sample learning method to perform remote sensing image target detection tasks. Background Art

[0002] With the rapid development of aerospace technology in recent years, the research on remote sensing image target detection has increased, which has brought new opportunities and challenges to the development of computer vision target detection tasks. At the same time, remote sensing image target detection is of great significance in fields closely related to people's livelihood, such as weather forecasting, urban planning, and disaster monitoring. With the rapid development of deep learning technology and the emergence of large-scale manually annotated data, target detection based on deep learning has achieved better performance in terms of accuracy and efficiency than traditional methods. In natural scenes, a large number of target detection methods based on convolutional neural networks have been proposed. In view of the large-scale changes, arbitrary directions, and complex backgrounds of remote sensing image geospatial targets, researchers have proposed various remote sensing image target detectors and achieved good results. However, these traditional remote sensing image target detection algorithms rely on a large amount of labeled data for training. In practical applications in special scenes such as natural disasters and military battlefields, it is difficult to collect enough training data because a large amount of manpower is required to annotate images. As a result, the existing remote sensing image target detection models are prone to overfitting and low detection accuracy when using a small amount of labeled data for model training. Therefore, it is particularly important to explore how to detect targets efficiently and accurately with a small number of samples. In recent years, the development of few-shot learning in the field of object detection has provided new research ideas for solving the above problems.

[0003] Few-shot learning can mine prior knowledge in samples by optimizing training strategies and network structures, thereby improving the generalization performance of the model when the number of samples is small. Few-shot object detection solves the shortcoming that traditional detectors require a large amount of training data, and is currently receiving more and more attention due to its broad application prospects.

[0004] Although few-shot object detection has achieved good results in natural scenes, due to the characteristics of remote sensing images such as many small objects and complex backgrounds, the detection effect of directly applying the detection model in natural scenes to remote sensing images is not ideal. Therefore, how to improve the accuracy of remote sensing image object detection under few-shot conditions has become a hot topic and difficulty in current research. Existing remote sensing image few-shot object detection algorithms only focus on how to improve the detection accuracy of new classes, ignoring the problem of low base class detection accuracy caused by the catastrophic forgetting of base class knowledge by the model.

[0005] In order to solve the problem of low detection accuracy in few-shot target detection in remote sensing images due to catastrophic forgetting of base class knowledge by the model, a few-shot target detection method for remote sensing images combined with a dual-branch evolutionary module was invented. Summary of the invention

[0006] The structure of the remote sensing image few-sample target detection model proposed in this invention is as follows: Figure 2 As shown. It is mainly divided into two stages: base training and few-sample fine-tuning. First, in the base training stage, FasterR-CNN is trained by using a large amount of annotated base class data, in which the feature extraction part uses ResNet and feature pyramid network to obtain pre-trained model parameters. Secondly, in the few-sample fine-tuning stage, in order to avoid the overfitting phenomenon of the model due to the small amount of sample data, the input of the model is a balanced data set composed of a small number of annotated base classes (K-shot), new classes (K-shot), and a balanced data set after horizontal rotation. The parameters of the base training are used as the initial weights to fine-tune the few-sample target detection model. In addition, the present invention proposes a dual-branch evolution module and a category separation ROI detector to promote the model to better learn new class knowledge in remote sensing images while maintaining the detection ability of the base class.

[0007] The present invention finds that in TFA, only the predictor is fine-tuned using a balanced data set, and other modules such as the backbone network and RPN are frozen. However, the RPN may extract a large number of target candidate regions in the image, which serves as the input for subsequent target detection tasks. In the base training stage, RPN only focuses on base class targets when generating candidate regions, which makes RPN more inclined to the base class when classifying foreground and background, and it is difficult to generalize well when facing new classes. Unlike TFA, FSCE freezes the backbone network and ROIPooling module, and other parts of the model participate in training. The present invention finds that the freezing mechanism of the above two methods severely restricts knowledge transfer, resulting in poor new class detection performance when facing remote sensing scenes, and the above freezing mechanism is prone to cause the model to catastrophically forget the base class knowledge. In remote sensing images with complex backgrounds, when base classes and new classes exist in the same image at the same time, the accuracy of few-sample target detection cannot be effectively improved. Therefore, in the fine-tuning stage, the present invention freezes the feature extraction part, the base class branch of the dual-branch evolution (DBE) module, and the base class detection head in the class-separated ROI (CSR) detector, and fine-tunes the RPN with a balanced data set to enable it to discover new class targets. The effectiveness of the freezing mechanism of the present invention method is demonstrated in subsequent ablation experiments. The freezing mechanism alleviates the overfitting phenomenon of the model due to the small amount of data samples while preventing the model from forgetting the base class knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only schematic diagrams of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0009] Figure 1 The present invention is summarized as a flow chart;

[0010] Figure 2 This is a diagram of the network architecture for few-shot target detection;

[0011] Figure 3 This is a diagram of the double-branch evolution module structure. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The embodiment of the present invention discloses a remote sensing image few-sample target detection method combined with a dual-branch evolution module, including two parts: a base training stage and a few-sample fine-tuning stage:

[0014] The basic training phase steps are as follows:

[0015] S1: build feature extraction network;

[0016] S2: build feature pyramid network;

[0017] The steps of the few-shot fine-tuning phase are as follows:

[0018] S3: Few-shot fine-tuning stage. In order to avoid the overfitting phenomenon of the model due to the small amount of sample data, the input of the model is a balanced data set consisting of a small number of annotated base classes (K-shot), new classes (K-shot) and a balanced data set after horizontal rotation. The parameters of the base training are used as the initial weights to fine-tune the few-shot object detection model.

[0019] S4: Dual-branch evolution module. The original model is evolved into a base class branch and a new class branch. By using the method of freezing pre-trained parameters in the base class branch, the model can effectively improve the detection accuracy of the new class while avoiding the problem of decreased detection accuracy caused by catastrophic forgetting of the base class.

[0020] S5: Category separation ROI detector. The base class and the new class are detected separately, so that the model can better adapt to the characteristics of different targets, thereby accurately locating the base class and the new class targets, and improving the accuracy and robustness of target detection.

[0021] S6: Distillation loss: Use distillation loss to guide the learning of new class detection heads, so that the new class detection heads inherit the knowledge of the base class model, thereby improving the accuracy of few-sample target detection in remote sensing images.

[0022] S7: End.

[0023] In order to further optimize the above technical solution. First, according to steps S1-S2, Faster R-CNN is trained by using a large amount of annotated base class data, wherein the feature extraction part uses ResNet and feature pyramid network to obtain pre-trained model parameters.

[0024] Secondly, the few-shot fine-tuning stage is performed according to step S3. In this stage, in order to avoid the overfitting phenomenon of the model due to the small amount of sample data, the input of the model is a balanced data set consisting of a small number of annotated base classes (K-shot), new classes (K-shot) and a balanced data set after horizontal rotation. The parameters of the base training are used as the initial weights to fine-tune the few-shot object detection model.

[0025] Again, in the few-sample fine-tuning stage, according to step S4, this method proposes a dual-branch evolution module to evolve the original model into a base class branch and a new class branch. By using the method of freezing pre-training parameters in the base class branch, the model can effectively improve the detection accuracy of the new class while avoiding the problem of detection accuracy degradation caused by catastrophic forgetting of the base class.

[0026] Finally, in the few-shot fine-tuning stage, according to S5-S6, this method proposes a category separation ROI detector to detect targets for the base class and the new class respectively, thus improving the few-shot target detection accuracy of the model. At the same time, the distillation loss is used to guide the learning of the new class detection head, so that the new class detection head inherits the knowledge of the base class model, thereby improving the accuracy of few-shot target detection in remote sensing images.

[0027] The method involved in the present invention is further described below.

[0028] In order to alleviate the problem of catastrophic forgetting of the base class knowledge of remote sensing images by the model, this paper proposes a dual-branch evolution module, which enables the model to improve the detection accuracy of new classes while maintaining the detection ability of base classes. Figure 3As shown, first, in order to solve the problem of model overfitting caused by insufficient sample data and uneven data distribution in few-sample target detection, the present invention introduces Dropblock regularization in the head of the DBE module. By randomly selecting a block area in the feature map and setting all pixels in the area to zero, the model is forced to learn more robust and generalized feature representations during training, thereby improving the generalization ability of the model. Then the two consecutive fully connected layers in Faster R-CNN are evolved into two parallel branches, namely the base class branch and the new class branch. In the few-sample fine-tuning stage, the base class branch with pre-trained parameters is frozen to achieve the purpose of protecting the previous base class knowledge, while the new class branch participates in training to learn new class knowledge. In addition, by fusing the outputs of two parallel fully connected layers, the ROI feature extractor will simultaneously consider the base class knowledge from the frozen layer and the new class knowledge from the trainable layer. The output of the DBE module is shown in the following formula:

[0029]

[0030] Where F out represents the output feature, F i Represents the input feature, α represents the weight of the frozen branch to adjust the influence of base class knowledge and new class knowledge on the model. and Represent the weight matrix and bias vector in the base class frozen branch respectively. FC and b FC Denote the weight matrix and bias vector in the trainable layer of the new class respectively. b Indicates that the input feature map is subjected to Dropblock regularization.

[0031] Compared with natural images, due to different imaging methods, the target categories in remote sensing images are usually diverse and complex, covering various types of ground objects. When facing the task of few-sample target detection, an image in the remote sensing image data set may contain both base class targets and new class targets. The traditional few-sample target detection algorithm cannot solve this complex situation well. Therefore, the method of the present invention proposes a category separation ROI detector to detect the base class and the new class separately, so that the model can better adapt to the characteristics of different targets, thereby accurately locating the base class and the new class targets, and improving the accuracy and robustness of target detection.

[0032] The structure of the category separation ROI detector is as follows Figure 2The red dotted box indicates that it consists of two detection heads, namely the base class ROI detection head and the new class ROI detection head. The two detect the base class and new class targets generated by the RPN and DBE modules in parallel. The base class detection head is frozen during the training process, and the same weights as in the base training are maintained to predict the base class targets. The new class detection head maintains the weights in the fine-tuning stage and detects the base class and new class targets at the same time. Since the base class detection head is obtained through rich data training, the distillation loss is used to transfer knowledge between the two to achieve the purpose of the new class detection head inheriting the knowledge of the base class model. The distillation loss is shown in the formula:

[0033]

[0034] Where k represents the category, C b Represents all base class data, represents the probability of the base class detection head predicting class k, represents the probability of the new class detection head predicting class k. By minimizing the distillation loss, the output of the new class detection head is made closer to the output of the base class detection head, thereby achieving the purpose of transferring the base class knowledge. Therefore, the total loss of the CSR detector in the few-sample fine-tuning stage is:

[0035]

[0036] In the formula, and represents the classification and regression loss of new classes in the fine-tuning stage. β represents the coefficient of knowledge distillation loss.

[0037] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined by the scheme may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the scheme, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims

1. A remote sensing image few-sample target detection method combined with a dual-branch evolutionary module, including a basic training stage and a few-sample fine-tuning stage, wherein: The model architecture of the few-shot fine-tuning stage consists of a dual-branch evolutionary module and a category-separable ROI detector. The basic training phase steps are as follows: S1: Build a feature extraction network. S2: Build a feature pyramid network. The steps of the few-shot fine-tuning phase are as follows: S3: Few-shot fine-tuning stage. In order to avoid the overfitting phenomenon of the model due to the small amount of sample data, the input of the model is a balanced data set consisting of a small number of annotated base classes (K-shot), new classes (K-shot) and a balanced data set after horizontal rotation. The parameters of the base training are used as the initial weights to fine-tune the few-shot object detection model. S4: Dual-branch evolution module. In the few-shot fine-tuning stage, the original model is evolved into a base class branch and a new class branch. By using the method of freezing pre-trained parameters in the base class branch, the model can effectively improve the detection accuracy of the new class while avoiding the problem of decreased detection accuracy caused by catastrophic forgetting of the base class. S5: Class separation ROI detector. The class separation ROI detector detects the base class and the new class separately, so that the model can better adapt to the characteristics of different targets, thereby accurately locating the base class and the new class targets, and improving the accuracy and robustness of target detection. S6: Distillation loss. Distillation loss is used to guide the learning of the new class detection head, so that the new class detection head inherits the knowledge of the base class model, thereby improving the accuracy of few-sample target detection in remote sensing images. S7: End.

2. According to claim 1, a remote sensing image few-sample target detection method combined with a dual-branch evolution module is characterized in that: In S3, the model pre-training parameters and balanced dataset of step S2 are used to fine-tune the model.

3. The method for detecting a small number of targets in remote sensing images using a dual-branch evolutionary module according to claim 1, characterized in that: In S3 and S4, the parameters of different modules are frozen to prevent the model from overfitting.

4. The method for detecting target with a small number of samples from remote sensing images according to claim 1 is characterized in that, in step S6, the target classification and positioning of the base class and the new class are completed.

Images