A method for target detection in small sample remote sensing images based on feature enhancement

Abstract

The present invention relates to a small sample remote sensing image target detection method based on feature enhancement, and belongs to the field of image processing technology. The method comprises: firstly dividing and processing the remote sensing image data set, and then in the training stage, inputting a large amount of base class data into the feature extraction network; then adding a context feature enhancement module after the feature extraction network, selectively strengthening the category perception features by using the image-level context information indicating the existence or non-existence of the object category, which can help the target detector to better understand the scene. In the small sample fine-tuning stage, the trained feature extraction network is used, some of its parameters are frozen, the balanced base class and new class data are input into the network, and then the revised proposal box is obtained by the subsequent RoI feature extractor and the representation compensation module, and then the target prediction is performed on the remote sensing image of the new category. The present invention can solve the small sample problem in remote sensing image target detection to a certain extent.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a small sample remote sensing image target detection method based on feature enhancement. Background Art

[0002] Object detection is a hot area in computer vision. Its task is to select any number of objects of interest in an image with a bounding rectangle and identify the object category. With the rise of deep learning, especially deep convolutional neural networks, a series of excellent object detection algorithms have emerged in natural scene images with their powerful feature extraction capabilities. Obviously, unlike natural scenes, remote sensing images usually show the characteristics of objects appearing in arbitrary directions, highly dispersed objects, and complex backgrounds. Identifying foreground targets from the complex background environment of remote sensing images is also a big problem. Therefore, how to combine small sample learning methods and expand and generalize the detection model through a small number of labeled samples so that it can more accurately detect new categories of targets in remote sensing images to meet the needs of practical applications is a challenging research.

[0003] The usual steps of the existing fine-tuning based few-shot object detection model are to first divide a set of data into two groups, one of which is a base class C with many instances. base , the other group is divided into a new class C with only K (K is usually less than 10) instances in each class novel The training strategy is that in the first stage, i.e. the basic training stage, the feature extractor and proposal box predictor are trained on a large number of base classes; then in the second stage, i.e. the small sample fine-tuning stage, a small balanced training set is created with K samples per category, including base classes and new classes. Then randomly initialized weights are assigned to the proposal box prediction network of the new category, and only the proposal box classification and regression network, i.e. the last few layers of the detection model, are fine-tuned.

[0004] At present, the research on small sample learning focuses on the detection and recognition of natural scene images, but the research on small sample target detection and recognition in remote sensing scenes is relatively lacking, and some characteristics of remote sensing images make this problem more difficult: there is a high similarity between different categories of targets in remote sensing images, and targets of the same category may contain large differences, that is, they have the characteristics of "high similarity between classes and large differences within classes"; the scale / degree of ground objects in remote sensing images varies greatly, and common single target detection heads may be difficult to adapt; compared with natural scenes, the number of remote sensing scene image samples is smaller, which makes it easy for the detection model to confuse foreground targets and backgrounds. If the existing small sample learning algorithm applied to natural scenes is directly applied to the remote sensing image target detection task, it may lead to a decrease in detection performance.

[0005] Therefore, a new small-sample remote sensing image target detection method is urgently needed to improve the accuracy of target detection. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a small sample remote sensing image target detection method based on feature enhancement, so as to solve the problem in the prior art that the detection performance of the detection model is reduced due to the small number of remote sensing scene image samples.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A method for object detection in small sample remote sensing images based on feature enhancement is proposed. It obtains more useful feature information by adding a context feature enhancement module. The added context feature enhancement module is expected to extract more semantic features related to the detection target, thereby more accurately locating the target information in the remote sensing image. After selecting the proposal box in the fine-tuning stage, a representation compensation module is introduced to compensate for the forgetting of the base class features in the fine-tuning stage and learn the features of the new category at the same time. The specific steps include:

[0009] S1: Obtain small sample remote sensing image data, and divide the remote sensing image data set into a base class and a new class to obtain a balanced small sample remote sensing image data set containing the base class and the new class;

[0010] S2: Construct a small sample target detection model based on feature enhancement;

[0011] S3: Organize and divide the small sample remote sensing image data obtained in step S1 to obtain the support set Si and the query set Qi;

[0012] S4: Extract features from the support samples in the support set Si and the query samples in the query set Qi to obtain the support image feature map F si and query image feature map F Qi ;

[0013] S5: will support image feature map F si Input to the context feature enhancement module to obtain the enhanced feature map F si ′;

[0014] S6: The enhanced feature map F si ′ and query image feature map F Qi After matching, it is input into the region proposal network module to generate a proposal box;

[0015] S7: Using the proposal box from the query image feature map F Qi Extract candidate targets, and classify and regress the proposal boxes through classifiers and box regressors;

[0016] S8: Small sample fine-tuning stage: Freeze the parameters related to the backbone network, context feature enhancement module and feature pyramid network in the detection model, and input the balanced small sample remote sensing image data into the trained backbone network to extract features;

[0017] S9: Process the extracted features and output the predicted center point position and category; then enter the representation compensation module to obtain the corrected proposal box;

[0018] S10: Use the real annotation information of the current image and the corrected proposal box to do IOU;

[0019] S11: Use IOU to calculate the precision and recall, and calculate the evaluation index F1.

[0020] Furthermore, in step S1, the remote sensing image data set is divided into base classes and new classes, specifically including: taking several categories in the data set as base categories and the remaining categories as new categories; when dividing, the data set can also be divided into multiple different groups, the categories of the new classes set for each group are different, and try to make each category of the data set become a new class.

[0021] Furthermore, in step S2, a small sample target detection model based on feature enhancement is constructed, specifically including: using Faster R-CNN as the main framework of the detection model; and Resnet101 as the backbone network for feature extraction.

[0022] Further, step S5 specifically includes: supporting the image feature map F si The features at different levels are input into a contextual feature enhancement module, and the contextual information of different receptive fields is obtained by using dilated convolutions with different dilation rates. The sum size used here is 3×3, and the dilation rates are 1, 3, and 5. Finally, the high-level semantic features and low-level detail features are fused through the conv1×1 convolution layer; on the basis of the fused features, two fully connected layers with sigmoid activation function are added to predict the credibility of the target category in the remote sensing scene, and binary cross entropy loss is used for training.

[0023] Further, in step S7, the feature enhanced detection model is fine-tuned by a loss function, and the loss function expression is:

[0024] L det =L cls +L reg +θL CFE

[0025] Among them, L cls represents the target category classification loss, L regrepresents the proposed box regression loss, θ controls the context feature enhancement module loss L CFE Hyperparameters of .

[0026] Furthermore, in step S9, the representation compensation module is composed of a representation decoupling mechanism module, which is used to compensate for the forgetting of base class features in the fine-tuning stage and simultaneously learn the features of the new category, so as to obtain a corrected proposal box.

[0027] Further, in step S9, the representation compensation module specifically introduces a representation decoupling mechanism into the RoI feature extractor, which modifies a single branch with a parallel branch to decouple its representation of previous knowledge and new knowledge; specifically, for two consecutive fully connected (FC) layers in the RoI feature extractor, a parallel fully connected (FC) layer is added, which has pre-trained parameters for each component to represent the previous knowledge learned from the basic training stage; in the fine-tuning stage, the newly added branch is frozen to protect the previous knowledge, and the other branch is trainable to learn new knowledge; in addition, by fusing the outputs of the two parallel fully connected (FC) layers, the RoI feature extractor will consider the previous knowledge from the frozen layer and the new knowledge from the trainable layer; each fully connected (FC) layer calculation equipped with the representation decoupling mechanism can be described as:

[0028] F=act(λ·(W fc `F i +b fc `)+(1-λ)·(W fc F i +b fc ))

[0029] Among them, W fc `, b fc `Indicates the parameter of freezing the branch; W fc , b fc represents the trainable parameters for learning new knowledge; λ represents the weight vector, which is set to 0.5 to balance the influence of previous knowledge and new knowledge; act(*) represents the nonlinear activation function; F represents the output of the fully connected layer, F i represents the input of the fully connected layer.

[0030] The beneficial effect of the present invention is that the present invention can effectively ensure that small targets can still be accurately detected when the number of remote sensing scene image samples is small. Specifically, in order to cope with the characteristics of remote sensing images of "high similarity between classes and large differences within classes", the present invention adds a context feature enhancement module after the feature extraction network, which selectively enhances the category-aware features by using image-level context information indicating the presence or absence of object categories, which can help the target detector better understand the scene. In the small sample fine-tuning stage, the trained feature extraction network is used, some of its parameters are frozen, and the balanced base class and new class data are input into the network. The revised proposal box is obtained by the subsequent RoI feature extractor and representation compensation module, and then the target prediction is performed on the remote sensing image of the new category.

[0031] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0033] Figure 1 It is a flow chart of the small sample remote sensing image detection method based on feature enhancement of the present invention;

[0034] Figure 2 It is a small sample remote sensing image detection model structure based on feature enhancement of the present invention;

[0035] Figure 3 It is a structural diagram of the context information enhancement module (CFE) of the present invention;

[0036] Figure 4 The figure shows the structure diagram of the compensation module of the present invention. DETAILED DESCRIPTION

[0037] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0038] See also Figure 1 to Figure 4 The present invention provides a small sample remote sensing image detection method based on feature enhancement, which specifically includes the following steps:

[0039] Step 1: Select several categories of the remote sensing image dataset and divide them into base classes and new classes. When dividing, the dataset can also be divided into multiple different groups, and the categories of the new classes set for each group are different, and try to make each category of the dataset become a new class.

[0040] Step 2: Construct a small sample target detection model based on feature enhancement, specifically including: using Faster R-CNN as the main framework of the detection model and Resnet101 as the backbone network.

[0041] Step 3: Organize and divide the selected small sample data to obtain the support set Si and query set Qi.

[0042] Step 4: Extract features from the support samples in the support set Si and the query samples in the query set Qi to obtain the support image feature map F si and query image feature map F Qi .

[0043] Step 5: The obtained support image feature map F si Input the following context feature enhancement module, through which a more accurate feature map F of the target location information is obtained. si ′.

[0044] In the context feature enhancement module, the features of different levels obtained above are used to obtain context information of different receptive fields using dilated convolutions with different dilation rates. The sum size used here is 3×3, and the dilation rates are 1, 3, and 5. Finally, the high-level semantic features and low-level detail features are fused through the conv1×1 convolution layer. Two fully connected layers with sigmoid activation functions are added on the basis of the fused features to predict the credibility of the target category in the remote sensing scene, and binary cross entropy loss is used for training.

[0045] Step 6: Transform the feature map F si ′Inject the feature pyramid network from top to bottom to enrich the context information, and then fuse the feature map output by the feature pyramid network and the query image feature map F Qi After matching, it is input into the region proposal network module to generate a proposal box.

[0046] Step 7: Use the proposal box to extract the feature map F from the query image Qi Extract candidate targets.

[0047] Step 8: Classify and regress the proposed box through the classifier and box regressor; finally, fine-tune the feature-enhanced detection model through the loss function, and the loss function expression is:

[0048] L det =L cls +L reg +θL CFE

[0049] Among them, L cls represents the target category classification loss, L reg represents the proposed box regression loss, θ controls the context feature enhancement module loss L CFE Hyperparameters of .

[0050] Step 9: In the second stage, i.e., the small sample fine-tuning stage, the parameters related to the backbone network, context feature enhancement module, and feature pyramid network in the model are frozen, and the balanced small sample remote sensing image data, i.e., with base classes and new classes, and K samples in each class, are input into the trained feature extraction network.

[0051] In the small sample fine-tuning stage, the parameters of the region proposal network will be frozen, which will prevent the region proposal network from learning the features of new categories. For this reason, the region proposal network is fine-tuned at this stage. The anchor generator will generate Hi×Wi×k anchor boxes to perform classification and regression tasks through 3×3 convolution and 1×1 convolution respectively. Then, RPN randomly selects the first m positive anchors that may contain objects in each layer. In order to retain more positive boxes for new categories, the preset m is doubled.

[0052] Step 10: Process the extracted features and output the predicted center point location and category.

[0053] Step 11: Then enter the representation compensation module. This representation compensation module, which consists of a representation decoupling mechanism module, can be used to compensate for the forgetting of base class features during the fine-tuning stage and learn the features of new categories at the same time, thereby obtaining a corrected proposal box.

[0054] A representation decoupling mechanism will be introduced into the RoI feature extractor, which modifies a single branch with a parallel branch to decouple its representation of previous knowledge and new knowledge. Specifically, for two consecutive fully connected (FC) layers in the RoI feature extractor, a parallel FC layer is added with pre-trained parameters for each component to represent the previous knowledge learned from the basic training stage. During the fine-tuning stage, the newly added branch is frozen to protect the previous knowledge, and the other branch is trainable to learn new knowledge. In addition, by fusing the outputs of two parallel FC layers, the RoI feature extractor will consider the previous knowledge from the frozen layer and the new knowledge from the trainable layer. The calculation of each FC layer equipped with the representation decoupling mechanism can be described as:

[0055] F=act(λ·(W fc `F i +b fc `)+(1-λ)·(W fc F i +b fc ))

[0056] Among them, W fc `, b fc `Indicates the parameter of freezing the branch; W fc , b fc represents the trainable parameters for learning new knowledge; λ represents the weight vector, which is set to 0.5 to balance the influence of previous knowledge and new knowledge; act(*) represents the nonlinear activation function; F represents the output of the fully connected layer, F i represents the input of the fully connected layer.

[0057] Step 12: Use the real annotation information of the current image and the corrected proposal box to do IOU.

[0058] Step 13: Use IOU to calculate the precision and recall, and calculate the evaluation index F1. The calculation method is:

[0059]

[0060] Among them, TP represents the positive samples predicted by the model as positive, and FP represents the negative samples predicted by the model as positive.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A small sample remote sensing image target detection method based on feature enhancement, characterized in that: The specific steps include: S1: Obtain small sample remote sensing image data, and divide the remote sensing image data set into a base class and a new class to obtain a balanced small sample remote sensing image data set containing the base class and the new class; S2: Construct a small sample target detection model based on feature enhancement; S3: Organize and divide the small sample remote sensing image data obtained in step S1 to obtain the support set Si and the query set Qi; S4: Extract features from the support samples in the support set Si and the query samples in the query set Qi to obtain the support image feature map F si and query image feature map F Qi ; S5: will support image feature map F si Input to the context feature enhancement module to obtain the enhanced feature map F si ′; S6: The enhanced feature map F si ′ and query image feature map F Qi After matching, it is input into the region proposal network module to generate a proposal box; S7: Using the proposal box from the query image feature map F Qi Extract candidate targets, classify and regress the proposal boxes through classifiers and box regressors, and finally fine-tune the feature-enhanced detection model through the loss function; S8: Small sample fine-tuning stage: Freeze the parameters related to the backbone network, context feature enhancement module and feature pyramid network in the detection model, and input the balanced small sample remote sensing image data into the trained backbone network to extract features; S9: Process the extracted features and output the predicted center point position and category; then enter the representation compensation module to obtain the corrected proposal box; The representation compensation module specifically introduces a representation decoupling mechanism into the RoI feature extractor, which modifies a single branch with a parallel branch to decouple its representation of previous knowledge and new knowledge; for two consecutive fully connected layers in the RoI feature extractor, a parallel fully connected layer is added, which has pre-trained parameters for each component to represent the previous knowledge learned from the basic training stage; in the fine-tuning stage, the newly added branch is frozen to protect the previous knowledge, and the other branch is trainable to learn new knowledge; in addition, by fusing the outputs of the two parallel fully connected layers, the RoI feature extractor will consider the previous knowledge from the frozen layer and the new knowledge from the trainable layer; the calculation of each fully connected layer equipped with the representation decoupling mechanism is described as: F=act(λ·(W fc `Fi+bfc`)+(1-λ)·(W fc F i +b fc )) Among them, W fc `, b fc `Indicates the parameter of freezing the branch; W fc , b fc represents the trainable parameters for learning new knowledge; λ represents the weight vector used to balance previous knowledge and new knowledge; act(*) represents the nonlinear activation function; F represents the output of the fully connected layer, F i Represents the input of the fully connected layer; S10: Use the real annotation information of the current image and the corrected proposal box to do IOU; S11: Use IOU to calculate the precision and recall, and calculate the evaluation index F1.

2. The method for target detection in small sample remote sensing images based on feature enhancement according to claim 1, characterized in that: In step S1, the remote sensing image data set is divided into base classes and new classes, specifically including: taking several categories in the data set as base categories and the remaining categories as new categories; when dividing, the data set can also be divided into multiple different groups, the categories of the new classes set for each group are different, and each category of the data set can become a new class.

3. The method for target detection in small sample remote sensing images based on feature enhancement according to claim 1, characterized in that: In step S2, a small sample target detection model based on feature enhancement is constructed, specifically including: using Faster R-CNN as the main framework of the detection model; Resnet101 as the backbone network for feature extraction.

4. The method for target detection in small sample remote sensing images based on feature enhancement according to claim 1, characterized in that: Step S5 specifically includes: supporting the image feature map F si The features at different levels are input into a contextual feature enhancement module, and the contextual information of different receptive fields is obtained by using dilated convolutions with different dilation rates. Finally, the high-level semantic features and low-level detail features are fused through a conv1×1 convolution layer. On the basis of the fused features, two fully connected layers with sigmoid activation functions are added to predict the credibility of the target category in the remote sensing scene, and binary cross entropy loss is used for training.

5. The method for target detection in small sample remote sensing images based on feature enhancement according to claim 1, characterized in that: In step S9, the representation compensation module is composed of a representation decoupling mechanism module, which is used to compensate for the forgetting of base class features in the fine-tuning stage and simultaneously learn the features of the new category, so as to obtain a corrected proposal box.

Images