A few-sample target detection method, electronic device and computer medium
Through remote sensing image category division and multi-scale feature extraction, combined with the comparison learning of multi-scale candidate box comparison loss function, the problem of insufficient characterization ability of remote sensing image object detection under the condition of few samples in the existing technology is solved, and the goal is achieved. Detection effect.
Patent Information
- Application Number
- CN202210399493.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-15
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-04-15
AI Technical Summary
The prior art is difficult to improve the characteristic characterization ability of remote sensing image object detection under the condition of few samples, and does not pay enough attention to the characteristics of remote sensing images, resulting in poor detection results.
Through remote sensing image category division, the basic network is trained and upper and lower branches are built, the candidate boxes and multi-scale instance features are obtained, and the multi-scale candidate boxes are used for comparison learning, which improves the feature expression ability of the target detection network.
The accuracy of the detection of few-sample object is improved, the impact of the target direction and scale of remote sensing images on accuracy is weakened, and the generalization ability of the model is improved.
Smart Images

Figure CN114926622B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing image detection, and in particular to a few-sample target detection method, electronic equipment and computer medium. Background Art
[0002] With the use of deep learning, the information mining and feature expression capabilities based on convolutional neural networks have become stronger, and remote sensing image target detection algorithms can achieve high-performance detection results. However, as the number of training samples decreases, it is difficult for deep convolutional neural networks to mine the feature distribution of the entire large data set with less training data (few samples). Whether it is remote sensing images or natural scene images, there will be new categories in reality, and they may have only a small amount of available data. Many studies have proposed few-sample target detection algorithms to address this challenge. They can be roughly divided into three types: ① based on fine-tuning; ② based on metric learning; ③ based on model improvement methods.
[0003] In the existing technology, the fine-tuning method can improve the detection ability of the model with a small number of samples, but some parameters of the fine-tuning model are more prone to overfitting; the method based on metric learning depends on whether the samples used to construct the metric space are representative, and there is a certain degree of randomness in the training results. Secondly, the method based on model improvement improves the network structure to adapt to the task of few-sample learning. These methods can improve the target detection effect with a small number of samples to a certain extent, but they do not pay enough attention to how to improve the feature representation ability of the convolutional neural network with a small number of training samples. The ground objects on the remote sensing image are imaged from a bird's-eye view, and have the characteristics of arbitrary direction and complex background. The existing technology does not combine the characteristics of remote sensing images.
[0004] In summary, there is an urgent need for a target detection method adapted to remote sensing images to solve the problems existing in the existing technology. Summary of the invention
[0005] The purpose of the present invention is to provide a few-sample target detection method, and the specific technical solution is as follows:
[0006] A few-sample target detection method, the specific steps are as follows:
[0007] Step S1: remote sensing image classification, specifically, the categories contained in the remote sensing image are simulated and divided into basic categories and small sample categories;
[0008] Step S2: training a basic network, specifically, training a basic network on the samples included in the basic categories divided in step S1, which is used as initialization parameters for training a small sample model;
[0009] Step S3: Obtaining contrast features, specifically, constructing an upper branch based on the basic network of step S2, and obtaining the features in the candidate frame in the remote sensing image through the encoding process of the upper branch for the training picture of the small sample category, that is, the first contrast feature; constructing a lower branch based on the basic network of step S2, the lower branch includes a multi-scale instance target feature resampling network, and obtaining the multi-scale instance feature, that is, the second contrast feature, through the encoding process of the lower branch after the target marked in the training picture of the small sample category is cropped, rotated and enhanced;
[0010] Step S4: Calculate contrast loss and construct a few-sample target detection network, specifically, perform contrast learning and calculate contrast loss at the same time through a multi-scale candidate box contrast loss function according to the first contrast feature and the second contrast feature, and construct a few-sample target detection network according to the basic network of step S2 and the contrast loss of step S4 to train the network;
[0011] Step S5: Few-sample target detection, specifically, performing few-sample target detection on the remote sensing image to be detected through the few-sample target detection network.
[0012] Specifically, in step S2, the basic network is the Faster RCNN network, and the loss function of the model training is:
[0013]
[0014] in, represents the binary cross entropy loss; represents the cross entropy loss; Represents the bounding box regression loss, using the SmoothL1 loss function.
[0015] Specifically, in step S3, the upper branch includes Resnet101, FPN, RPN, RoIAlign, Box Head and MLP;
[0016] Among them, Resnet101: a fast training residual network used to extract the features of remote sensing images and the features of instance targets of training images;
[0017] FPN: Feature Pyramid Network, used to extract multi-scale features of training images and multi-scale features of instance objects in training images;
[0018] RPN: Region Candidate Network, used for target detection network to generate candidate box regions for images;
[0019] RoIAlign: Boundary regression network, used to map the candidate box to the feature map of the corresponding scale to obtain the candidate box features of the candidate box area;
[0020] BoxHead: Linear regression network, used to linearize the two-dimensional candidate box features to obtain linear features;
[0021] MLP: Multi-layer perceptron, used to obtain normalized features of multi-scale instance targets and candidate box areas, that is, contrast features.
[0022] Specifically, the process of encoding the first contrast feature is as follows:
[0023] y p =Mlp(B head (ROI(X)));
[0024] Where X represents the training image input to the target detection network; ROI() represents the process by which RoIAlign obtains the features of the candidate box area by mapping the candidate box to the feature map of the corresponding scale; B head () indicates the process of Box Head linearizing features to obtain linear features; MLP() indicates the process of multi-layer perceptron converting linear features into contrast features for model contrast learning; p Represents the first contrast feature.
[0025] Specifically, in step S3, the lower branch includes Resnet101, FPN, MSIFR, Box Head and MLP;
[0026] Among them, MSIFR: Multi-scale instance target feature resampling network, used to resample the obtained multi-scale instance target features.
[0027] Specifically, the process of encoding the second contrast feature is as follows:
[0028]
[0029] Among them, Aug() represents the process of instance enhancement of the target in the training image; f() represents the process of FPN extracting multi-scale features of the instance target of the training image; represents the second contrast feature, i represents the scale features of P2, P3, P4, and P5, that is, i is 2, 3, 4, and 5.
[0030] Specifically, in step S4, the multi-scale candidate box contrast loss function is as follows:
[0031]
[0032] in, Represents y p Positive candidate box features greater than the threshold σ; Represents y pThe negative candidate box features that are less than the threshold σ; δ(,) represents the cosine similarity function; n is the number of positive candidate box features; m is the number of negative candidate box features; N represents the batch size of model training; min() represents the minimum function.
[0033] Specifically, the threshold σ is the intersection-over-union ratio between the candidate box and the true label.
[0034] In addition, the present invention also provides an electronic device, comprising:
[0035] Memory for storing computer programs;
[0036] A processor is used to implement the few-sample target detection method as described above when executing the computer program.
[0037] In addition, the present invention also proposes a computer medium, on which a computer program is stored, and when the computer program is executed by a processor, the few-sample target detection method as described above is implemented.
[0038] The application of the technical solution of the present invention has the following beneficial effects:
[0039] The present invention proposes a few-sample target detection method, an electronic device and a computer medium. The present invention uses the features in the candidate frame of the target detection algorithm as the first contrast feature of contrastive learning; uses the multi-scale instance features extracted from the instance target in the few-sample training data as the second contrast feature of the model contrastive learning; and performs contrastive learning on the two contrast features, thereby improving the feature expression ability of the few-sample target detection network and weakening the influence of the direction and scale of the remote sensing image target on the accuracy.
[0040] The present invention designs a multi-scale candidate box contrast loss function for multi-scale contrastive learning, which is used for contrastive learning self-supervised training to eliminate the scale difference between two contrast features used for contrastive learning; the multi-scale candidate box contrast loss function compares the candidate box features with the multi-scale instance target features, and the multi-scale candidate box contrast loss function takes into account the multi-scale features, and returns the feature with the smallest loss as the loss result.
[0041] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0043] Figure 1It is a flowchart of the steps of the few-shot target detection method;
[0044] Figure 2 It is a schematic diagram of model training for the few-shot target detection method. DETAILED DESCRIPTION
[0045] In view of the problems in the prior art of insufficient feature representation of few-sample target detection and the model not paying attention to image characteristics, the present invention proposes a few-sample target detection method. The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. However, the present invention can be implemented in a variety of different ways as defined and covered by the claims.
[0046] Embodiment 1:
[0047] This embodiment implements a few-sample target detection method, such as Figure 1 As shown, the specific steps are as follows:
[0048] Step S1: remote sensing image classification, specifically, obtaining remote sensing images, and simulating the classification of the categories contained in the remote sensing images into basic categories and small sample categories, wherein remote sensing image categories with more than 100 instance images in the data are classified as basic category data, and remote sensing image categories with no more than 100 instance images in the data are classified as small sample categories; the basic category data includes all labeled samples of the category in the data, and the small sample category only uses pictures containing k instances in the data;
[0049] Step S2: training a basic network, specifically, training a basic network on the basic categories divided in step S1, which is used as initialization parameters for training a small sample model;
[0050] Step S3: Obtaining contrast features, specifically, constructing an upper branch in the basic network of step S2, the upper branch includes Resnet101, FPN, RPN, RoIAlign, Box Head and MLP, and the training picture of the small sample category is encoded by the upper branch to obtain the features in the candidate box in the remote sensing image, that is, the first contrast feature; constructing a lower branch in the target detection network of step S2, the lower branch includes Resnet101, FPN, MSIFR, Box Head and MLP, and after the target marked in the training picture is cropped, rotated and enhanced, the multi-scale instance feature is obtained by the encoding process of the lower branch, that is, the second contrast feature;
[0051] Step S4: calculating contrast loss and constructing a few-sample target detection network, specifically, performing contrast learning and calculating contrast loss at the same time through a multi-scale candidate box contrast loss function according to the first contrast feature and the second contrast feature, and constructing a few-sample target detection network according to the basic network of step S2 and the contrast loss of step S4 to train the model;
[0052] Step S5: Few-sample target detection, specifically, performing few-sample target detection on the remote sensing image to be detected through the few-sample target detection network.
[0053] Specifically, in step S2, the basic network is the Faster RCNN network, and the loss function of the model training is:
[0054]
[0055] in, represents the binary cross entropy loss; represents the cross entropy loss; Represents the bounding box regression loss, specifically the SmoothL1 loss function.
[0056] like Figure 2 The model training schematic diagram of the few-sample target detection method shown in FIG. 1 is as follows. In step S3,
[0057] Resnet101: A fast-trained residual network for extracting features of remote sensing images and features of instance targets in training images;
[0058] FPN: Feature Pyramid Network, used to extract multi-scale features of training images and multi-scale features of instance objects in training images;
[0059] RPN: Region Candidate Network, used for target detection network to generate candidate box regions for images;
[0060] RoIAlign: Boundary regression network, used to map the candidate box to the feature map of the corresponding scale to obtain the candidate box features of the candidate box area;
[0061] BoxHead: Linear regression network, used to linearize the two-dimensional candidate box features to obtain linear features;
[0062] MLP: Multi-layer perceptron, used to obtain normalized features of multi-scale instance targets and candidate box regions, i.e., contrast features;
[0063] Furthermore, the process of encoding the first contrast feature is as follows:
[0064] y p =Mlp(B head (ROI(X)));
[0065] Where X represents the training image input to the target detection network; ROI() represents the process by which RoIAlign obtains the features of the candidate box area by mapping the candidate box to the feature map of the corresponding scale; B head() indicates the process of Box Head linearizing features to obtain linear features; MLP() indicates the process of multi-layer perceptron converting linear features into contrast features for model contrast learning; p Represents the first contrast feature.
[0066] Specifically, in step S3,
[0067] MSIFR: Multi-scale instance target feature resampling network, used to resample the obtained multi-scale instance target features;
[0068] Furthermore, the process of encoding the second contrast feature is as follows:
[0069]
[0070] Among them, Aug() represents the process of instance enhancement of the target in the training image; f() represents the process of FPN extracting multi-scale features of the instance target of the training image; represents the second contrast feature, i represents scale features such as P2, P3, P4, P5, that is, i=2, 3, 4, 5.
[0071] Specifically, in step S4, the multi-scale candidate box contrast loss function is as follows:
[0072]
[0073] in, Represents y p Positive candidate box features greater than the threshold σ; Represents y p The negative candidate box features that are less than the threshold σ; δ(,) represents the cosine similarity function; n is the number of positive candidate box features; m is the number of negative candidate box features; N represents the batch size of model training; min() represents the minimum function; e is a natural constant.
[0074] It should be noted that the multi-scale candidate box contrast loss function in this embodiment compares the candidate box features with the multi-scale instance target features. The multi-scale candidate box contrast loss function takes into account multi-scale features and returns the feature with the smallest loss as the loss result.
[0075] The threshold σ preferably in this embodiment is the intersection-over-union ratio between the candidate box and the true label.
[0076] In order to better illustrate the advantages and purposes of this embodiment, the following will be described in combination with specific test results and calculation results of the prior art:
[0077] In step S1, the remote sensing image selected in this example is the DIOR remote sensing image dataset, which contains 20 categories, of which 15 are basic categories, including airplanes, airports, dams, highway service areas, highway toll stations, ports, golf courses, ground track and field stadiums, overpasses, stadiums, storage tanks, tennis courts, railway stations, vehicles and wind mills; 5 are small sample categories, including baseball fields, basketball courts, bridges, chimneys and ships. It should be noted that the number of training images of small sample categories in this example is represented as k (k in this experiment is set to 1, 3, 5, 10, and 20 respectively).
[0078] In step S2, a FasterRCNN network is trained as the basic network based on all the data in the 15 basic categories divided in step S1.
[0079] In step S3, the upper branch and the lower branch in this embodiment are constructed based on the basic network trained in step S2, and the training images in the small sample category are respectively processed through the upper branch and the lower branch to obtain the first contrast feature and the second contrast feature.
[0080] In step S4, the first contrast feature and the second contrast feature are contrastively learned according to the multi-scale candidate box contrast loss function. It should be noted that the threshold σ of the intersection-over-union ratio between the candidate box and the true label selected in this example is 0.4.
[0081] In step S5, a few-sample target detection network is constructed based on the basic network in step S2 and the contrastive learning in step S4 to perform few-sample target detection of remote sensing images.
[0082] Table 1 Comparison of accuracy evaluation of different few-shot object detection methods on the DIOR dataset. The accuracy reported by all models is the average percentage accuracy obtained from three random sampling trainings.
[0083]
[0084] Note: Meta RCNN, RepMet, MPSR, FsDet, P-CNN, TFA, and FSCE are all target detection methods in the prior art, among which TFA is a method based on fine-tuning, Meta RCNN is a method based on metric learning, and RepMet, MPSR, FsDet, P-CNN, and FSCE are methods based on model improvement. OURS is the target detection method proposed in this embodiment. Split1, Split2, Split3, and Split4 represent the division of four different basic categories and small sample categories, respectively.
[0085] As can be seen from Table 1, the few-shot target detection method based on multi-scale target contrastive learning proposed in this embodiment has the highest accuracy compared with other methods under different category divisions, except that the accuracy is 0.1% less than FSCE under the setting of split41shots. When the categories to be detected are relatively simple (categories included in split1), the few-shot target detection based on multi-scale target contrastive learning has better performance than other types of methods, and the overall accuracy is the highest.
[0086] Table 2 Ablation experiments of the main components of the model
[0087] baseline Contrastive Aug Muti-Scale nAp50 √ × × × 38.319 √ √ × × 39.448 √ √ × √ 40.149 √ √ √ × 40.408 √ √ √ √ 41.334
[0088] Note: baseline represents the improved baseline model; Contrastive is the target contrast learning in this case; Aug is instance target enhancement; Muti-Scale represents multi-scale instance target features; nAp50 represents the accuracy evaluation index on the small sample category; '√' in Table 2 means using the corresponding strategy, and '×' means not using the corresponding strategy.
[0089] As can be seen from Table 2, the multi-scale target contrast learning few-sample target detection method includes three components: self-supervised contrast learning, instance target enhancement, and multi-scale feature contrast. On the basis of the Baseline, ablation experiments were carried out on the three components on the DIOR dataset, and the specific results are shown in Table 2. After using target contrast learning on the basis of the baseline, the few-sample target detection model has the most obvious improvement in detection effect due to the improvement of feature extraction ability; the enhancement of instance targets and the contrast of multi-scale features make the model have a stronger response to complex remote sensing image ground objects, and correspondingly improve the generalization of the few-sample target detection model to remote sensing images. When target enhancement and multi-scale feature contrast are used in target contrast learning, the few-sample target detection effect is best. It shows that this embodiment has a positive effect on few-sample target detection.
[0090] This embodiment implements a few-sample target detection method, electronic device and computer medium. This embodiment uses the features in the candidate box of the two-stage target detection as the first contrast feature of contrast learning; uses the multi-scale instance features extracted from the instance target in the few-sample training data as the second contrast feature of the model contrast learning; and conducts contrast learning of the two contrast features, thereby improving the feature expression ability of the few-sample target detection network and weakening the influence of the direction and scale of the remote sensing image target on the accuracy. In addition, this embodiment designs a multi-scale candidate box contrast loss function for multi-scale contrast learning, which is used for contrast learning self-supervised training to eliminate the scale difference between the two contrast features used for contrast learning.
[0091] Embodiment 2:
[0092] This embodiment implements an electronic device, including:
[0093] Memory for storing computer programs;
[0094] A processor is used to implement the few-sample target detection method as described in Example 1 when executing the computer program.
[0095] Embodiment 3:
[0096] This embodiment implements a computer medium, on which a computer program is stored. When the computer program is executed by a processor, the few-sample target detection method as described in Embodiment 1 is implemented.
[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0099] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A few-sample target detection method, characterized in that: Here are the steps: Step S1: remote sensing image classification, specifically, the categories contained in the remote sensing image are simulated and divided into basic categories and small sample categories; Step S2: training a basic network, specifically, training a basic network on the samples included in the basic categories divided in step S1, which is used as initialization parameters for training a small sample model; Step S3: Obtaining contrast features, specifically, constructing an upper branch based on the basic network in step S2, and obtaining the features in the candidate frame in the remote sensing image through the encoding process of the upper branch for the training image of the small sample category, that is, the first contrast feature; A lower branch is constructed based on the basic network in step S2, wherein the lower branch includes a multi-scale instance target feature resampling network, and the target annotated in the training picture of the small sample category is cropped, rotated and enhanced, and then a multi-scale instance feature, i.e., a second contrast feature, is obtained through an encoding process of the lower branch; Step S4: Calculate contrast loss and construct a few-sample target detection network, specifically, perform contrast learning and calculate contrast loss at the same time through a multi-scale candidate box contrast loss function according to the first contrast feature and the second contrast feature, and construct a few-sample target detection network according to the basic network of step S2 and the contrast loss of step S4 to train the network; Step S5: Few-sample target detection, specifically, performing few-sample target detection on the remote sensing image to be detected through the few-sample target detection network.
2. The few-sample target detection method according to claim 1, characterized in that: In step S2, the basic network is the Faster RCNN network, and the loss function of the model training is: in, represents the binary cross entropy loss; represents the cross entropy loss; Represents the bounding box regression loss, specifically the Smooth L1 loss function.
3. The method for detecting a small number of targets according to claim 2, wherein: In step S3, the upper branch includes Resnet101, FPN, RPN, RoIAlign, Box Head and MLP; Among them, Resnet101: a fast training residual network used to extract the features of remote sensing images and the features of instance targets of training images; FPN: Feature Pyramid Network, used to extract multi-scale features of training images and multi-scale features of instance objects in training images; RPN: Region Candidate Network, used for target detection network to generate candidate box regions for images; RoIAlign: Boundary regression network, used to map the candidate box to the feature map of the corresponding scale to obtain the candidate box features of the candidate box area; Box Head: Linear regression network, used to linearize the two-dimensional candidate box features to obtain linear features; MLP: Multi-layer perceptron, used to obtain normalized features of multi-scale instance targets and candidate box areas, that is, contrast features.
4. The method for detecting a small number of targets according to claim 3, wherein: The process of the first contrast feature encoding is as follows: and p =Mlp(B head (ROI(X))); Where X represents the training image input to the target detection network; ROI() represents the process of RoI Align obtaining the regional features of the candidate box by mapping the candidate box to the feature map of the corresponding scale; B head () indicates the process of Box Head linearizing features to obtain linear features; MLP() indicates the process of multi-layer perceptron converting linear features into contrast features for model contrast learning; p Represents the first contrast feature.
5. The method for detecting a small number of targets according to claim 4, characterized in that: In step S3, the lower branch includes Resnet101, FPN, MSIFR, Box Head and MLP; Among them, MSIFR: Multi-scale instance target feature resampling network, used to resample the obtained multi-scale instance target features.
6. The few-sample target detection method according to claim 5, characterized in that: The process of the second contrast feature encoding is as follows: Among them, Aug() represents the process of instance enhancement of the target in the training image; f() represents the process of FPN extracting multi-scale features of the instance target of the training image; represents the second contrast feature, i represents the P2, P3, P4, P5 scale features, that is, i=2, 3, 4, 5.
7. The method for detecting a small number of targets according to claim 6, wherein: In step S4, the multi-scale candidate box contrast loss function is as follows: in, Represents y p Positive candidate box features greater than the threshold σ; Represents y p The negative candidate box features that are less than the threshold σ; δ(,) represents the cosine similarity function; n is the number of positive candidate box features; m is the number of negative candidate box features; N represents the batch size of model training; min() represents the minimum function.
8. The method for detecting a small number of targets according to claim 7, characterized in that: The threshold σ is the intersection-over-union ratio between the candidate box and the true label.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the few-sample target detection method according to any one of claims 1 to 8 when executing the computer program.
10. A computer medium, characterized in that The computer medium stores a computer program, and when the computer program is executed by the processor, the method for detecting a small number of targets according to any one of claims 1 to 8 is implemented.
