Remote sensing few-sample target detection method based on YOLOv7

Abstract

The invention provides a remote sensing few-sample target detection method based on YOLOv7. The whole training process is divided into two stages, firstly, a YOLOv7 network is used for training rich base class data, and a model with good base class detection performance is obtained; and secondly, training new class data by using the model obtained in the previous stage according to a K-shot normal form and through a YOLOv7 network added with a deviation module, and finally obtaining a model which not only can maintain a good detection effect on a base class, but also can detect a new class. The method is mainly applied to a remote sensing video monitoring intelligent analysis application system, and has a wide application prospect in the fields of military, agriculture, aviation and the like under the advantage of high detection speed.

Description

Technical Field

[0001] The present invention relates to a remote sensing few-sample target detection method based on YOLOv7, and belongs to the field of computer vision and intelligent information processing. Background Art

[0002] With the advent of the big data era, object detection based on deep learning is no longer a fantasy. In the past decade, relevant work in this area has achieved remarkable results, but it can be generally divided into two major directions: the two-stage target detection network that pursues high precision and the one-stage target detection network that pursues high speed. The two-stage target detection network mainly generates candidate regions first, and then classifies and accurately locates the generated candidate regions. Faster R-CNN and R-CNN series methods are typical two-stage target detection methods. These networks usually have high accuracy, but the detection speed is relatively slow. On the contrary, the one-stage target detection network completes the target detection task by directly predicting the target category and bounding box in the image, omitting the stage of candidate region generation, which makes its detection faster and suitable for real-time application scenarios. YOLO (You Only Look Once) series methods and SSD (Single Shot MultiBox Detector) are representatives of one-stage target detection. Based on these networks, according to different research directions, multiple sub-fields have also been derived, such as remote sensing target detection, cross-domain target detection, and few-sample target detection.

[0003] Few Shot Object Detection in Remote Sensing Images (FSOD in RSIs) is a new direction in the field of few-shot object detection in recent years, which aims to train a robust model with limited labeled samples in remote sensing scenarios. At present, most deep learning-based object detection requires a huge amount of training data to obtain a good model, but in real situations, data in many fields is not easy to obtain, especially in remote sensing scenarios involving confidential data or new enemy equipment. In addition, a large amount of data labeling will also consume a lot of manpower and material resources. Therefore, remote sensing few-shot object detection provides a solution for the above scenarios. However, the multi-scale and dense problems of targets in remote sensing scenarios have exacerbated the difficulty of detecting such targets.

[0004] For remote sensing few-shot target detection technology, there are three main paradigms today: meta-learning, metric learning, and fine-tuning learning, and most of them are improved based on the two-stage Faster R-CNN framework. Based on the meta-learning paradigm, iterative training is performed on a task-by-task basis, and a set of initialization parameters that perform well in different tasks are learned from it. For a specific task, only a small number of iterations are needed to obtain good performance in the new task, but the overall detection accuracy of this method is not high at present; the metric learning-based method is to convert the target area features and the supporting image features into the same embedding space on the premise of obtaining the potential target area features, and classify the potential target areas by calculating the distance or similarity, thereby realizing the detection of different targets in the image; the fine-tuning learning paradigm is the most widely used method, which first pre-trains the base class data (containing a large number of annotations) and then fine-tunes on the new class (few-shot annotations) data to achieve few-shot target detection. This paradigm can achieve good detection accuracy and is a simple but relatively effective paradigm. Although remote sensing few-sample target detection technology initially focused on how to better train models for new categories, as researchers comprehensively considered and studied, they gradually focused on how to enable this model to retain good detection effects on base categories while also being able to detect new categories without forgetting prior knowledge. Summary of the invention

[0005] This paper proposes a remote sensing few-sample target detection method based on YOLOv7, aiming to utilize the powerful detection performance of YOLOv7 itself and combine it with a new fine-tuning learning paradigm, so that the network can detect targets of new classes and base classes.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] (1) Use a novel two-stage fine-tuning paradigm to complete two-stage training of training on base class data and fine-tuning training on new class data;

[0008] (2) Add the Deviation Compensation (DC) module to the network during fine-tuning training, that is, learn the deviation between the base class and the new class features from the RepConv layer in DC to increase the feature distance between classes;

[0009] (3) Finally, the above features are directly used for multi-classification and regression operations, and the model is continuously trained and iterated to obtain the final detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is the overall block diagram of the remote sensing few-sample target detection method based on YOLOv7 and the new fine-tuning paradigm of the present invention;

[0011] Figure 2 is a structural diagram of the deviation module of the present invention; DETAILED DESCRIPTION

[0012] The present invention will be further described below in conjunction with the accompanying drawings:

[0013] Figure 1 Shown is the overall network structure and new fine-tuning paradigm of the remote sensing few-sample target detection algorithm based on YOLOv7:

[0014] The remote sensing few-shot target detection framework based on YOLOv7 is trained and tested in an end-to-end manner. The network mainly needs to go through a new fine-tuning paradigm of two stages: base class training and few-shot fine-tuning training, namely, Two-stage Fine-tuning Approach to YOLOv7 (TFA-Y). In the base class training stage, YOLOv7 is used as the benchmark network, and the network is used to train the rich base class data, calculate the loss, update the network parameters, and obtain a model with good base class detection performance in preparation for the next training stage. Its loss function is shown in formula (1):

[0015] L=L cls +L obj +L box (1)

[0016] Where, L cls is the category confidence loss, L obj is the target confidence loss, L box is the coordinate regression loss.

[0017] In the few-shot fine-tuning training phase, the network adds a bias module and freezes the parameters of the Backbone, some Head and RepConv layers in the network, that is, the parameters of the 0-75 layers and 102-104 layers of the network. Following the K-shot paradigm, that is, each type of training data has only K instance annotations, and the base class and new class data are trained at the same time, the loss function is calculated using the same loss function as in formula (1), and the network parameters are updated to obtain the final detection model.

[0018] The bias module is divided into two branches: the main branch and the bias branch. The main branch is the 102-104 layers in the original YOLOv7 network, which are all RepConv layers. They will be frozen during the fine-tuning training phase to retain the model's ability to detect the base class. The bias branch has the same structure as the main branch, which is a RepConv layer. It is connected in parallel with the main branch to learn the deviation between the new class and the base class features and increase the feature distance between classes. The outputs of the main branch and the bias branch are combined and passed through the convolution function (conv) to obtain the features output by the YOLOv7 network. Figure 1The final feature map of the sample size is used to calculate the loss and update the network parameters.

[0019] The DIOR dataset contains 23,463 images, including 11,725 ​​training images and 11,738 test images. The NWPU VHR-10.v2 dataset contains 1,172 images, including 879 training images and 293 test images. All stages of the training in the experiment are performed on the training set and tested on the test set.

[0020] The present invention conducts a network structure ablation study on the DIOR dataset, where Baseline represents the baseline network YOLOv7, "TFA-Y" represents the adoption of the new fine-tuning paradigm proposed by the present invention, and "DC" represents the adoption of the deviation compensation module proposed by the present invention. In addition, it is compared with some mainstream algorithms on the DIOR dataset and the NWPU VHR-10.v2 dataset, using bAP and nAP as evaluation indicators. The experimental results are shown in Table 1, Table 2, and Table 3. The experimental accuracy is significantly improved compared with the baseline network and other comparison algorithms. During the test, for the input of 640×640, the batch size is set to 1. We compare the model parameter amount and detection speed with these mainstream algorithms, using M and fps as evaluation indicators. The experimental results are shown in Table 4. The model parameter amount is reduced compared with other comparison algorithms, and the detection speed is far higher than other comparison algorithms, which verifies that the present invention can be combined with embedded devices to achieve real-time detection.

[0021] Table 1 Ablation test results of each module

[0022]

[0023] Table 2 DIOR dataset test results

[0024]

[0025] Table 3 Test results of NWPU VHR-10.v2 dataset

[0026]

[0027] Table 4 Comparison of model parameters and inference speed

[0028]

Claims

1. A remote sensing few-sample target detection method based on YOLOv7, characterized in that The following steps are involved: (1) Construct a new two-stage fine-tuning approach to YOLOv7 (TFA-Y) to complete the two-stage training of training the base class data and fine-tuning the new class data; (2) Construct a Deviation Compensation (DC) module and add it to the network in the fine-tuning training phase. That is, learn the deviation between the base class and the new class features from the RepConv layer in DC, thereby increasing the feature distance between classes.

2. The method according to claim 1, characterized in that (1) constructs a new two-stage fine-tuning paradigm, and the construction method is as follows: The new two-stage fine-tuning paradigm is mainly aimed at the YOLOv7 network, and includes two stages: base class training and few-sample fine-tuning training, as shown in the manual. In the base class training stage, YOLOv7 is used as the benchmark network, and the network is used to train the rich base class data, calculate the loss, update the network parameters, and obtain a model with good base class detection performance in preparation for the next training stage. Its loss function is shown in formula (1): L=L cls +L obj +L box (1) Where, L cls is the category confidence loss, L obj is the target confidence loss, L box is the coordinate regression loss; In the few-shot fine-tuning training stage, the parameters of the Backbone, part of the Head and RepConv layers in the network, i.e., the parameters of the 0-75 layers and 102-104 layers of the network, are frozen; under the K-shot paradigm, i.e., each type of training data has only K instance annotations, the base class and new class data are trained at the same time, and the loss function as in formula (1) is used to calculate the loss and update the network parameters.

3. The method according to claim 1, characterized in that (2) The middle deviation module is constructed as follows: During the fine-tuning training stage, the patent of the present invention adds a deviation module to the YOLOv7 network; the deviation module is divided into two branches: a trunk branch and a deviation branch; the trunk branch is the 102-104 layers in the original YOLOv7 network, which are all RepConv layers, which will be frozen during the fine-tuning training stage to retain the model's detection ability for the base class; the deviation branch has the same structure as the trunk branch, which is a RepConv layer, connected in parallel with the trunk branch, and is used to learn the deviation between the new class and the base class features, and to increase the feature distance between classes; the outputs of the trunk branch and the deviation branch are combined and passed through a convolution function (conv) to obtain a final feature map of the same size as the feature map output by the YOLOv7 network, and the loss is calculated based on this, and the network parameters are updated to finally obtain a model that can retain a good detection effect for the base class and can also detect new classes.

Images