Lightweight remote sensing target detection method based on dense feature fusion network

Abstract

The present invention provides a lightweight remote sensing target detection method based on a dense feature fusion network. The method comprises: Step 1: constructing a backbone network, utilizing the backbone network to extract feature information of different sizes from remote sensing images; Step 2: constructing a dense feature fusion neck network, utilizing the dense feature fusion neck network to fuse the feature information of different sizes; and Step 3: sending the fused feature information to a prediction head for detection to obtain detection results. This method can better address the problem of external interference factors caused by complex and ever-changing remote sensing images, thereby extracting feature information of targets within complex backgrounds in remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing target detection, and in particular to a lightweight remote sensing target detection method based on a dense feature fusion network. Background Art

[0002] The development of remote sensing image sensors has enabled the acquisition of high-quality, high-resolution remote sensing images. Detecting targets in remote sensing images also holds significant significance in both military and civilian applications. The application of deep learning models to remote sensing target detection is attracting increasing attention.

[0003] Many detection models based on convolutional neural networks have achieved good results. However, due to the particularity of remote sensing images, ordinary target detection frameworks cannot be well applied to remote sensing images. Compared with target detection on natural images, target detection on remote sensing images usually faces the following challenges: (1) The size of remote sensing targets is usually very small compared to the image, and most existing networks have difficulty extracting features from remote sensing images, which easily leads to a high missed detection rate; (2) Remote sensing images are often interfered with by external factors such as shadows, similar objects, and complex backgrounds. This is because the network's feature extraction ability is insufficient, resulting in difficulty in distinguishing texture rules and feature information between objects and falsely detected objects. Most detection methods have difficulty extracting features on remote sensing images, and detection performance is limited. Summary of the Invention

[0004] In the process of remote sensing image target detection, the size of remote sensing targets is usually very small compared to the image, and remote sensing images are often interfered by external factors such as shadows, similar objects, and complex backgrounds, which makes remote sensing target detection difficult. Taking into account the accuracy and speed of detection, the present invention provides a lightweight remote sensing target detection method based on a dense feature fusion network.

[0005] The present invention provides a lightweight remote sensing target detection method based on a dense feature fusion network, comprising:

[0006] Step 1: Construct a backbone network and use the backbone network to extract feature information of different sizes of remote sensing images;

[0007] Step 2: Construct a dense feature fusion neck network, and use the dense feature fusion neck network to fuse feature information of different sizes;

[0008] Step 3: The fused feature information is sent to the prediction head for detection to obtain the detection results.

[0009] Furthermore, step 1 specifically includes:

[0010] The backbone network outputs feature maps of four sizes, which are in descending order: feature map C1, feature map C2, feature map C3 and feature map C4;

[0011] A first feature multiplexing module and a second feature multiplexing module are used to perform feature multiplexing on feature maps C2 and C3, respectively. The feature multiplexing process of the first feature multiplexing module is specifically as follows: the size of feature map C4 is adjusted to be consistent with the size of feature map C3, a new feature map C4' is obtained, and then the feature map C4' and the feature map C3 are concat-operated and output; the feature multiplexing process of the second feature multiplexing module is specifically as follows: the sizes of feature maps C4 and C3 are adjusted to be consistent with the size of feature map C2, new feature maps C4" and C3" are obtained, respectively, and then the feature maps C4", C3" and C2 are concat-operated and output;

[0012] The output of the first feature multiplexing module, the output of the second feature multiplexing module and the feature map C4 are the feature information of different sizes extracted from the remote sensing image.

[0013] Furthermore, the dense feature fusion neck network is a network obtained by replacing the CSP module of the feature fusion network in the original YOLOv5 network structure with the CSRDB module;

[0014] Among them, the CSRDB module includes two branches, the upper branch includes the first CBS unit, the RDB unit and the first convolutional layer in sequence; the lower branch includes the second convolutional layer; the inputs of the upper and lower branches are the same, and the outputs of the upper and lower branches are input into the second CBS unit after the concat operation.

[0015] Beneficial effects of the present invention:

[0016] (1) By adopting two feature reuse modules in the backbone network, the feature map of the backbone network is reused to fuse the semantic information and location information in the feature map, thereby enhancing the detection ability of the entire network for small and medium-sized remote sensing targets;

[0017] (2) The CSRDB module is constructed using the RDB module, and a dense feature fusion neck network is constructed based on the CSRDB module to fuse feature information of different sizes from the backbone network, which can better solve the problem of external interference factors caused by complex and changeable remote sensing images, thereby extracting the feature information of targets under complex backgrounds in remote sensing images.

[0018] Experiments on the DOTA dataset show that compared with other algorithms, the method of the present invention has better performance in multi-scale remote sensing target detection; from the ablation experiment, it can be seen that each improved module proposed by the method of the present invention can effectively improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of a lightweight remote sensing target detection method based on a dense feature fusion network provided by an embodiment of the present invention;

[0020] Figure 2 A diagram of the overall network structure constructed by the method of the present invention provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a feature multiplexing module provided in an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of the CSRDB module provided in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the structure of an RDB unit provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, the embodiment of the present invention provides a lightweight remote sensing target detection method based on a dense feature fusion network. The overall network structure constructed by this method is as follows: Figure 2 As shown, the method includes the following steps:

[0026] S101: constructing a backbone network, and using the backbone network to extract feature information of different sizes of remote sensing images;

[0027] like Figure 2 As shown, the backbone network outputs feature maps of four sizes, which are in descending order: feature map C1 (64×64), feature map C2 (32×32), feature map C3 (16×16) and feature map C4 (8×8).

[0028] Since low-level feature maps and middle-level feature maps contain less semantic information, and low-level feature maps are mainly responsible for small target detection, and middle-level feature maps are mainly responsible for medium target detection, insufficient processing of feature information in low-level feature maps and middle-level feature maps may affect the detection performance of small and medium targets.

[0029] As an implementable embodiment, the embodiment of the present invention adopts two feature reuse modules (Feature Reuse Model, FRM) in the backbone network: a first feature reuse module and a second feature reuse module; the first feature reuse module and the second feature reuse module are used to perform feature reuse on feature graph C2 and feature graph C3 respectively; these two modules provide an efficient reuse mechanism based on fusion for the backbone network.

[0030] like Figure 3 As shown, the first feature multiplexing module ( Figure 3 The feature reuse process of FRM1 in the example is as follows: the size of the feature map C4 is adjusted to be consistent with the size of the feature map C3 to obtain a new feature map C4', and then the feature map C4' and the feature map C3 are concat-operated and output; the second feature reuse module ( Figure 3 The feature reuse process of FRM2 in the figure is as follows: the sizes of the feature maps C4 and C3 are adjusted to be consistent with the size of the feature map C2, and new feature maps C4'' and C3'' are obtained respectively. Then, the feature maps C4'', C3'' and C2 are concat-operated and output.

[0031] The output of the first feature multiplexing module, the output of the second feature multiplexing module and the feature map C4 are the feature information of different sizes extracted from the remote sensing image.

[0032] This step can enrich the semantic information of small and medium targets by fusing low-level feature maps and mid-level feature maps using the feature reuse module, thereby improving detection accuracy.

[0033] It is worth noting that there are two types of reuse operations: concatenation and element-wise addition. In this embodiment of the present invention, a concat concatenation method is used to reuse the feature information of the backbone network. This is because the concat operation can fuse the extracted convolutional features, increasing the dimension while maintaining the amount of information, thereby using the reused features as the input of the dense feature fusion neck network.

[0034] S102: Constructing a dense feature fusion path aggregation network (DFF-PANet), and utilizing the dense feature fusion path aggregation network to fuse feature information of different sizes to enrich feature information;

[0035] Specifically, combined Figure 2 and Figure 4As shown in FIG, the dense feature fusion neck network is a network obtained by replacing the cross-stage partial (CSP) module of the feature fusion network in the original YOLOv5 network structure with the CSRDB module;

[0036] The CSP module in the original YOLOv5 network structure can be referred to the literature "BochkovskiyA, Wang CY, LiaoH. YOLOv4: Optimal Speed ​​and Accuracy of Object Detection[J]. 2020.", which will not be repeated here.

[0037] Among them, Figure 4 As shown in the figure, the CSRDB (Cross Stage Residual Dense Block) module includes two branches, the upper branch includes the first CBS unit, the RDB unit and the first convolutional layer in sequence; the lower branch includes the second convolutional layer; the inputs of the upper and lower branches are the same, and the outputs of the upper and lower branches are input to the second CBS unit after the concat operation.

[0038] Among them, the RDB unit refers to the residual dense block (RDB), whose structure is as follows Figure 5 As shown in Figure 2. The residual dense block consists of dense connection layers, local feature fusion, and local residual learning. These three parts constitute a continuous memory mechanism. The so-called continuous memory mechanism is to pass the state of the previous convolutional layer to the current convolutional layer. Let F RDB-1 and F RDB These are the previous layer of the residual dense block (i.e., the output of the CBS layer) and the next layer of the residual dense block (i.e., the input of the Conv layer). The residual dense block extracts rich local features from the feature map through densely connected layers. It also allows the previous convolutional layer to be directly connected to the current convolutional layer to form a continuous memory mechanism. Local feature fusion is then used to enable the residual dense block to adaptively learn effective local features. After fully acquiring local dense features, global feature fusion is used to retain the accumulated features in a holistic manner and learn effective global features.

[0039] In the embodiment of the present invention, in the RDB unit used to construct the CSRDB module, the dense connection layer is composed of 6 convolutions densely connected, with a growth rate of 32. RDB,1 represents the output of the first convolution, F RDB,C Represents the output of any convolution in the middle. In this invention, C∈{2,3,4,5}, F RDB,6Represents the output of the last convolution. Taking any convolution as an example, the output of this convolution is obtained by convolving the previous layer of the residual dense block, the convolution within the residual dense block, and concatenating the first convolution to the previous convolution of this convolution. This convolution is then weighted by the convolution layer and finally activated by the ReLU function. All feature information in the residual dense block is locally fused using concatenation. Furthermore, a 1×1 convolution layer is introduced to reduce the number of features and adaptively control the output information. Local residual learning incorporates not only the features before the residual dense block, but also the local dense features extracted by the residual dense block.

[0040] Most remote sensing targets in remote sensing images have different aspect ratios. However, ordinary convolution cannot fully utilize the hierarchical features in the original feature map, thus only achieving low performance. In the embodiment of the present invention, in order to better fuse the feature information from the backbone network, the residual dense block (RDB) is used to construct the CSRDB module, thereby replacing the original CSP module with the CSRDB module. In this way, the powerful feature fusion capability of the residual dense block can be used to fully fuse the feature information extracted by the backbone network.

[0041] S103: The fused feature information is sent to the prediction head (Predict) for detection to obtain the detection result.

[0042] First, in order to enhance the network's detection capability for small and medium-sized remote sensing targets, the feature map of the backbone network is reused to fuse the semantic information and location information in the feature map, thereby enriching the feature information extracted by the backbone network.

[0043] Second, unlike the classic feature fusion pyramid network, the present invention uses the CSRDB module in the feature fusion network to design a dense feature fusion neck network, which can better solve the problem of external interference factors caused by complex and changeable remote sensing images.

[0044] In addition, to verify the effectiveness of the remote sensing target detection method of the present invention, the following experiments were conducted using the challenging DOTA dataset. A series of comparative experiments and ablation experiments were also conducted to evaluate and analyze the detection performance of the present invention after the two improvements mentioned above. The experimental results show that the lightweight remote sensing target detection method based on the dense feature fusion network proposed in this invention can perform accurate and real-time detection on large-scale remote sensing images. The relevant experimental data are shown in Table 1:

[0045] Table 1 Ablation experiment results

[0046]

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A lightweight remote sensing target detection method based on a dense feature fusion network, characterized by: include: Step 1: Construct a backbone network and use the backbone network to extract feature information of different sizes of remote sensing images; specifically, the backbone network outputs feature maps of four sizes, which are in descending order of size: feature map C1, feature map C2, feature map C3, and feature map C4; A first feature multiplexing module and a second feature multiplexing module are used to perform feature multiplexing on the feature map C2 and the feature map C3 respectively; wherein, the feature multiplexing process of the first feature multiplexing module is specifically as follows: the size of the feature map C4 is adjusted to be consistent with the size of the feature map C3, a new feature map C4' is obtained, and then the feature map C4' and the feature map C3 are concat-operated and output; the feature multiplexing process of the second feature multiplexing module is specifically as follows: the sizes of the feature maps C4 and C3 are adjusted to be consistent with the size of the feature map C2, new feature maps C4'' and C3'' are obtained respectively, and then the feature maps C4'', C3'' and C2 are concat-operated and output; The output of the first feature multiplexing module, the output of the second feature multiplexing module and the feature map C4 are the feature information of different sizes extracted from the remote sensing image; Step 2: Construct a dense feature fusion neck network, and use the dense feature fusion neck network to fuse feature information of different sizes. The dense feature fusion neck network is a network obtained by replacing the CSP module of the feature fusion network in the original YOLOv5 network structure with a CSRDB module. The CSRDB module includes two branches, the upper branch, which includes the first CBS unit, the RDB unit, and the first convolutional layer in sequence; the lower branch includes the second convolutional layer. The inputs of the upper and lower branches are the same, and the outputs of the upper and lower branches are input to the second CBS unit after the concat operation. Step 3: The fused feature information is sent to the prediction head for detection to obtain the detection results.

Images