A CNN+Transformer remote sensing image detection method

Abstract

The invention discloses a CNN+Transformer remote sensing image target detection method, which is applied to the target detection field. The existing target detection algorithm based on convolutional neural network has the problem of low detection accuracy. The invention first constructs a CNN and Transformer parallel target detection network, improves the effect of multi-scale target detection task by using global information and local information interaction, fuses multi-scale information by using top-down and bottom-up paths in the network, and finally completes feature selection by using coordinate attention CA mechanism, assigns adaptive weight to each channel, and realizes accurate detection and positioning of multi-scale targets in remote sensing images. The method of the invention combines the respective advantages of CNN and Transformer, accurately locates the target area, can highly guarantee the detection rate after training, and realizes accurate detection of remote sensing image targets.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a remote sensing image multi-scale target detection technology. Background Art

[0002] At present, remote sensing technology has played a vital role in many fields, such as ocean monitoring, geological survey, military target detection, natural disaster monitoring, etc. As an important direction in the development of remote sensing technology, remote sensing image target detection has the main task of finding the location of the target object in the remote sensing image and identifying the category of the target object. Due to the characteristics of remote sensing images such as complex background, diverse target sizes and low pixel size of small targets, the traditional target detection method uses a sliding window to traverse the original remote sensing image to extract the target frame, which is prone to problems such as long time consumption and low detection accuracy. With the rapid development of deep learning and the introduction of convolutional neural network (CNN), an end-to-end artificial intelligence model has been provided for the field of computer vision. Since convolutional neural networks have the characteristics of automatic learning feature extraction and high detection accuracy, target detection algorithms based on convolutional neural networks are gradually replacing traditional target detection algorithms and becoming the current mainstream target detection algorithms.

[0003] At this time, remote sensing image target detection mostly uses context information, attention mechanism and multi-scale feature fusion to solve the multi-scale problem of remote sensing images. It was not until the Transformer, a novel architecture in the field of natural language processing, was introduced into the field of target detection that further development in this field was achieved. At the same time, DETR proposed the earliest model that brought Transformer into the field of target detection in 2020, which achieved an accuracy comparable to Faster R-CNN. It first extracts features with CNN and flattens the output into a sequence vector, and then processes the output through Transformer to obtain the final prediction result.

[0004] The paper "Xiao T, Singh M, Mintun E, et al. Early Convolutions Help Transformers See Better, 2021, 14881." proposed a new hybrid architecture called CMT for visual recognition and other downstream visual tasks to address the limitations of using Transformers in a crude way in the field of computer vision. The proposed CMT simultaneously leverages the advantages of CNN and Transformers to capture local and global information and promote the representation ability of the network. However, the weight sharing method used by the model may cause some information loss because they are shared in multiple locations in the model.

[0005] The document "Li J, Xia X, Li W, et al. Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios, 2022, 4698." proposed the Next-ViT model, and proposed two information learning paradigms, NCB and NTB, to capture local representation and global information. Finally, the hybrid paradigm was explored to obtain an effective deployment architecture model for CNN and Transformer. Due to its limitations in applicability and versatility, the Next-ViT model may not perform as well as other advanced visual models on some datasets. The paper demonstrates the advantages of the model in CPU-based deployment, but whether its performance is competitive for deployment based on GPU or other accelerators still needs further evaluation. The above models are mostly used in the field of image classification, and their use in the field of object detection is still in the development stage. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a multi-scale target detection method for remote sensing images with a CNN+Transformer parallel structure.

[0007] The technical solution adopted by the present invention is: a multi-scale target detection method for remote sensing images with a CNN+Transformer parallel structure, and the specific steps are as follows:

[0008] S1. Construct a multi-layer CNN convolutional layer structure;

[0009] S2, build a multi-layer Transformer structure;

[0010] S3, building an information exchange path for the structures of steps S1 and S2 to obtain a preliminary parallel backbone network;

[0011] S4, constructing a feature pyramid structure for the preliminary parallel backbone network obtained in step S3 to form a neck network;

[0012] S5, connecting the CA attention module to the network in step S4 to construct a deep neural network;

[0013] S6, collecting remote sensing image target detection data, and generating a sample training set, a validation set, and a test set based on the obtained remote sensing image target detection data set;

[0014] S7, training the deep neural network constructed in step S5 according to the sample training set and verification set in step S6;

[0015] S8. Input the test set into the deep neural network trained in step S7, so as to perform target detection on the remote sensing image.

[0016] Beneficial effects of the present invention: The method of the present invention first constructs a target detection network in which CNN and Transformer are in parallel, and further improves the effect of the multi-scale target detection task by using the interaction of global information and local information, and uses top-down and bottom-up paths in the network to fuse multi-scale information, and finally uses the coordinate attention CA mechanism to complete feature selection, and assigns adaptive weights to each channel to achieve accurate detection and positioning of multi-scale targets in remote sensing images. Then, the remote sensing image target detection data set can be used to generate sample training sets, verification sets, and test sets, and the method can be used to complete the target detection task of remote sensing images. The method of the present invention combines the respective advantages of CNN and Transformer to accurately locate the target area, and can guarantee a high detection rate after sample training, thereby achieving accurate detection of multi-scale targets in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a multi-scale target detection method for remote sensing images with a CNN+Transformer parallel structure of the present invention.

[0018] Figure 2 A schematic diagram of an information exchange path provided by an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the CA attention mechanism provided for an embodiment of the present invention.

[0020] Figure 4 A schematic diagram of the neural network structure SPP structure provided in an embodiment of the present invention.

[0021] Figure 5 A schematic diagram of multi-scale target detection results of remote sensing images provided by an embodiment of the present invention;

[0022] in, Figure 5 (a) represents the ground-truth box of the original image, Figure 5 (b) shows the detection results of Faster-RCNN-FPN. Figure 5 (c) shows the detection result of YoLov3. Figure 5 (d) shows the detection results of the present invention. DETAILED DESCRIPTION

[0023] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0024] like Figure 1As shown in the flowchart of a remote sensing image multi-scale target detection method with a CNN+Transformer parallel structure of the present invention, the specific steps are as follows:

[0025] S1. Construct a multi-layer CNN convolutional layer structure;

[0026] S2, build a multi-layer Transformer structure;

[0027] S3, constructing an information exchange path for the structures of steps S1 and S2 to obtain a preliminary parallel backbone network;

[0028] S4, constructing a feature pyramid structure for the preliminary parallel backbone network obtained in step S3 to form a neck network;

[0029] S5, connecting the CA attention module to the network in step S4 to construct a deep neural network;

[0030] S6. In this embodiment, a remote sensing image target detection data set is used to reasonably generate a sample training set, a validation set, and a test set;

[0031] S7. Train deep neural networks.

[0032] In this embodiment, in step S1, five convolutional layers are constructed by introducing residual connections, the first convolutional layer is mainly used for channel adjustment, and the rest are used to extract local information.

[0033] In this embodiment, in step S2, a four-layer Transformer structure is formed using LayerNorm (layer normalization), MHSF (Multi-Head Self-Attention) and MLP (Multi-Layer Perceptron) to extract global information.

[0034] In this embodiment, in step S3, dimension transformation is performed by interpolation and down sampling to achieve information exchange, such as Figure 2 As shown in the figure, Maxpooling represents maximum pooling, Interpolating represents the nearest neighbor interpolation operation, and Reshape represents the dimension adjustment operation.

[0035] Due to the different computational characteristics of CNN and Transformer, the feature map of CNN is three-dimensional, while the feature map of Transformer is two-dimensional. Therefore, when information interacts, the feature maps of both need to be deformed. When the local information of CNN is fused to the Transformer branch, the feature map is firstly subjected to the maximum pooling operation. The maximum pooling formula is as follows:

[0036]

[0037] Input is the input feature map, output is the result after pooling, m and n are the positions of the output feature map, and p and q are the positions within the pooling window.

[0038] The maximum pooling operation takes the maximum value in each pooling window on the input feature map as output, so each element in the output feature map corresponds to the maximum value in a pooling window in the original input map. Usually, the size of the maximum pooling window and the pooling stride are fixed and can be adjusted according to the needs of the network.

[0039] Downsampling is completed through maximum pooling, and then the channel dimension and height dimension of each pixel are saved for deformation operation, and the width dimension is erased. When the Transformer global information is fused to the CNN branch, it is the inverse operation of the above operation. First, the feature map is deformed to construct the width dimension, and then the nearest neighbor interpolation operation is performed to upsample the feature map to complete the final information interaction. The nearest neighbor interpolation formula is as follows:

[0040] x out (m,n)=x in (round(m / scale),round(n / scale))(2)

[0041] x in and x out Represents the pixel values ​​of input and output respectively, round is the rounding function, and scale is the scaling factor between input and output.

[0042] This information interaction ensures that global information always exists in the CNN branch, which can effectively improve the ability to express target features. For example, in the sea, ships are more likely to appear, so the ships not only have their own characteristics, but also the information of the scene is integrated. The same principle applies to airplanes at airports and oil tanks at factories.

[0043] In this embodiment, in step S4, the backbone network obtained in step S3 extracts features of different levels (P2, P3, P4, P5). Since P5 is the deepest feature, P5 is specified as the top and P2 as the bottom. The top-down path uses the interpolation method to transfer information, and passes the deep semantic information to the shallow layer for fusion (addition operation of corresponding elements at each position) to obtain the {C2, C3, C4, C5} feature map. The bottom-down path passes the shallow position information to the deep network to improve the accuracy of positioning, and uses 3×3 convolution to downsample the shallow feature map to complete the matching and fusion of the feature map.

[0044] In this embodiment, in step S5, a CA attention mechanism (such as Figure 3 As shown in (a), feature selection of the feature map in S4 is realized. Figure 3 Input and output represent the input feature map and output feature map respectively; H, W, C and r represent the height dimension, width dimension, channel dimension and reduced scale of the feature map respectively; Residual represents the residual structure; AvgPool represents average pooling; Concat+Conv2d represents concatenation plus two-dimensional convolution operation; BatchNorm+Non-linear represents channel normalization and nonlinear operation; Sigmoid represents activation function; Re-Weight represents reweighting.

[0045] like Figure 4 As shown in FIG. 1 , the specific network structure of this embodiment, wherein the convolutional layer representation method is “CNN block”, Figure 4 In the example, the stem module is used to represent the network, which contains three convolution operations. The convolution operation is represented by "Conv.(convolution kernel size))@(convolution kernel size)". The Transformer layer is represented by "Transformer block". Figure 4 The trans module is used in the code, which consists of LayerNorm, MHSF and MLP. Coordinate Attention is the coordinate attention mechanism. Figure 4 In the example, CA is used to represent it. feature map represents feature map. {P2, P3, P4, P5} and {C2, C3, C4, C5} represent feature maps of different scales. Finally, after adjustment through a 1×1 convolutional layer, three feature maps of different scales are obtained. classfication and regression represent classifier and regressor respectively.

[0046] In this embodiment, in step S6, a remote sensing image target detection dataset is used to reasonably generate a sample training set, a validation set, and a test set. In this embodiment, a LEVIR remote sensing image dataset is used, which contains aircraft, ships, and oil tank targets of different scales, and the images are divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0047] In this embodiment, in step S7, the training set obtained in step S6 is input into the deep convolutional neural network constructed in step S5 for forward propagation, the cost function value is calculated, and the parameters of the multi-input parallel deep convolutional neural network are updated using a backward propagation algorithm based on gradient descent, and forward and backward propagation are iteratively performed until the cost function converges. The specific steps are as follows:

[0048] S71, forward propagation,

[0049] by represents the nth feature map of the mth layer (m≥2). If the mth layer is a convolutional layer, then:

[0050]

[0051] in, represents the convolution kernel connecting the s-th input feature map and the n-th output feature map, represents the bias term, σ(·) represents the nonlinear activation function, and the symbol “*” represents the convolution operation;

[0052] If the mth layer is the maximum pooling layer, then:

[0053]

[0054] in, represents the coordinate value of the position (x′, y′) in the nth feature map of the mth layer (m ≥ 2), r1 and r2 represent the size of the pooling window, and u and v represent the variables pre-set in the pooling window;

[0055] If the mth layer is an average pooling layer, then:

[0056]

[0057] Among them, r1 and r2 represent the size of the pooling window;

[0058] If the mth layer is a fully connected layer, then:

[0059] f (m) =σ(w (m) f (m-1) +b (m) )(7)

[0060] Among them, f (m) represents the feature map of the mth layer, w (m) represents the weight of the mth layer, b (m) is the bias term for this layer.

[0061] S72, calculating the cost function value,

[0062] Taking the cross entropy function as the cost function, the calculation formula is:

[0063]

[0064] The first two terms of formula (8) represent the regression bounding box loss, and the third and fourth terms represent the confidence loss. The confidence losses of positive samples (there are objects in the unit grid) and negative samples (there are no objects in the unit grid, which is the background) are calculated. The final feature map is divided into M×M grids, and each grid unit can predict A bounding boxes. (x i ,y i ) represents the coordinates of the center of the target bounding box in the i-th unit grid, (w i ,h i ) represents the width and height of the target bounding box in the i-th unit grid, then the actual position coordinates are available (x i ,y i ,w i ,h i )express, Represents the predicted position coordinates; C i Indicates whether there is a real target in the grid cell. If so, it is 1, and if not, it is 0. Represents the prediction result within the grid cell, if there is a target, it is 1, if not, it is 0; represents the cell grid i with objects, represents the jth bounding box in grid cell i that is responsible for predicting the object, represents the jth bounding box in grid cell i that is responsible for predicting no object, and λ coord , obj , noobj Represent the weights of each loss.

[0065] S73, the network parameters are updated based on the back propagation algorithm of gradient descent, as shown in the following formula:

[0066]

[0067]

[0068] Among them, α is the learning rate, L represents the loss function, and w and b represent the sets of weights and bias terms in the network respectively.

[0069] In this example, step S8 is also included, in which the detection performance test of the network trained in step S7 is performed, the images in the test set are input into the network trained in step S7 for forward propagation, and the MAP evaluation index is used to describe the results of target detection and positioning. .5 and MAP .75 Respectively represent the MAP values ​​with thresholds of 0.5 and 0.75; MAP .5:.95 Indicates the average MAP value between the threshold of 0.5-0.95. s , MAP m and MAPl They represent the MAP values ​​of small, medium and large targets respectively.

[0070] The validation set is used to verify the effect during network training, with one round of training and one round of validation.

[0071] Table 1 shows the number of images used for network training, verification and testing in this embodiment. Table 2 shows the detection results obtained in this embodiment and the comparison with some mainstream object detection networks. Figure 5 This is a schematic diagram of the multi-scale target detection results of remote sensing images in this example. Figure 5 (a) represents the ground-truth box of the original image, Figure 5 (b) shows the detection results of Faster-RCNN-FPN. Figure 5 (c) shows the detection result of YoLov3. Figure 5 (d) shows the detection result of this example. It can be seen from the numerical results that the method of the present invention can achieve the target detection task. .5 It can reach 87.2%. From the result picture, it can be seen that this method has better detection results and more accurate positioning than other network models in Table 2.

[0072] Table 1 Number of images used for network training, verification and testing in this embodiment

[0073] category Training set Validation set Test Set Picture (piece) 2275 758 758

[0074] Table 2 Test results obtained in this embodiment

[0075] Methods / Indicators <![CDATA[MAP .5:.95 ]]> <![CDATA[MAP .5 ]]> <![CDATA[MAP .75 ]]> <![CDATA[MAP s ]]> <![CDATA[MAP m ]]> <![CDATA[MAP l ]]> Faster-RCNN 47.7％ 77.5％ 54.1％ 12.7％ 54.1％ 63.8％ Faster-RCNN-FPN 56.0％ 85.9％ 64.8％ 30.1％ 58.6％ 70.0％ YoLov3 52.4％ 72.6％ 61.2％ 25.4％ 55.2％ 67.5％ YoLov5s 47.3％ 68.4％ 55.1％ 18.2％ 48.7％ 66.0％ YoLov5m 49.7％ 69.1％ 57.9％ 19.1％ 51.7％ 67.2％ Ours 57.1％ 87.2％ 66.5％ 32.4％ 60.1％ 71.8％

[0076] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-scale target detection method for remote sensing images based on a CNN+Transformer parallel structure, characterized in that: The specific steps are as follows: S1, construct a convolutional layer structure of a multi-layer CNN; the convolutional layer structure of the multi-layer CNN in step S1 includes six layers of CNN, the first layer of CNN is used for channel adjustment, and the remaining layers of CNN are used to extract local information, and each layer of CNN specifically includes: a first 1×1 convolutional layer, a 3×3 convolutional layer, and a second 1×1 convolutional layer; S2. Construct a multi-layer Transformer structure. The multi-layer Transformer structure specifically includes four layers of Transformers. Each layer of Transformer includes: the first layer of standardization, multi-head self-attention, the second layer of standardization, and a multi-layer perceptron. S3, building an information exchange path for the structures of steps S1 and S2 to obtain a preliminary parallel backbone network; the preliminary parallel backbone network in step S3 is specifically: The input image after the first layer CNN channel adjustment is input into the first layer Transformer and the second layer CNN; The global information extracted by the first layer of Transformer is input into the second layer of CNN after upsampling. The local features obtained by the second layer of CNN are used as the input of the third CNN. The local features obtained by the second layer of CNN are input into the second layer of Transformer after downsampling. The global information extracted by the second layer Transformer is input into the third layer CNN after upsampling. The local features obtained by the third layer CNN are used as the input of the fourth CNN. The local features obtained by the third layer CNN are input into the third layer Transformer after downsampling. The global information extracted by the third-layer Transformer is input into the fourth-layer CNN after upsampling. The local features obtained by the fourth-layer CNN are used as the input of the fifth CNN. The local features obtained by the fourth-layer CNN are input into the fourth-layer Transformer after downsampling. The global information extracted by the fourth-layer Transformer is input into the fifth-layer CNN after upsampling, and the local features taken by the fifth-layer CNN are used as the input of the sixth CNN; S4, constructing a feature pyramid structure for the preliminary parallel backbone network obtained in step S3 to form a neck network; the neck network in step S4 is specifically as follows: the local feature extracted by the sixth CNN in the preliminary parallel backbone network is recorded as P5, the feature obtained by adding P5 to the local feature extracted by the fifth CNN after upsampling is recorded as P4, the feature obtained by adding P4 to the local feature extracted by the fourth CNN after upsampling is recorded as P3, and the feature obtained by adding P3 to the local feature extracted by the third CNN after upsampling is recorded as P2; The feature obtained by downsampling P2 is recorded as C2, the feature obtained by adding downsampling C2 to P3 is recorded as C3; the feature obtained by adding downsampling C3 to P4 is recorded as C4; the feature obtained by adding downsampling C4 to P5 is recorded as C5; S5, connecting the CA attention module to the network of step S4 to construct a deep neural network; step S5, connecting the coordinate attention module to the network of step S4, specifically: connecting C3, C4, and C5 to a coordinate attention module respectively; S6, collecting remote sensing image target detection data, and generating a sample training set, a validation set, and a test set based on the obtained remote sensing image target detection data set; S7, training the deep neural network constructed in step S5 according to the sample training set and verification set in step S6; S8. Input the test set into the deep neural network trained in step S7, so as to perform target detection on the remote sensing image.

2. According to the CNN+Transformer parallel structure remote sensing image multi-scale target detection method of claim 1, it is characterized in that: The deep neural network also includes a 1×1 convolutional layer connected after each coordinate attention module.

Images