A remote sensing image target detection method and device

Abstract

The embodiment of the present application provides a kind of remote sensing image target detection method and device, it is characterized in that, comprising: obtaining image to be identified;Through first feature extraction layer, obtain first feature map;According to the first feature map, through the cross-layer non-dense network module and semantic guide module of series obtain semantic feature map, the semantic guide module includes channel and spatial attention calculation layer, coordinate attention calculation layer and feature map basic calculation layer;According to the semantic feature map, through second feature extraction layer obtain second feature map;According to the second feature map, frame selection and subsequent target identification operation are carried out.It can be seen that the remote sensing image target detection method and device provided by the embodiment of the present application can improve the accuracy of small target frame selection and detection.

Description

Technical Field

[0001] This application relates to the field of target recognition technology. Background Technology

[0002] Target recognition is a task in computer vision, and commonly used detection methods include those based on neural networks. Compared to target recognition in natural images, target recognition in remote sensing images has unique characteristics. Due to the high precision and wide range of remote sensing images, targets may occupy a small proportion of the entire image, possibly only one or a few pixels, hence the term "small targets." In high-resolution remote sensing images, small targets undergo multiple downsampling processes, leading to a reduction in discriminative features and making accurate bounding and detection difficult.

[0003] Therefore, how to improve the accuracy of small target selection and detection in remote sensing image small target recognition tasks has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for remote sensing image target detection, so as to accurately extract candidate boxes in the task of small target recognition in remote sensing images.

[0005] To achieve the above objectives, embodiments of the present invention provide a remote sensing image target detection method, comprising:

[0006] Acquire the image to be recognized;

[0007] The first feature map is obtained through the first feature extraction layer;

[0008] Based on the first feature map, a semantic feature map is obtained through a series of cross-layer non-dense network modules and a semantic guidance module. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer.

[0009] Based on the semantic feature map, a second feature map is obtained through a second feature extraction layer;

[0010] The second feature map is used to perform bounding box selection and subsequent target recognition operations.

[0011] Optionally, the feature map basic computation layer includes:

[0012] Convolutional layers, activation layers, negation layers, progressive multiplication layers, cascaded layers along channels, discriminant layers, convolutional batch normalization and activation layers, and progressive addition layers.

[0013] Optionally, obtaining the semantic feature map based on the first feature map through a series of cross-layer non-dense network modules and a semantic guidance module includes:

[0014] The first feature map is passed through a cross-layer non-dense network module and then through a convolutional layer to obtain a first convolutional feature map. The first convolutional feature map is then passed through a coordinate attention calculation layer and multiplied unequally with the first convolutional feature map to obtain a first intermediate feature map. The first convolutional feature map is then passed through a convolutional layer again to obtain a second convolutional feature map. The second convolutional feature map is then passed through a coordinate attention calculation layer and multiplied unequally with the second convolutional feature map to obtain a second intermediate feature map.

[0015] The first intermediate feature map and the second intermediate feature map are cascaded along the channel to obtain the third intermediate feature map. The third intermediate feature map is multiplied by the second intermediate feature map element by element after passing through the discriminant layer and the activation layer to obtain the fourth intermediate feature map. The second convolutional feature map is passed through the convolutional layer to obtain the third convolutional feature map. The third convolutional feature map is passed through the activation layer and the inversion layer to obtain the fifth intermediate feature map.

[0016] The fourth and fifth intermediate feature maps are multiplied one by one to obtain the sixth intermediate feature map. The sixth intermediate feature map is then processed through a convolutional batch normalization layer, an activation layer, and a discriminant layer to obtain the seventh intermediate feature map. Finally, the sixth intermediate feature map is processed through a channel and spatial attention computation layer to obtain the eighth intermediate feature map.

[0017] The seventh and eighth intermediate feature maps are multiplied by an epoch-wise layer to obtain the ninth intermediate feature map. The ninth intermediate feature map and the third convolutional feature map are added by an epoch-wise layer to obtain the semantic feature map.

[0018] Optionally, the first feature extraction layer and the second feature extraction layer each include a ResConv layer, an activation layer, and a pooling layer, wherein the ResConv layer includes alternating convolutional layers and activation layers.

[0019] Optionally, the difference between the ResConv layers of the first feature extraction layer and the second feature extraction layer is less than 10 layers.

[0020] Optionally, the first feature extraction layer and the second feature extraction layer each include 25 ResConv layers, 25 activation layers and 2 pooling layers, with the last layer of the first feature extraction layer being a ResConv layer and the first layer of the second feature extraction layer being a ResConv layer.

[0021] Optionally, the bounding box selection and subsequent target recognition operation based on the second feature map includes:

[0022] Based on the second feature map, target detection is performed through the conv6 layer, conv7 layer, Extra Feature Layers layer, target detection layer, and non-maximum suppression layer in the SSD model.

[0023] This invention also provides a remote sensing image target detection device, comprising:

[0024] An image acquisition unit is used to acquire the image to be recognized.

[0025] The first feature extraction unit is used to obtain a first feature map through the first feature extraction layer;

[0026] A semantic feature guidance unit is used to obtain a semantic feature map based on the first feature map through a series of cross-layer non-dense network modules and a semantic guidance module. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer.

[0027] The second feature extraction unit is used to obtain a second feature map through the second feature extraction layer based on the semantic feature map;

[0028] The target recognition unit is used to perform bounding box selection and subsequent target recognition operations based on the second feature map.

[0029] Thus, the remote sensing image target detection method and apparatus provided by this invention, after acquiring the image to be identified and obtaining the first feature map through the first feature extraction layer, uses a cross-layer non-dense network module and a semantic guidance module including a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer in series to aggregate contextual semantic information of the three feature maps, capturing the long-range dependencies of the input feature maps while highlighting the boundary information of small targets, thereby ensuring the accuracy of small target bounding box selection and detection. It is evident that the remote sensing image target detection method and apparatus provided by this invention can improve the accuracy of small target bounding box selection and detection. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a remote sensing image target detection neural network provided by the present invention;

[0032] Figure 2 A schematic diagram of a ResConv architecture provided by the present invention;

[0033] Figure 3 This is a schematic diagram of a cross-layer non-dense network module provided by the present invention;

[0034] Figure 4 A schematic diagram of a semantic guidance module provided by the present invention;

[0035] Figure 5A schematic diagram of a coordinate attention calculation layer provided by the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] Please refer to Figure 1 , Figure 1 This is a schematic diagram of a remote sensing image target detection neural network provided by the present invention.

[0039] like Figure 1 As shown, in one embodiment, the remote sensing image target detection method provided by the present invention includes:

[0040] In step S1, the image to be identified is acquired.

[0041] The image to be identified is a preprocessed image. The preprocessing includes operations such as normalizing the size, adjusting the color and saturation, and labeling the targets in the images in the training dataset.

[0042] In step S2, the ResNet50 model processes the image to be recognized, and its 25th ResConv layer outputs the first feature map.

[0043] As shown in the figure, the ResNet50 model consists of 50 ResConv layers, 50 activation layers, and 4 pooling layers. For the architecture of the ResConv layers, please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of a ResConv architecture provided by the present invention, as shown below. Figure 2 As shown, ResConv consists of alternating convolutional layers and activation layers. In the ResNet50 model, layers 1 through 25 of ResConv are the first feature extraction layers.

[0044] ResNet50 contains a moderate number of convolutional and pooling layers, making its computational power acceptable. It uses convolutions with a stride of 2 to reduce the size of the feature maps. It uses feature maps of different scales as detection layers to predict the category and location coordinates of targets at different scales. Larger feature maps are used to detect small targets, and smaller feature maps are used to detect large targets, thus achieving multi-scale target detection. It extracts feature maps of different sizes to participate in detection. Large-sized feature maps go through fewer convolutional layers and contain more detailed information such as edge textures, while small-sized feature maps go through more convolutional layers, extracting more semantic information and making the target features more obvious.

[0045] In some embodiments, the ResConv layers of the first feature extraction layer and the second feature extraction layer do not need to be 25 layers. As long as the difference between them is less than 10 layers, that is, they are in the middle layer, so that both positional features and semantic features can be extracted.

[0046] In step S3, the cross-layer non-dense network module and the semantic guidance module obtain a semantic feature map based on the first feature map. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer.

[0047] Please refer to Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of a cross-layer non-dense network module provided by the present invention. Figure 4 This is a schematic diagram of a semantic guidance module provided by the present invention. Figure 3 and Figure 4 As shown, the first feature map first passes through a cross-layer non-dense network module and then is input into a convolutional layer to obtain the first convolutional feature map. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer. The feature map basic calculation layer includes a convolutional layer, an activation layer, a negation layer, a progressive multiplication layer, a channel-cascaded layer, a discriminant layer, a convolutional batch normalization and activation layer, and a progressive addition layer. In the figure, C3, C4, and C5 are convolutional layers. The channel and spatial attention calculation layer performs channel attention calculation and spatial attention calculation, and the calculation formula is:

[0048]

[0049] Where F represents the feature map of the input channel and the spatial attention computation layer, and M... c () represents channel attention calculation, M s () is the formula for calculating spatial attention. Specifically, M c The formula for calculating channel attention is:

[0050]

[0051] Ms The formula for calculating spatial attention is:

[0052]

[0053] Please refer to the coordinate attention calculation layer. Figure 5 , Figure 5 The coordinate attention mechanism provided by this invention is shown in the figure. AvegPool is the average pooling operation, Concat is the connection operation, Conv2d is the two-dimensional convolution operation, BatchNorm+Non_linear is the batch normalization and non-linear processing operation, Sigmoid is the non-linear processing operation, Residual is the residual layer, and Re_weight is the weight integration operation.

[0054] Taking an input feature map of size C×H×W as an example, we illustrate coordinate attention calculation, where C is the number of channels, and H and W are the height and width, respectively. Given an input feature map X, we encode each channel using two pooling kernels with spatial ranges of (H,1) or (1,W), along the horizontal and vertical coordinates. Therefore, the output of the c-th channel with height h is shown in formula (1).

[0055]

[0056] Similarly, the output of the c-th channel with width w is shown in Equation (2).

[0057]

[0058] F1 is a 1x1 convolution operation.

[0059] f=δ(F1([z h ,z w ])) (3)

[0061] Where [.,.] denotes the connection operation along the spatial dimension, and δ is a nonlinear activation function. We divide f along the spatial dimension into two independent tensors fh and fw. Two other 1x1 convolution transformations F and F" are used to convert fh and f" into tensors with the same number of channels as the input X, respectively, thus obtaining features with the same number of channels, as shown in equations (4) and (5).

[0062] g h =δ(F h (f h )) (4)

[0064] g w =δ(F w (f w )) (5)

[0066] Then, the outputs gh and gw are expanded and used as attention weights respectively. Finally, the output Y of our coordinate attention block can be represented as shown in Equation (6).

[0067]

[0068] Please continue. Figure 4 The semantic guidance module obtains the first feature map through the following steps: First, the first feature map passes through a non-dense network module and then through a convolutional layer to obtain a first convolutional feature map. This first convolutional feature map then passes through a coordinate attention calculation layer and is multiplied unequally with the first convolutional feature map to obtain a first intermediate feature map. This first convolutional feature map then passes through another convolutional layer to obtain a second convolutional feature map. This second convolutional feature map then passes through a coordinate attention calculation layer and is multiplied unequally with the second convolutional feature map to obtain a second intermediate feature map. Second, the first intermediate feature map and the second intermediate feature map are cascaded along the channels to obtain a third intermediate feature map. This third intermediate feature map then passes through a discriminant layer and an activation layer and is multiplied unequally with the second intermediate feature map. The feature maps are multiplied one by one to obtain the fourth intermediate feature map. The second convolutional feature map is passed through a convolutional layer to obtain the third convolutional feature map. The third convolutional feature map is passed through an activation layer and a negation layer to obtain the fifth intermediate feature map. Next, the fourth and fifth intermediate feature maps are multiplied one by one to obtain the sixth intermediate feature map. The sixth intermediate feature map is passed through a convolutional batch normalization layer, an activation layer, and a discriminant layer to obtain the seventh intermediate feature map. After passing through a channel and spatial attention calculation layer, the eighth intermediate feature map is obtained. Finally, the seventh and eighth intermediate feature maps are multiplied one by one to obtain the ninth intermediate feature map. The ninth intermediate feature map and the third convolutional feature map are added one by one to obtain the semantic feature map.

[0069] The coordinate attention computation layer embedded after the low-level convolutional layers C3, C4, and C5 can extract the low-level positional information. The convolutional, batch normalization, and activation layers set in the high-level layers can extract features with a larger receptive field. The channel and spatial attention computation layers can enhance the interaction between channel and spatial information. When combined with the convolutional, batch normalization, and activation layers, they can jointly extract semantic information.

[0070] In this way, the coordinate attention calculation layer can preserve positional information and emphasize the representations of interest. Through the channel and spatial attention calculation layers, and by further aggregating the output features with contextual semantic information, boundary information can be highlighted. Therefore, the semantic guidance module can highlight the discriminative features of small targets, thereby refining the boundaries of small targets and improving the accuracy of detecting small targets in remote sensing images.

[0071] In step S4, the ResNet50 model further extracts semantic feature maps to obtain a second feature map.

[0072] Please continue to refer to this. Figure 1 As shown in the figure, the semantic guidance module outputs the semantic feature map to the 26th ResConv layer of the ResNet50. The ResNet50 model processes the semantic feature map from the 26th ResConv layer to the last layer of the model to obtain the second feature map. It can be seen that the 26th ResConv layer to the last layer of the ResNet50 is the second feature extraction layer, and this 26th ResConv layer is the first layer of the second feature extraction layer.

[0073] The second feature map is obtained by processing the semantic feature map, and therefore has sufficient discriminative features for small targets, which facilitates subsequent target recognition operations.

[0074] In step S5, bounding boxes and subsequent target recognition operations are performed based on the second feature map.

[0075] Box selection and target detection are performed based on the fourth feature map by the box selection and target detection layer. In some embodiments, this part can be implemented based on the backbone network of the SSD model, that is, the box selection and target detection layer can be the conv6 layer, conv7 layer, ExtraFeature Layers layer, target detection layer and nonmaximum suppression layer in the SSD model.

[0076] Thus, the remote sensing image target detection method provided by this invention, after acquiring the image to be identified and obtaining a first feature map through a first feature extraction layer, uses a semantic guidance module, including a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer, to aggregate contextual semantic information in the first feature map to capture the long-range dependencies of the input feature map, while highlighting the boundary information of small targets, thereby ensuring the accuracy of small target selection and detection. It is evident that the remote sensing image target detection method provided by this embodiment can improve the accuracy of small target selection and detection.

[0077] This invention also provides a remote sensing image target detection device, comprising:

[0078] The first feature extraction unit is used to obtain a first feature map through the first feature extraction layer;

[0079] A semantic feature guidance unit is used to obtain a semantic feature map based on the first feature map by a series of non-dense network modules and a semantic guidance module. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer.

[0080] The second feature extraction unit is used to obtain a second feature map through the second feature extraction layer based on the semantic feature map;

[0081] The target recognition unit is used to perform bounding box selection and subsequent target recognition operations based on the second feature map.

[0082] Thus, the remote sensing image target detection device provided by this invention, after acquiring the image to be identified and obtaining a first feature map through a first feature extraction layer, uses a non-dense network module and a semantic guidance module including a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer to aggregate contextual semantic information to capture the long-range dependencies of the input feature map, while highlighting the boundary information of small targets, thereby ensuring the accuracy of small target selection and detection. It is evident that the remote sensing image target detection device provided by this embodiment can improve the accuracy of small target selection and detection.

[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. The foregoing description of multiple embodiments of the present invention provides various alternative methods that can be combined and cross-referenced without conflict, thereby extending to various possible embodiments, all of which can be considered as embodiments disclosed in the present invention.

[0084] While the embodiments of the present invention have been disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for target detection in remote sensing images, characterized in that, include: Acquire the image to be recognized; The first feature map is obtained through the first feature extraction layer; Based on the first feature map, a semantic feature map is obtained through a series of cross-layer non-dense network modules and a semantic guidance module. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer. Based on the semantic feature map, a second feature map is obtained through a second feature extraction layer; Perform bounding selection and subsequent target recognition operations based on the second feature map; The feature map basic computation layer includes: Convolutional layers, activation layers, negation layers, progressive multiplication layers, cascaded layers along channels, discriminant layers, convolutional batch normalization and activation layers, and progressive addition layers; The step of obtaining a semantic feature map based on the first feature map through a series of cross-layer non-dense network modules and a semantic guidance module includes: The first feature map is passed through a cross-layer non-dense network module and then through a convolutional layer to obtain a first convolutional feature map. The first convolutional feature map is then passed through a coordinate attention calculation layer and multiplied unequally with the first convolutional feature map to obtain a first intermediate feature map. The first convolutional feature map is then passed through a convolutional layer again to obtain a second convolutional feature map. The second convolutional feature map is then passed through a coordinate attention calculation layer and multiplied unequally with the second convolutional feature map to obtain a second intermediate feature map. The first intermediate feature map and the second intermediate feature map are cascaded along the channel to obtain the third intermediate feature map. After the third intermediate feature map passes through the discriminant layer and the activation layer, it is multiplied by the second intermediate feature map element by element to obtain the fourth intermediate feature map. The second convolutional feature map passes through the convolutional layer to obtain the third convolutional feature map. The third convolutional feature map passes through the activation layer and the inversion layer to obtain the fifth intermediate feature map. The fourth and fifth intermediate feature maps are multiplied one by one to obtain the sixth intermediate feature map. The sixth intermediate feature map is then processed through a convolutional batch normalization layer, an activation layer, and a discriminant layer to obtain the seventh intermediate feature map. Finally, the sixth intermediate feature map is processed through a channel and spatial attention computation layer to obtain the eighth intermediate feature map. The seventh and eighth intermediate feature maps are multiplied by an epoch-wise layer to obtain the ninth intermediate feature map. The ninth intermediate feature map and the third convolutional feature map are added by an epoch-wise layer to obtain the semantic feature map.

2. The remote sensing image target detection method as described in claim 1, characterized in that, The first feature extraction layer and the second feature extraction layer each include a ResConv layer, an activation layer and a pooling layer, and the ResConv layer includes alternating convolutional layers and activation layers.

3. The remote sensing image target detection method as described in claim 2, characterized in that, The difference between the ResConv layers of the first feature extraction layer and the second feature extraction layer is less than 10 layers.

4. The remote sensing image target detection method as described in claim 3, characterized in that, The first feature extraction layer and the second feature extraction layer each include 25 ResConv layers, 25 activation layers and 2 pooling layers. The last layer of the first feature extraction layer is a ResConv layer, and the first layer of the second feature extraction layer is a ResConv layer.

5. The remote sensing image target detection method as described in claim 4, characterized in that, The step of selecting boxes and subsequent target recognition based on the second feature map includes: Based on the second feature map, target detection is performed through the conv6 layer, conv7 layer, Extra Feature Layers layer, target detection layer, and non-maximum suppression layer in the SSD model.

6. A remote sensing image target detection device, employing the remote sensing image target detection method according to any one of claims 1 to 5, characterized in that, include: An image acquisition unit is used to acquire the image to be recognized. The first feature extraction unit is used to obtain a first feature map through the first feature extraction layer; A semantic feature guidance unit is used to obtain a semantic feature map based on the first feature map through a series of cross-layer non-dense network modules and a semantic guidance module. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer. The second feature extraction unit is used to obtain a second feature map through the second feature extraction layer based on the semantic feature map; The target recognition unit is used to perform bounding box selection and subsequent target recognition operations based on the second feature map.

Images