Remote sensing image target detection method and device

By using cross-layer non-intensive network modules and semantic guidance modules in remote sensing images, semantic feature maps are extracted, and the problem of reducing discriminative features of small objects in remote sensing images is solved, and the accurate box selection and detection of small objects is achieved.

CN119091318AActive Publication Date: 2024-12-06NEIJIANG NORMAL UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411211714.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

In remote sensing images, the discriminant features of small objects are reduced due to multiple downsamplings, making it difficult to accurately box selection and detect.

Method used

The feature map is obtained through the first feature extraction layer, and a series of cross-layer non-intensive network modules and semantic guidance modules include channel and spatial attention calculation layer, coordinate attention calculation layer and feature map basic calculation layer to generate semantic features Figure, highlighting the boundary information of small targets.

Benefits of technology

Improve the accuracy of small object box selection and detection, ensuring accuracy in small object recognition tasks in high-resolution remote sensing images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091318A_ABST
    Figure CN119091318A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a remote sensing image target detection method and device, and the method is characterized in that the method comprises the steps: obtaining a to-be-recognized image; obtaining a first feature map through the first feature extraction layer; according to the first feature map, a semantic feature map is obtained through a cross-layer non-intensive network module and a semantic guidance module which are connected in series, and the semantic guidance module comprises a channel and space attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer; obtaining a second feature map through a second feature extraction layer according to the semantic feature map; and frame selection and subsequent target identification operation are carried out according to the second feature map. According to the remote sensing image target detection method and device provided by the embodiment of the invention, the accuracy of small target frame selection and detection can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of target recognition. Background Art

[0002] Target recognition is a task in computer vision, and commonly used detection methods include those based on neural networks. Compared with target recognition in natural images, target recognition in remote sensing images is special. Because remote sensing images have high precision and wide range, the proportion of the target in the entire image may be small, and there may be only one or a few pixels in the entire image, so it is called a small target. After multiple downsampling, small targets in high-resolution remote sensing images will lead to a reduction in discriminative features, making it difficult to accurately select and detect.

[0003] Therefore, how to improve the accuracy of small target selection and detection in the task of small target recognition in remote sensing images has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a remote sensing image target detection method and device to accurately extract candidate boxes in the remote sensing image small target recognition task.

[0005] To achieve the above object, an embodiment of the present invention provides a remote sensing image target detection method, comprising:

[0006] Obtain an image to be recognized;

[0007] A first feature map is obtained through a first feature extraction layer;

[0008] According to the first feature map, a semantic feature map is obtained by connecting a cross-layer non-dense network module and a semantic guidance module in series, wherein the semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer;

[0009] According to the semantic feature map, obtaining a second feature map through a second feature extraction layer;

[0010] Perform frame selection and subsequent target recognition operations based on the second feature map.

[0011] Optionally, the feature map basic calculation layer includes:

[0012] Convolutional layer, activation layer, negation layer, element-by-element multiplication layer, channel cascade layer, discriminant layer, convolution batch normalization and activation layer, element-by-element addition layer.

[0013] Optionally, obtaining a semantic feature map by connecting a cross-layer non-dense network module and a semantic guidance module in series according to the first feature map includes:

[0014] The first feature map passes through the cross-layer non-dense network module and then passes through the convolution layer to obtain a first convolution feature map. The first convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the first convolution feature map to obtain a first intermediate feature map. The first convolution feature map passes through the convolution layer again to obtain a second convolution feature map. The second convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the second convolution feature map to obtain a second intermediate feature map.

[0015] The first intermediate feature map and the second intermediate feature map are concatenated along the channel layer to obtain the third intermediate feature map. After passing through the discriminant layer and the activation layer, the third intermediate feature map is multiplied element by element with the second intermediate feature map to obtain the fourth intermediate feature map. After passing through the convolution layer, the second convolution feature map obtains the third convolution feature map. After passing through the activation layer and the negation layer, the third convolution feature map obtains the fifth intermediate feature map.

[0016] The fourth intermediate feature map and the fifth intermediate feature map are multiplied element by element to obtain the sixth intermediate feature map. The sixth intermediate feature map is subjected to convolution batch normalization, activation layer and discrimination layer to obtain the seventh intermediate feature map. The eighth intermediate feature map is obtained after passing through the channel and spatial attention calculation layer.

[0017] The seventh intermediate feature map and the eighth intermediate feature map are subjected to a unit-by-unit multiplication layer to obtain a ninth intermediate feature map, and the ninth intermediate feature map and the third convolutional feature map are subjected to a unit-by-unit addition layer to obtain a semantic feature map.

[0018] Optionally, the first feature extraction layer and the second feature extraction layer include a ResConv layer, an activation layer and a pooling layer respectively, and the ResConv layer includes alternating convolution layers and activation layers.

[0019] Optionally, the ResConv layers of the first feature extraction layer and the second feature extraction layer differ by less than 10 layers.

[0020] Optionally, the first feature extraction layer and the second feature extraction layer respectively include 25 ResConv layers, 25 activation layers and 2 pooling layers, the last layer of the first feature extraction layer is a ResConv layer, and the first layer of the second feature extraction layer is a ResConv layer.

[0021] Optionally, the performing frame selection and subsequent target recognition operations according to the second feature map includes:

[0022] According to the second feature map, target detection is performed through the conv6 layer, conv7 layer, Extra Feature Layers layer, target detection layer and non-maximum suppression layer in the SSD model.

[0023] The embodiment of the present invention further provides a remote sensing image target detection device, comprising:

[0024] An image acquisition unit, used for acquiring an image to be recognized;

[0025] A first feature extraction unit, used for obtaining a first feature map through a first feature extraction layer;

[0026] A semantic feature guiding unit, configured to obtain a semantic feature map according to the first feature map by connecting a cross-layer non-dense network module and a semantic guiding module in series, wherein the semantic guiding module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer;

[0027] A second feature extraction unit, configured to obtain a second feature map through a second feature extraction layer according to the semantic feature map;

[0028] A target recognition unit is used to perform frame selection and subsequent target recognition operations based on the second feature map.

[0029] In this way, the remote sensing image target detection method and device provided by the present invention, after acquiring the image to be identified and obtaining the first feature map through the first feature extraction layer, connects in series a cross-layer non-intensive network module and a semantic guidance module including a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer, to aggregate the contextual semantic information of the three feature maps to capture the long-range dependency of the input feature map while highlighting the boundary information of small targets, thereby ensuring the accuracy of small target selection and detection. It can be seen that the remote sensing image target detection method and device provided by the embodiment of the present invention can improve the accuracy of small target selection and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of a remote sensing image target detection neural network provided by the present invention;

[0032] Figure 2 A schematic diagram of a ResConv architecture provided by the present invention;

[0033] Figure 3 A schematic diagram of a cross-layer non-intensive network module provided by the present invention;

[0034] Figure 4 A schematic diagram of a semantic guidance module provided by the present invention;

[0035] Figure 5A schematic diagram of a coordinate attention calculation layer provided by the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1

[0038] Please refer to Figure 1 , Figure 1 A schematic diagram of a remote sensing image target detection neural network provided by the present invention.

[0039] like Figure 1 As shown, in one embodiment, the remote sensing image target detection method provided by the present invention includes:

[0040] In step S1, an image to be recognized is obtained.

[0041] The image to be identified is a preprocessed image, and the preprocessing includes normalizing the size, adjusting the hue and saturation, etc., marking the targets in the image in the training data set, and other operations.

[0042] In step S2, the ResNet50 model processes the image to be recognized, and its 25th ResConv layer outputs the first feature map.

[0043] As shown in the figure, the ResNet50 model includes 50 ResConv layers, 50 activation layers, and 4 pooling layers. For the architecture of the ResConv layer, please refer to Figure 2 , Figure 2 A schematic diagram of a ResConv architecture provided by the present invention is shown in FIG. Figure 2 As shown in the figure, ResConv includes convolutional layers and activation layers that are alternately arranged. The first to 25th ResConv layers of the ResNet50 model are the first feature extraction layers.

[0044] ResNet50 contains a moderate number of convolutional layers and pooling layers, with acceptable computing power. It uses convolution with a step size of 2 to reduce the size of feature maps, and uses feature maps of different scales as detection layers to predict the categories and position coordinates of targets of different scales. Larger feature maps are used to detect small targets, and smaller feature maps are used to detect large targets, realizing multi-scale target detection. Feature maps of different sizes are extracted to participate in detection together. Large-size feature maps pass through fewer convolutional layers and contain a lot of detail information such as edge textures. Small-size feature maps pass through more convolutional layers, and the extracted information is more semantic and the target features are obvious.

[0045] In some embodiments, the ResConv layers of the first feature extraction layer and the second feature extraction layer may not be 25 layers, as long as the difference is less than 10 layers, that is, they are in the middle layer, so that both position features and semantic features can be extracted.

[0046] In step S3, a cross-layer non-intensive network module and a semantic guidance module obtain a semantic feature map according to the first feature map, and the semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer.

[0047] Please refer to Figure 3 and Figure 4 , Figure 3 A schematic diagram of a cross-layer non-intensive network module provided by the present invention, Figure 4 A schematic diagram of a semantic guidance module provided by the present invention. Figure 3 and Figure 4 As shown, the first feature map first passes through the cross-layer non-intensive network module and then enters the convolution layer to obtain the first convolution feature map. The semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer. The feature map basic calculation layer includes a convolution layer, an activation layer, a negation layer, an element-by-element multiplication layer, a channel cascade layer, a discriminant layer, a convolution batch normalization and activation layer, and an element-by-element addition layer. In the figure, C3, C4 and C5 are convolution layers. The channel and spatial attention calculation layer performs channel attention calculation and spatial attention calculation, and the calculation formula is:

[0048]

[0049] Where F represents the feature map of the input channel and the spatial attention calculation layer, M c () is the channel attention calculation, M s () is the spatial attention calculation formula. Specifically, M c () The channel attention calculation formula is:

[0050]

[0051] Ms () The calculation formula for spatial attention is:

[0052]

[0053] For coordinate attention calculation layer, please refer to Figure 5 , Figure 5 The one-coordinate attention provided by the present invention, as shown in the figure, AvegPool is an average pooling operation, Concat is a connection operation, Conv2d is a two-dimensional convolution operation, BatchNorm+Non_linear is a batch normalization and nonlinear processing operation, Sigmoid is a nonlinear processing operation, Residual is a residual layer, and Re_weight is a weight integration operation.

[0054] Take the input feature map size of C×H×W as an example to illustrate the coordinate attention calculation, where C is the number of channels, H and W are the height and width respectively. Given the input feature map X, we use two pooling kernels with spatial range of (H, 1) or (1, W) to encode each channel along the horizontal coordinate and vertical coordinate respectively. Therefore, the output of the c-th channel with height h is shown in formula (1).

[0055]

[0056] Similarly, the output of the cth channel with width w is shown in formula (2).

[0057]

[0058] Where F1 is a 1x1 convolution operation.

[0059] f=δ(F 1 ([z h ,z w ])) (3)

[0061] Among them, [.,.] represents the connection operation along the spatial dimension, and δ is a nonlinear activation function. We split f into two independent tensors fh and fw along the spatial dimension. Another two 1x 1 convolution transformations F and F are used to transform fh and f" into tensors with the same number of channels as the input X, respectively, thereby obtaining features with the same number of channels, as shown in Equations (4) and (5).

[0062] g h =δ(F h (f h )) (4)

[0064] g w =δ(F w (fw )) (5)

[0066] Then, both the outputs gh and gw are expanded and used as attention weights respectively. Finally, the output Y of our coordinate attention block can be expressed as shown in Equation (6).

[0067]

[0068] Please continue Figure 4 The specific steps of obtaining the semantic feature map by the semantic guidance module are as follows: first, the first feature map passes through the non-intensive network module and then passes through the convolution layer to obtain the first convolution feature map; the first convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the first convolution feature map to obtain the first intermediate feature map; the first convolution feature map passes through the convolution layer again to obtain the second convolution feature map; the second convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the second convolution feature map to obtain the second intermediate feature map; secondly, the first intermediate feature map and the second intermediate feature map are cascaded along the channel to obtain the third intermediate feature map; the third intermediate feature map passes through the discriminant layer and the activation layer and then is multiplied with the second intermediate The feature maps are multiplied element by element to obtain the fourth intermediate feature map. The second convolution feature map passes through the convolution layer to obtain the third convolution feature map. The third convolution feature map passes through the activation layer and the negation layer to obtain the fifth intermediate feature map. Again, the fourth intermediate feature map and the fifth intermediate feature map are multiplied element by element to obtain the sixth intermediate feature map. The sixth intermediate feature map obtains the seventh intermediate feature map after passing through the convolution batch normalization, activation layer and discriminant layer. The eighth intermediate feature map is obtained after passing through the channel and spatial attention calculation layer. Finally, the seventh intermediate feature map and the eighth intermediate feature map pass through the element-by-element multiplication layer to obtain the ninth intermediate feature map. The ninth intermediate feature map and the third convolution feature map pass through the element-by-element addition layer to obtain the semantic feature map.

[0069] The coordinate attention calculation layer embedded in the low-level convolutional layers C3, C4, and C5 can extract the position information of the low-level layers. The convolution, batch normalization, and activation layers set at high levels can extract features with a larger receptive field. The channel and spatial attention calculation layer can strengthen the interaction between channel and spatial information, and is set in combination with the convolution, batch normalization, and activation layers to jointly extract semantic information.

[0070] In this way, the coordinate attention calculation layer can retain the position information and emphasize the representation of interest. The channel and spatial attention calculation layer, as well as the output features that aggregate contextual semantic information, can highlight the boundary information. It can be seen that the semantic guidance module can highlight the discriminative features of small targets, thereby refining the boundaries of small targets and improving the accuracy of detecting small targets in remote sensing images.

[0071] In step S4, the ResNet50 model further performs semantic feature mapping to obtain a second feature map.

[0072] Please continue to refer to Figure 1 As shown in the figure, the semantic guidance module outputs the semantic feature map to the 26th ResConv layer of the ResNet50, and the ResNet50 model processes the semantic feature map through the 26th ResConv layer to the last layer of the model to obtain the second feature map. It can be seen that the 26th ResConv layer to the last layer of ResNet50 is the second feature extraction layer, and the 26th ResConv layer is the first layer of the second feature extraction layer.

[0073] The second feature map is obtained by processing the semantic feature map, and therefore has sufficient discriminative features of small targets, facilitating subsequent target recognition operations.

[0074] In step S5, frame selection and subsequent target recognition operations are performed according to the second feature map.

[0075] Box selection and target detection and recognition are performed according to the fourth feature map through the box selection and target detection layer. In some embodiments, this part can be implemented based on the backbone network of the SSD model, that is, the box selection and target detection layer can be the conv6 layer, conv7 layer, ExtraFeature Layers layer, target detection layer and non-maximum suppression layer in the SSD model.

[0076] In this way, the remote sensing image target detection method provided by the present invention, after acquiring the image to be identified and obtaining the first feature map through the first feature extraction layer, gathers contextual semantic information on the first feature map through a semantic guidance module including a channel and spatial attention calculation layer, a coordinate attention calculation layer and a feature map basic calculation layer, while capturing the long-range dependency of the input feature map, and highlights the boundary information of small targets, thereby ensuring the accuracy of small target selection and detection. It can be seen that the remote sensing image target detection method provided by the embodiment of the present invention can improve the accuracy of small target selection and detection.

[0077] The embodiment of the present invention further provides a remote sensing image target detection device, comprising:

[0078] A first feature extraction unit, used for obtaining a first feature map through a first feature extraction layer;

[0079] A semantic feature guiding unit, configured to obtain a semantic feature map according to the first feature map by connecting a non-dense network module and a semantic guiding module in series, wherein the semantic guiding module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer;

[0080] A second feature extraction unit, configured to obtain a second feature map through a second feature extraction layer according to the semantic feature map;

[0081] A target recognition unit is used to perform frame selection and subsequent target recognition operations based on the second feature map.

[0082] In this way, after acquiring the image to be identified and obtaining the first feature map through the first feature extraction layer, the remote sensing image target detection device provided by the present invention aggregates contextual semantic information on the first feature map through a non-intensive network module and a semantic guidance module including a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer to capture the long-range dependencies of the input feature map while highlighting the boundary information of small targets, thereby ensuring the accuracy of small target selection and detection. It can be seen that the remote sensing image target detection device provided by the embodiment of the present invention can improve the accuracy of small target selection and detection.

[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein. The above describes multiple embodiment schemes provided by the embodiments of the present invention, and the various optional methods introduced in the various embodiment schemes may be combined and cross-referenced with each other without conflict, thereby extending a variety of possible embodiment schemes, which may all be considered as the embodiment schemes disclosed and disclosed by the embodiments of the present invention.

[0084] Although the embodiments of the present invention are disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A remote sensing image target detection method, characterized in that: include: Obtain an image to be recognized; A first feature map is obtained through a first feature extraction layer; According to the first feature map, a semantic feature map is obtained by connecting a cross-layer non-dense network module and a semantic guidance module in series, wherein the semantic guidance module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer; According to the semantic feature map, obtaining a second feature map through a second feature extraction layer; Perform frame selection and subsequent target recognition operations based on the second feature map.

2. The remote sensing image target detection method according to claim 1, characterized in that: The feature map basic calculation layer includes: Convolutional layer, activation layer, negation layer, element-by-element multiplication layer, channel cascade layer, discriminant layer, convolution batch normalization and activation layer, element-by-element addition layer.

3. The remote sensing image target detection method according to claim 3, characterized in that: The method of obtaining a semantic feature map by connecting a cross-layer non-dense network module and a semantic guidance module in series according to the first feature map includes: The first feature map passes through the cross-layer non-dense network module and then passes through the convolution layer to obtain a first convolution feature map. The first convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the first convolution feature map to obtain a first intermediate feature map. The first convolution feature map passes through the convolution layer again to obtain a second convolution feature map. The second convolution feature map passes through the coordinate attention calculation layer and then is multiplied element by element with the second convolution feature map to obtain a second intermediate feature map. The first intermediate feature map and the second intermediate feature map are concatenated along the channel layer to obtain the third intermediate feature map. After passing through the discriminant layer and the activation layer, the third intermediate feature map is multiplied element by element with the second intermediate feature map to obtain the fourth intermediate feature map. After passing through the convolution layer, the second convolution feature map obtains the third convolution feature map. After passing through the activation layer and the negation layer, the third convolution feature map obtains the fifth intermediate feature map. The fourth intermediate feature map and the fifth intermediate feature map are multiplied element by element to obtain the sixth intermediate feature map. The sixth intermediate feature map is subjected to convolution batch normalization, activation layer and discrimination layer to obtain the seventh intermediate feature map. The eighth intermediate feature map is obtained after passing through the channel and spatial attention calculation layer. The seventh intermediate feature map and the eighth intermediate feature map are subjected to a unit-by-unit multiplication layer to obtain a ninth intermediate feature map, and the ninth intermediate feature map and the third convolutional feature map are subjected to a unit-by-unit addition layer to obtain a semantic feature map.

4. The remote sensing image target detection method according to claim 4, characterized in that: The first feature extraction layer and the second feature extraction layer respectively include a ResConv layer, an activation layer and a pooling layer, and the ResConv layer includes convolution layers and activation layers that are alternately arranged.

5. The remote sensing image target detection method according to claim 5, characterized in that: The ResConv layers of the first feature extraction layer and the second feature extraction layer differ by less than 10 layers.

6. The remote sensing image target detection method according to claim 6, characterized in that: The first feature extraction layer and the second feature extraction layer respectively include 25 ResConv layers, 25 activation layers and 2 pooling layers, the last layer of the first feature extraction layer is a ResConv layer, and the first layer of the second feature extraction layer is a ResConv layer.

7. The remote sensing image target detection method according to claim 6, characterized in that: The frame selection and subsequent target recognition operations according to the second feature map include: According to the second feature map, target detection is performed through the conv6 layer, conv7 layer, ExtraFeature Layers layer, target detection layer and non-maximum suppression layer in the SSD model.

8. A remote sensing image target detection device, characterized in that: include: An image acquisition unit, used for acquiring an image to be recognized; A first feature extraction unit, used for obtaining a first feature map through a first feature extraction layer; A semantic feature guiding unit, configured to obtain a semantic feature map according to the first feature map by connecting a cross-layer non-dense network module and a semantic guiding module in series, wherein the semantic guiding module includes a channel and spatial attention calculation layer, a coordinate attention calculation layer, and a feature map basic calculation layer; A second feature extraction unit, configured to obtain a second feature map through a second feature extraction layer according to the semantic feature map; A target recognition unit is used to perform frame selection and subsequent target recognition operations based on the second feature map.

Citation Information

Patent Citations

  • Object detection method based on attention-enhanced bidirectional feature pyramid network (a-bifpn)

    GB202217717D0

  • Cross-view image optimizing method, apparatus, computer equipment, and readable storage medium

    US20220272312A1