Remote sensing image target detection method and device based on improved Faster-Rcnn
Through the improved Faster-Rcnn method, the candidate boxes are extracted in remote sensing images using convolutional calculation and attention mechanism, which solves the problem of boundary and semantic information fusion in small target recognition, and realizes the precise detection of small targets in remote sensing images.
Patent Information
- Application Number
- CN202411211716.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
AI Technical Summary
In remote sensing images, small targets are difficult to accurately extract candidate boxes due to small proportions, incomplete structures and messy backgrounds, which affects the target recognition effect.
The improved Faster-Rcnn method is used to obtain the first, second and third feature maps through the preliminary feature extraction layer, and combine convolutional calculation, coordinate attention calculation and downsampling activation calculation to generate discriminative feature maps to highlight boundary and semantic information. , realize high and low-level information interaction and support precise detection of small targets.
Accurately extract candidate boxes in remote sensing images, providing support for subsequent target recognition, and improving the detection accuracy of small targets.
Smart Images

Figure CN119091319A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of small target recognition in remote sensing images, and in particular to a method and device for remote sensing image target detection based on improved Faster-Rcnn. Background Art
[0002] Object recognition is a task in computer vision. Common detection methods include neural network-based methods, which are mainly divided into single-stage models and two-stage models. Among them, the two-stage model first extracts candidate boxes for the locations where objects may exist on the feature map, and then recognizes the candidate boxes.
[0003] Compared with target recognition in natural images, target recognition in remote sensing images is special. Due to the high precision and wide range of remote sensing images, the proportion of the target in the entire image may be small, and there may be only one or a few pixels in the entire image, so it is called a small target. The same remote sensing image may contain multiple small targets. The small targets have no fixed spatial relationship and are disorganized, which can easily form complex spatial patterns. In addition, due to the relative angle and occlusion between the small targets and the sensor, the structure of the small targets in the remote sensing image may be incomplete and the outline is not clear. The above characteristics and the presence of a large number of similar non-uniform backgrounds in remote sensing images together result in a lack of obvious differences between small targets and other small targets and backgrounds, making it difficult to accurately extract candidate frames, thereby affecting the subsequent recognition of targets in the candidate frames.
[0004] Therefore, how to accurately extract candidate boxes in the task of small target recognition in remote sensing images has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a remote sensing image target detection method and device based on improved Faster-Rcnn, so as to accurately extract candidate boxes in the remote sensing image small target recognition task.
[0006] To achieve the above object, an embodiment of the present invention provides a remote sensing image target detection method based on an improved Faster-Rcnn, comprising:
[0007] Obtain an image to be recognized;
[0008] Through the preliminary feature extraction layer, a first feature map, a second feature map and a third feature map are obtained, wherein the second feature map is higher than the first feature map and lower than the third feature map;
[0009] The discriminative feature extraction layer obtains a discriminative feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation according to the first feature map, the second feature map and the third feature map;
[0010] The identification feature map is used to perform frame selection and subsequent target recognition operations.
[0011] Optionally, the discriminative feature extraction layer includes a first pre-convolutional layer, a second pre-convolutional layer, a third pre-convolutional layer, a first convolutional attention calculation layer, a second convolutional attention calculation layer, a third convolutional attention calculation layer, a fourth convolutional attention calculation layer, a high-level semantic information activation module, and a fifth convolutional attention calculation layer. The discriminative feature map is obtained by convolution calculation, coordinate attention calculation, and down-sampling activation calculation according to the first feature map, the second feature map, and the third feature map, including:
[0012] The second feature map is passed through the first pre-convolutional layer to obtain a first convolutional feature map, the third feature map and the first convolutional feature map are passed through the second pre-convolutional layer to obtain a second convolutional feature map, and the first feature map and the second convolutional feature map are passed through the third pre-convolutional layer to obtain a third convolutional feature map;
[0013] The third convolutional feature map is passed through the third convolutional attention calculation layer to obtain a first intermediate feature map, the first intermediate feature map is passed through the fourth convolutional attention calculation layer to obtain a second intermediate feature map, the second intermediate feature map is passed through a high-level semantic information activation module to obtain an activation feature map, and the third convolutional feature map and the activation feature map are passed through a fifth convolutional attention calculation layer to obtain a high-level semantic feature map;
[0014] The second convolutional feature map and the first intermediate map are passed through the second convolutional attention calculation layer to obtain a third intermediate map, the first convolutional feature map and the third intermediate map are passed through the first convolutional attention calculation layer to obtain a low-level boundary position map, and the bottom-level boundary position map and the high-level semantic layer are fused to obtain a discriminative feature map.
[0015] Optionally, the second intermediate feature map is activated by a high-level semantic information activation module to obtain an activated feature map, including:
[0016] The second intermediate feature map is convolved and then downsampled to obtain a downsampled feature map;
[0017] Perform coordinate attention calculation on the downsampled feature map to obtain a coordinate attention feature map, and perform activation calculation on the coordinate attention feature map to obtain a preliminary activation feature map;
[0018] The preliminary activation feature map and the down-sampled feature map are multiplied element by element and convolved again to obtain an activation feature map.
[0019] Optionally, the performing activation calculation on the coordinate attention feature map to obtain a preliminary activation feature map includes: performing activation calculation by using a Hard_STP(x) activation function:
[0020]
[0021] Optionally, the parameters of the Hard_STP(x) activation function are: a=1.05, b=1.5, c=1.6.
[0022] The embodiment of the present invention also provides a remote sensing image target detection device based on improved Faster-Rcnn, comprising:
[0023] An image acquisition unit, used for acquiring an image to be recognized;
[0024] A preliminary feature extraction unit, configured to obtain a first feature map, a second feature map, and a third feature map through a preliminary feature extraction layer, wherein the second feature map is higher than the first feature map and lower than the third feature map;
[0025] A discriminative feature extraction unit, configured to obtain a discriminative feature map by performing convolution calculation, coordinate attention calculation and downsampling activation calculation on the discriminative feature extraction layer according to the first feature map, the second feature map and the third feature map;
[0026] A frame selection and target recognition unit is used to perform frame selection and subsequent target recognition operations through the identification feature map.
[0027] It can be seen that the remote sensing image target detection method and device based on improved Faster-RCNN provided by the present invention, after obtaining the image to be identified, obtains the first feature map, the second feature map and the third feature map according to the preliminary feature extraction layer in the Faster-RCNN backbone network through the discriminative feature extraction layer, and highlights the boundary and position information in the first feature map and the semantic information of the third feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation, and realizes the information flow and interaction ability between high and low layer channels through the second feature map, and finally generates a feature map containing discriminative features, thereby realizing the accurate detection of small targets in remote sensing images. Therefore, the remote sensing image target detection method and device based on improved Faster-RCNN provided by the embodiment of the present invention can accurately extract candidate boxes in the remote sensing image small target recognition task, and provide support for the recognition of subsequent targets in the candidate boxes. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0029] Figure 1 A flow chart of a remote sensing image target detection method based on improved Faster-Rcnn provided by the present invention;
[0030] Figure 2 A schematic diagram of a remote sensing image target detection neural network based on an improved Faster-Rcnn provided by the present invention;
[0031] Figure 3 Schematic diagram of coordinate attention calculation;
[0032] Figure 4 Schematic diagram of the high-level semantic information activation module. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] Please refer to Figure 1 and Figure 2 , Figure 1 A flow chart of a remote sensing image target detection method based on improved Faster-Rcnn provided by the present invention, Figure 2 A schematic diagram of a remote sensing image target detection neural network based on an improved Faster-RCNN provided by the present invention.
[0035] like Figure 1 As shown, the remote sensing image target detection method based on improved Faster-Rcnn provided by the present invention includes:
[0036] In step S1, an image to be recognized is obtained.
[0037] The image to be identified is a preprocessed image, and the preprocessing includes normalizing the size, adjusting the hue and saturation, etc., marking the targets in the image in the training data set, and other operations.
[0038] In step S2, a first feature map, a second feature map and a third feature map are obtained through a preliminary feature extraction layer, wherein the second feature map is higher than the first feature map and lower than the third feature map.
[0039] The preliminary feature extraction layer includes a series of convolutional layers, pooling layers and activation layers. Specifically, it may include 13 convolutional layers, 13 pooling layers and 4 activation layers.
[0040] It is easy to understand that the feature map obtained at the lower layer retains more boundary and position information, and the feature map obtained at the higher layer contains more high-level semantic information. The second feature map is higher than the first feature map and lower than the third feature map, which means that the pooling layer that outputs the second feature map is higher than the pooling layer that outputs the first feature map and lower than the pooling layer that outputs the third feature map. Therefore, the first feature map contains the most boundary and position information, and the third feature map contains the most high-level semantic information.
[0041] In a specific embodiment, the first feature map, the second feature map and the third feature map are the outputs of the last three pooling layers of the FasterRCNN backbone network, so that the pooling layer can extract the boundary information of small targets and reduce the amount of feature map information, thereby improving the detection speed in the later stage. The last three layers of the Faster-RCNN backbone network contain more high-level semantic information, which is convenient for the recognition and positioning of later targets. Figure 2 As shown, in some embodiments, when the preliminary feature extraction layer includes 13 convolutional layers, 13 pooling layers and 4 activation layers, the first feature map may be the output of the 11th pooling layer, the second feature map may be the output of the 12th pooling layer, and the third feature map may be the output of the 13th pooling layer.
[0042] In step S3, the discriminative feature extraction layer obtains a discriminative feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation according to the first feature map, the second feature map and the third feature map.
[0043] For coordinate attention calculation, please refer to Figure 3 , Figure 3 This is a schematic diagram of coordinate attention calculation. Figure 3 As shown in the figure, Aveg Pool is the average pooling operation, Concat is the connection operation, Conv2d is the two-dimensional convolution operation, BatchNorm+Non_linear is the batch normalization and nonlinear processing operation, Sigmoid is the nonlinear processing operation, Residual is the residual layer, and Re_weight is the weight integration operation.
[0044] Take the input feature map size of C×H×W as an example to illustrate the coordinate attention calculation, where C is the number of channels, H and W are the height and width respectively. Given the input feature map X, we use two pooling kernels with spatial range of (H, 1) or (1, W) to encode each channel along the horizontal coordinate and vertical coordinate respectively. Therefore, the output of the c-th channel with height h is shown in formula (1).
[0045]
[0046] Similarly, the output of the cth channel with width w is shown in formula (2).
[0047]
[0048] Where F1 is a 1x1 convolution operation.
[0049] f=δ(F1([z h ,z w ])) (3)
[0051] Among them, [.,.] represents the connection operation along the spatial dimension, and δ is a nonlinear activation function. We split f into two independent tensors fh and fw along the spatial dimension. Another two 1x 1 convolution transformations F and F are used to transform fh and f" into tensors with the same number of channels as the input X, respectively, thereby obtaining features with the same number of channels, as shown in Equations (4) and (5).
[0052] g h =δ(F h (f h )) (4)
[0054] g w =δ(F w (f w )) (5)
[0056] Then, both the outputs gh and gw are expanded and used as attention weights respectively. Finally, the output Y of our coordinate attention block can be expressed as shown in Equation (6).
[0057]
[0058] In this way, the coordinate attention calculation obtains feature maps that capture height direction information and width direction information respectively. Each map captures the long-range dependency of the input feature map in one spatial direction, and after operations such as recalibration, the final output feature map combines the channel information and spatial position information of the original input feature.
[0059] Through coordinate attention calculation, the low-level feature boundary and position information of the first feature map, the second feature map and the third feature map can be fused with the high-level feature high-level semantic information to realize the interaction of high-level and low-level information, so that the discriminative feature map contains the fused boundary position information and semantic information, thereby supporting the subsequent small target selection operation.
[0060] In one embodiment, please continue to refer to Figure 2 ,like Figure 2As shown, the discriminative feature extraction layer includes a first pre-convolutional layer C3, a second pre-convolutional layer C4, a third pre-convolutional layer C5, a first convolutional attention calculation layer P3, a second convolutional attention calculation layer P4, a third convolutional attention calculation layer P5, a fourth convolutional attention calculation layer P6, a high-level semantic information activation module, and a fifth convolutional attention calculation layer P7. The discriminative feature map is obtained by convolution calculation, coordinate attention calculation and downsampling activation calculation according to the first feature map, the second feature map and the third feature map, including:
[0061] In step S31, the second feature map passes through the first pre-convolutional layer C3 to obtain a first convolutional feature map, the third feature map and the first convolutional feature map pass through the second pre-convolutional layer C4 to obtain a second convolutional feature map, and the first feature map and the second convolutional feature map pass through the third pre-convolutional layer C5 to obtain a third convolutional feature map.
[0062] In step S32, the third convolutional feature map is passed through the third convolutional attention calculation layer P5 to obtain a first intermediate feature map, the first intermediate feature map is passed through the fourth convolutional attention calculation layer P6 to obtain a second intermediate feature map, the second intermediate feature map is passed through a high-level semantic information activation module to obtain an activation feature map, and the third convolutional feature map and the activation feature map are passed through the fifth convolutional attention calculation layer P7 to obtain a high-level semantic feature map.
[0063] In step S33, the second convolutional feature map and the first intermediate map are passed through the second convolutional attention calculation layer P4 to obtain a third intermediate map, the first convolutional feature map and the third intermediate map are passed through the first convolutional attention calculation layer P3 to obtain a low-level boundary position map, and the bottom-level boundary position map and the high-level semantic layer are fused to obtain a discriminative feature map.
[0064] The second intermediate feature map is activated by a high-level semantic information activation module to obtain an activated feature map. Please refer to Figure 4 , Figure 4 This is a schematic diagram of the high-level semantic information activation module. Figure 4 As shown, the second intermediate feature map is first convolved and then downsampled to obtain a downsampled feature map; coordinate attention calculation is performed on the downsampled feature map to obtain a coordinate attention feature map, and activation calculation is performed on the coordinate attention feature map to obtain a preliminary activation feature map; the preliminary activation feature map and the downsampled feature map are multiplied element by element and convolved again to obtain an activation feature map.
[0065] In order to improve the ability to perceive semantic information, in some embodiments, the activation calculation can be calculated using the Hard_STP(x) activation function, and the function formula is shown in (7):
[0066]
[0067] In this way, the Hard_STP(x) activation function can retain the feature information of non-zero distribution and further improve the perception of semantic information in the feature fusion stage. In a specific embodiment, the above Hard_STP(x) activation function has a=1.05, b=1.5, c=1.6. Experiments have shown that this is the best effect, which is demonstrated by the following experiment:
[0068]
[0069] Table 1
[0070]
[0071] Table 2
[0072] Among them, Table 1 is the comparison of indicators on the DIOR dataset, and Table 2 is the comparison of indicators on the VisDrone dataset. mAP50 is the average performance of the model on all categories and all detection difficulties when the prediction correctness standard is that the predicted box and the true box overlap by at least 50%. mAP50-95 is the prediction correctness standard when the minimum overlap between the predicted box and the true box is between 50% and 95%, and the average value is calculated at an interval of 5%. va1 / box_loss is the loss for evaluating the target bounding box, val / obj_loss is the loss for evaluating the target object, and val / cls_loss is the loss for evaluating the classification of the target object.
[0073] In addition, the optimization algorithm for model training is SGD with Momentum, and the value of Momentum is always set to 0.9. For the learning rate, the model uses a fixed learning rate = 0.01. The activation calculation in the high-level semantic information activation module uses the Hard_STP(x) activation function, with a = 1.05, b = 1.5, and c = 1.6.
[0074] In step S4, frame selection and subsequent target recognition operations are performed using the identification feature map.
[0075] It can be seen that the remote sensing image target detection method and device based on improved Faster-RCNN provided by the present invention, after obtaining the image to be identified, obtains the first feature map, the second feature map and the third feature map according to the preliminary feature extraction layer in the Faster-RCNN backbone network through the discriminative feature extraction layer, and highlights the boundary and position information in the first feature map and the semantic information of the third feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation, and realizes the information flow and interaction ability between high and low layer channels through the second feature map, and finally generates a feature map containing discriminative features, thereby realizing the accurate detection of small targets in remote sensing images. Therefore, the remote sensing image target detection method based on improved Faster-RCNN provided by the embodiment of the present invention can accurately extract candidate boxes in the remote sensing image small target recognition task, and provide support for the recognition of subsequent targets in the candidate boxes.
[0076] The embodiment of the present invention also provides a remote sensing image target detection device based on improved Faster-Rcnn, comprising:
[0077] An image acquisition unit, used for acquiring an image to be recognized;
[0078] A preliminary feature extraction unit, configured to obtain a first feature map, a second feature map, and a third feature map through a preliminary feature extraction layer, wherein the second feature map is higher than the first feature map and lower than the third feature map;
[0079] A discriminative feature extraction unit, configured to obtain a discriminative feature map by performing convolution calculation, coordinate attention calculation and downsampling activation calculation on the discriminative feature extraction layer according to the first feature map, the second feature map and the third feature map;
[0080] A frame selection and target recognition unit is used to perform frame selection and subsequent target recognition operations through the identification feature map.
[0081] It can be seen that the remote sensing image target detection device based on improved Faster-RCNN provided by the present invention, after acquiring the image to be identified, obtains the first feature map, the second feature map and the third feature map according to the preliminary feature extraction layer in the Faster-RCNN backbone network through the discriminative feature extraction layer, and highlights the boundary and position information in the first feature map and the semantic information of the third feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation, and realizes the information flow and interaction ability between high and low layer channels through the second feature map, and finally generates a feature map containing discriminative features, thereby realizing the accurate detection of small targets in remote sensing images. Therefore, the remote sensing image target detection device based on improved Faster-RCNN provided by the embodiment of the present invention can accurately extract candidate boxes in the remote sensing image small target recognition task, and provide support for the recognition of subsequent targets in the candidate boxes.
[0082] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein. The above describes multiple embodiment schemes provided by the embodiments of the present invention, and the various optional methods introduced in the various embodiment schemes may be combined and cross-referenced with each other without conflict, thereby extending a variety of possible embodiment schemes, which may all be considered as the embodiment schemes disclosed and disclosed by the embodiments of the present invention.
[0083] Although the embodiments of the present invention are disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.
Claims
1. A remote sensing image target detection method based on improved Faster-Rcnn, characterized in that: include: Obtain an image to be recognized; Through the preliminary feature extraction layer, a first feature map, a second feature map and a third feature map are obtained, wherein the second feature map is higher than the first feature map and lower than the third feature map; The discriminative feature extraction layer obtains a discriminative feature map through convolution calculation, coordinate attention calculation and downsampling activation calculation according to the first feature map, the second feature map and the third feature map; The identification feature map is used to perform frame selection and subsequent target recognition operations.
2. The remote sensing image target detection method based on improved Faster-Rcnn as claimed in claim 1, characterized in that: The discriminative feature extraction layer includes a first pre-convolutional layer, a second pre-convolutional layer, a third pre-convolutional layer, a first convolutional attention calculation layer, a second convolutional attention calculation layer, a third convolutional attention calculation layer, a fourth convolutional attention calculation layer, a high-level semantic information activation module, and a fifth convolutional attention calculation layer. The discriminative feature map is obtained by convolution calculation, coordinate attention calculation, and down-sampling activation calculation according to the first feature map, the second feature map, and the third feature map, including: The second feature map is passed through the first pre-convolutional layer to obtain a first convolutional feature map, the third feature map and the first convolutional feature map are passed through the second pre-convolutional layer to obtain a second convolutional feature map, and the first feature map and the second convolutional feature map are passed through the third pre-convolutional layer to obtain a third convolutional feature map; The third convolutional feature map is passed through the third convolutional attention calculation layer to obtain a first intermediate feature map, the first intermediate feature map is passed through the fourth convolutional attention calculation layer to obtain a second intermediate feature map, the second intermediate feature map is passed through a high-level semantic information activation module to obtain an activation feature map, and the third convolutional feature map and the activation feature map are passed through a fifth convolutional attention calculation layer to obtain a high-level semantic feature map; The second convolutional feature map and the first intermediate map are passed through the second convolutional attention calculation layer to obtain a third intermediate map, the first convolutional feature map and the third intermediate map are passed through the first convolutional attention calculation layer to obtain a low-level boundary position map, and the bottom-level boundary position map and the high-level semantic layer are fused to obtain a discriminative feature map.
3. The remote sensing image target detection method based on improved Faster-Rcnn as claimed in claim 2, characterized in that: The second intermediate feature map is activated by a high-level semantic information activation module to obtain an activated feature map, including: The second intermediate feature map is convolved and then downsampled to obtain a downsampled feature map; Perform coordinate attention calculation on the downsampled feature map to obtain a coordinate attention feature map, and perform activation calculation on the coordinate attention feature map to obtain a preliminary activation feature map; The preliminary activation feature map and the down-sampled feature map are multiplied element by element and convolved again to obtain an activation feature map.
4. The remote sensing image target detection method based on improved Faster-Rcnn as claimed in claim 3, characterized in that: The activation calculation of the coordinate attention feature map to obtain the preliminary activation feature map includes: calculating activation through the Hard_STP(x) activation function:
5. The remote sensing image target detection method based on improved Faster-Rcnn as claimed in claim 4, characterized in that: The parameters of the Hard_STP(x) activation function are: a=1.05, b=1.5, c=1.
6.
6. A remote sensing image target detection device based on improved Faster-Rcnn, characterized in that: include: An image acquisition unit, used for acquiring an image to be recognized; A preliminary feature extraction unit, configured to obtain a first feature map, a second feature map, and a third feature map through a preliminary feature extraction layer, wherein the second feature map is higher than the first feature map and lower than the third feature map; A discriminative feature extraction unit, configured to obtain a discriminative feature map by performing convolution calculation, coordinate attention calculation and downsampling activation calculation on the discriminative feature extraction layer according to the first feature map, the second feature map and the third feature map; A frame selection and target recognition unit is used to perform frame selection and subsequent target recognition operations through the identification feature map.
