Remote sensing image target detection method for enhancing bottom layer information

Abstract

The invention provides a remote sensing image target detection method for enhancing bottom layer information, and the method comprises the steps: in a target detection model, a feature extraction module extracts four layers of feature maps which are arranged from bottom to top according to the image size; three edge extraction modules respectively perform edge extraction on the feature map at the bottommost layer, so that the size of the feature map subjected to edge extraction is respectively the same as the sizes of the feature maps at the other three layers; the three global enhancement modules respectively take the corresponding feature maps and the bottommost layer feature maps which have the same size and are subjected to edge extraction as input, channel attention extraction and space attention extraction are carried out, and the attention feature maps are multiplied to obtain three fusion feature maps; and the three detection head modules respectively process the fused feature map and output the category and bounding box information of the target. According to the method, high-level semantic information of a traditional feature pyramid structure is reserved, priori structures such as edges and textures of a bottom layer and position information are further strengthened, position features are effectively refined, and target positioning is more accurate.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a remote sensing image target detection method that enhances underlying information. Background Art

[0002] Accurate and efficient detection and identification of remote sensing images from drones or satellites has become an important scientific and technological means in fields such as agriculture and urban management and planning. In recent years, with the increase in space imaging equipment, the spatial resolution of remote sensing images has continued to improve. Compared with traditional medium and low resolution images, high-resolution remote sensing images can more clearly express the characteristic distribution and spatial association of ground objects. The large amount of texture, color and other details they contain provide new information channels and intelligence sources, which facilitates the refined comprehensive management and decision-making of the region.

[0003] It is worth noting that high-resolution remote sensing images have long imaging distances. Compared with targets in natural scene images, the target sizes of remote sensing images in the same data set vary greatly, which poses certain challenges to the accuracy of target detection. Using feature pyramid networks for multi-scale problems is an effective measure, but traditional feature pyramids have the following problems when recognizing remote sensing images: Although traditional feature pyramid structures better capture high-level semantic information when performing lateral connections after forward extraction of image features, and are more advantageous for class distinction, they ignore the underlying prior structures and position information such as edges and textures, resulting in inaccurate detection of remote sensing images with dense targets. Summary of the invention

[0004] The purpose of the present invention is to solve the problem in the prior art that the use of pyramid networks to detect remote sensing images leads to the inability to accurately detect targets, and to provide a remote sensing image target detection method that strengthens the underlying information.

[0005] To achieve the above purpose, the technical solution provided by the present invention is:

[0006] A remote sensing image target detection method for enhancing underlying information is provided, comprising the following steps:

[0007] Step 1, obtaining a data set of remote sensing images;

[0008] Step 2, constructing a remote sensing image target detection model that strengthens the underlying information. The target detection model includes a feature extraction module, three edge extraction modules, three global enhancement modules and three detection head modules;

[0009] The feature extraction module uses ResNet-34 as the backbone network to extract features from the input image, and obtains four layers of feature maps arranged from bottom to top according to the image size;

[0010] The three edge extraction modules are respectively used to extract edges from the feature map of the bottom layer, so that the image size of the feature map of the bottom layer after edge extraction is the same as the image sizes of the feature maps of the other three layers;

[0011] The three global enhancement modules take the other three layers of feature maps and the edge-extracted bottom-layer feature maps of the same size as input, perform channel attention extraction and spatial attention extraction on the two input feature maps, and multiply the channel attention feature map with the spatial attention feature map to obtain the final three fused feature maps;

[0012] The three detection head modules are used to process the fused feature map and output the target category and bounding box information through the classification branch and regression branch respectively;

[0013] Step 3, iteratively train the constructed target detection model using the acquired data set, and obtain the optimal network model parameters after the detection model converges;

[0014] Step 4: Use the obtained optimal network model parameters to detect the target remote sensing image and obtain the classification and positioning information of the target in the image.

[0015] Furthermore, in step 1, the acquired dataset is a public dataset or a self-photographed dataset.

[0016] Furthermore, step 1 also includes: dividing the acquired data into a training set and a test set, with a division ratio of 8:2 or 7:3.

[0017] Furthermore, in step 2, the image sizes from bottom to top extracted by the feature extraction module are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image size respectively, and the image sizes after the three edge extraction modules perform edge extraction on the feature map of the bottom layer are 1 / 8, 1 / 16 and 1 / 32 of the input image respectively.

[0018] Furthermore, each edge extraction module processes the input feature map as follows: perform global average pooling operations along the height and width directions to obtain two attention feature maps of different sizes; concatenate the two attention feature maps to obtain a concatenated feature map; perform convolution and nonlinear operations on the concatenated feature map, and then perform feature map cutting; perform dot multiplication calculations with the input original feature map along the height and width directions respectively; and adjust the feature map to the required size through the convolution layer.

[0019] Furthermore, the processing process of each global enhancement module on the input feature map is as follows: the two feature maps are added point by point to obtain the feature map M i+1 ; Perform a depth-separable convolution operation; Perform a deformable convolution operation; Transform the feature map M i+1Perform global maximum pooling and global average pooling operations along the height and width directions to obtain two feature maps, and then perform element-by-element addition operations after convolution operations, and generate the final channel attention feature map S after sigmoid activation operations; add the channel attention feature map S and M i+1 Multiply them to get the attention-enhanced feature map; finally, add it to the feature map obtained by performing depth-wise separable convolution and deformable convolution operations to output the final fused feature map.

[0020] Furthermore, in step 2, the detection head module adopts an SSD detection head, and the loss function includes category loss and positioning loss.

[0021] The advantages of the present invention are:

[0022] 1. The remote sensing image target detection method of the present invention strengthens the bottom layer information. After extracting features from the image, the edge extraction module of the bottom layer is adopted to improve the idea of ​​layer-by-layer fusion of the traditional feature pyramid structure. First, the position and saliency information are extracted simultaneously from the lowest-level feature map, and the bottom-level detail information and visual features are embedded into each subsequent high-level network. Not only the high-level semantic information captured by the traditional feature pyramid structure after forward extraction of image features is retained, but also the prior structures and position information such as the bottom edge and texture are further strengthened, the entity position information of the target is retained to the maximum extent, the position features are effectively refined, and the positioning of the target is more accurate.

[0023] 2. The global enhancement module of the present invention adopts four means including deep separable convolution, deformable convolution, channel and spatial attention mechanism to comprehensively integrate local information, global information, channel and other aspects of information, significantly reducing the amount of calculation and the amount of parameters, thereby improving the efficiency of the target detection model, increasing the relationship interaction between space and channels, significantly increasing the receptive field of the model, and strengthening the relationship between non-local positions. The non-local interaction of each part injects the original position information extracted in the edge extraction module into different layers of the feature extraction image, retaining the implicit prior structure information in the original image, and significantly enhancing the ability to extract target edge features, thereby further improving the detection accuracy of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or other features and advantages of the present invention will become more easily understood through the following description with reference to the accompanying drawings, in which:

[0025] Figure 1 is a flow chart of a remote sensing image target detection method for enhancing underlying information of the present invention;

[0026] Figure 2 It is the overall structure diagram of the remote sensing image target detection model in the present invention;

[0027] Figure 3 It is a structural schematic diagram of the edge extraction module in the present invention;

[0028] Figure 4 It is a structural diagram of the global enhancement module;

[0029] FIG5( a ) is a remote sensing image of Example 1 of the present invention;

[0030] FIG5( b ) is a remote sensing image of Example 2 of the present invention;

[0031] FIG6( a ) is a detection result of the present invention in Example 1;

[0032] FIG6( b ) is the detection result of the present invention in Example 2. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the accompanying drawings by means of exemplary embodiments of the present invention. It should be noted that the following detailed description of the present invention is only for the purpose of illustration, and is not intended to limit the present invention.

[0034] The present invention provides a remote sensing image target detection method that strengthens underlying information, which can effectively extract a feature map of a target and improve the accuracy of remote sensing image target detection.

[0035] First, the overall reference Figure 1 As an exemplary embodiment of the present invention, a remote sensing image target detection method for enhancing underlying information includes the following steps:

[0036] Step S1, obtaining a data set of remote sensing images;

[0037] Step S2, constructing a remote sensing image target detection model that strengthens the underlying information, the target detection model includes a feature extraction module, three edge extraction modules, three global enhancement modules and three detection head modules;

[0038] The feature extraction module uses ResNet-34 as the backbone network to extract features from the input image, and obtains four layers of feature maps arranged from bottom to top according to the image size;

[0039] The three edge extraction modules are respectively used to extract edges from the feature map of the bottom layer, so that the image size of the feature map of the bottom layer after edge extraction is the same as the image sizes of the feature maps of the other three layers;

[0040] The three global enhancement modules take the other three layers of feature maps and the edge-extracted bottom-layer feature maps of the same size as input, perform channel attention extraction and spatial attention extraction on the two input feature maps, and multiply the channel attention feature map with the spatial attention feature map to obtain the final three fused feature maps;

[0041] The three detection head modules are used to process the fused feature map and output the target category and bounding box information through the classification branch and regression branch respectively;

[0042] Step S3, iteratively training the constructed target detection model using the acquired data set, and obtaining the optimal network model parameters after the detection model converges;

[0043] Step S4, using the obtained optimal network model parameters to detect the target remote sensing image, and obtain the classification and positioning information of the target in the image.

[0044] In step S1, the remote sensing dataset obtained can be a public dataset, including optical remote sensing datasets such as DIOR and NWPU VHR-10; it can also be a self-photographed dataset. If a self-photographed dataset is used, the image needs to be size normalized first, and then the normalized image needs to be color normalized. In order to prevent overfitting, the number of self-photographed training set images should be no less than 800.

[0045] In addition, the acquired data can be divided into a training set and a test set, and the ratio of the training set to the test set can be 8:2 or 7:3. Accordingly, in step S3, the processed training set data can be input into the model for iterative training, and the optimal network model parameters are obtained after the detection model converges, and then the test set data is detected using the obtained optimal network model parameters.

[0046] For step S2, Figure 2The overall structure of the remote sensing image target detection model is shown. The remote sensing image target detection model constructed to strengthen the underlying information includes a feature extraction module, an edge extraction module, a global enhancement module and a detection head module. In order to effectively solve the gradient vanishing and model saturation problems in the network model, the feature extraction module uses ResNet-34 as the backbone network to extract features from the image. A total of 4 layers of feature maps, C1, C2, C3, and C4, are extracted. The size of each layer of feature map is 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, and the number of channels is 64, 128, 256, and 512 respectively; in the public high-resolution remote sensing image data, a considerable part of the images have densely arranged targets and overlapping occlusions. In this case, it is necessary to accurately identify the edges of the objects, and a large amount of edge details are retained in the underlying network. In order to retain more detailed information and ensure the detection accuracy of dense small targets in remote sensing images, after obtaining feature maps of different sizes, the bottom layer, i.e., C1, is first operated with the edge extraction module three times. The size of the three feature maps F'1, F'2, and F'3 after the operation is different. They are 1 / 8, 1 / 16, and 1 / 32 of the input image respectively, which are the same as C2, C3, and C4; at the same time, in order to increase the interaction between the space and the channel and strengthen the relationship between the non-local positions, the feature map F'1 and C2 are then sent to the global enhancement module 1 to obtain the feature map F2, and the size of the feature map F2 is consistent with that of C2. F'2 and C3 are sent to the global enhancement module 2 to obtain the feature map F3, and the size of the feature map F3 is consistent with that of C3. F'3 and C4 are sent to the global enhancement module 3 to obtain the feature map F4, and the size of the feature map F4 is consistent with that of C4. The global enhancement module fuses the enhanced original underlying information into each layer of features, and further pays attention to the target area. In the subsequent lateral connection fusion operation, the feature map F3 is the same as P3, the feature map P2 is obtained by fusion of P3 and F2, and the feature map P4 is obtained by fusion of P3 and F4. The feature maps P2, P3, and P4 of different scales are sent to the detection head.

[0047] Combination Figure 3There are three edge extraction modules in total. The edge extraction module takes the feature map C1 as input, performs global average pooling operations along the height H and width W, and obtains two attention feature maps of size C×1×W and C×H×1, which can better obtain global information as well as position information and channel dependency. Then, the two feature maps of different sizes are spliced ​​to obtain a C×(H+W)×1 feature map. After convolution and nonlinear operations are performed on the feature map, the feature map is cut and still cut into two feature maps of size C×1×W and C×H×1. After nonlinear operations, dot multiplication is performed with the feature map C1 along the height and width directions respectively. Finally, a convolution layer is passed to adjust to the required size, and three feature maps F'1, F'2, and F'3 are output. The size of the feature map F'1 output by edge extraction module 1 is the same as C2, the size of the feature map F'2 output by edge extraction module 1 is the same as C3, and the size of the feature map F'3 output by edge extraction module 1 is the same as C4. The three edge extraction modules are consistent in the early pooling, splicing, convolution, cutting, and dot multiplication. They are different only in the last convolution operation. The purpose is to output the corresponding feature map C i The same size facilitates subsequent global enhancement module operations.

[0048] The above formula for the global average pooling operation along the height H direction is:

[0049]

[0050] In the formula, j represents the number index in the height H direction, It represents the average value of each element calculated along the height H direction in a certain channel C and a certain width W. Represents the values ​​of each element in a certain channel C and a certain width W along the height H direction.

[0051] The above formula for global average pooling operation along the width W direction is:

[0052]

[0053] Where i represents the number index in the width W direction, It represents the average value of each element calculated along the width W direction at a certain channel C and a certain height H. Represents the values ​​of each element along the width W at a certain channel C and a certain height H.

[0054] The above concatenation, convolution and nonlinear operation formulas after completing the pooling operations in two directions are:

[0055] f = Conv[y w ,y h ]

[0056] In the formula, y w represents the feature map obtained by performing a global average pooling operation along the height H direction, with a size of C×1×W, y h It represents the feature map obtained by performing a global average pooling operation along the width W, with a size of C×H×1. Conv represents convolution calculation, the size of the convolution kernel is 1×1, and the size of f is C×(H+W)×1.

[0057] The above formula for dot multiplication with the feature map C1 along the height and width directions is:

[0058] y c (i, j) = x c (i, j)×(f c h (i))×f c w (j)

[0059] In the formula, f h and f w They respectively indicate that the sizes of the feature maps after cutting and sigmoid operation are C×H×1 and C×1×W, and the subscript c indicates calculation along the channel direction.

[0060] Combination Figure 4 There are three global enhancement modules in total. There are two inputs to the global enhancement module. The first one is the feature map C i+1 , and the second one is F' obtained from the edge extraction module i , the processing process of the input feature map is: first, add the elements of the two feature maps point by point to obtain the feature map M i+1 , the size remains unchanged; in order to increase the interaction between space and channels, strengthen the relationship between non-local positions, complement the information of space and channels, and better focus on the area or target of interest, M is then i+1 Perform operations 1 and DSConv respectively, and keep the same as M after the operation. i+1 The same size; Operation 2, deformable convolution operation (DCN) after keeping the same as M i+1 The size is consistent; then the feature map M i+1 Perform global maximum pooling and global average pooling operations along the H and W (height and width) directions to obtain two 1×1×C feature maps, which are then element-wise added after convolution operations, and finally the final channel attention feature map S is generated after sigmoid activation operation. Then add the channel attention feature map S to M i+1 Multiply them together to get the feature map of attention enhancement; finally add them to the feature maps obtained by operation 1 and operation 2 to output the feature map F i+1. The calculation processes of the three global enhancement modules are the same, only the input feature map sizes are inconsistent. Therefore, the global enhancement module of the present invention uses four means including depthwise separable convolution, deformable convolution, and channel and spatial attention mechanisms to comprehensively integrate local information, global information, channel and other information, significantly reducing the amount of calculation and the amount of parameters, thereby improving the efficiency of the target detection model, increasing the relationship interaction between space and channels, significantly increasing the receptive field of the model, and strengthening the relationship between non-local positions. The non-local interaction of each part injects the original position information extracted in the edge extraction module into different layers of the feature extraction image, retaining the implicit prior structure information in the original image, and significantly enhancing the ability to extract target edge features.

[0061] Then, the three fused feature maps P2, P3, and P4 are sent to the detection head module. The detection head module adopts the SSD detection head, and the loss function includes category loss and positioning loss. The feature map X2 is sent to the detection head and finally compared with the label value to calculate the loss.

[0062] The category loss formula is:

[0063] F L (p t )=-α(1-p t ) γ log(p t )

[0064] In the formula, p t Indicates confidence, p is the prediction probability, y=1 indicates foreground, y≠1 indicates background, α is a weight parameter that can suppress the imbalance in the number of positive and negative samples, and γ is a hyperparameter with a value range of [0, 5].

[0065] The positioning loss formula is:

[0066]

[0067] In the formula, y i is the true value, is the predicted value, and n is the number of sample points.

[0068] Step S3 may specifically include the following steps:

[0069] Store the training data set and the test data set in the train and test folders respectively, and move them into the data folder of the algorithm package. Run the train.py file in the terminal to start training the network. During the training process, first call the feature extraction module to perform feature extraction on the input image, then call the enhancement module to fuse high-level semantic information with underlying detail information, and finally call the detection head network to output the fused feature map X2 to the loss function to calculate the loss value between the true value, and then complete a forward propagation. Subsequently, the neural network will adjust the corresponding parameter values ​​in the model according to the loss value, and continuously update and iterate until the preset number of iterations is completed. At this time, the loss value and time relationship curve tends to be flat and converged. At this time, the entire model is the trained network model with optimal parameters.

[0070] Step S4 may include the following steps:

[0071] The target remote sensing image is used as the detection model trained in step S3 for forward propagation to obtain the location information and category confidence information of the predicted target. Finally, the generated bounding box is screened using the non-maximum suppression method (NMS) to obtain the bounding box and category confidence of each target.

[0072] The present invention exemplarily takes the remote sensing images of FIG. 5( a ) and FIG. 5( b ) as input, and example diagrams of the detection results are shown in FIG. 6( a ) and FIG. 6( b ), which verify the effectiveness of the method proposed in the present invention.

[0073] As described above, after the present invention extracts features from the image, it uses an edge extraction module that strengthens the bottom layer to improve the idea of ​​layer-by-layer fusion of the traditional feature pyramid structure. First, by simultaneously extracting position and saliency information from the lowest-level feature map, and embedding the lowest-level detail information and visual features into each subsequent high-level network, it not only retains the high-level semantic information captured by the traditional feature pyramid structure after forward extraction of image features, but also further strengthens the bottom-level prior structures and position information such as edges and textures, thereby retaining the physical position information of the target to the maximum extent, effectively refining the position features, and locating the target more accurately.

[0074] Finally, it should be noted that the features mentioned and / or shown in the above description of the exemplary embodiments of the present invention may be combined in the same or similar manner into one or more other embodiments, combined with the features in other embodiments or substituted for the corresponding features in other implementations. The technical solutions obtained by these combinations or substitutions shall also be deemed to be included in the protection scope of the present invention.

Claims

1. A remote sensing image target detection method for enhancing underlying information, characterized in that: The following steps are involved: Step 1, obtaining a data set of remote sensing images; Step 2, constructing a remote sensing image target detection model that strengthens the underlying information, wherein the target detection model includes a feature extraction module, three edge extraction modules, three global enhancement modules and three detection head modules; The feature extraction module uses ResNet-34 as the backbone network to extract features from the input image, and obtains four layers of feature maps arranged from bottom to top according to the image size; The three edge extraction modules are respectively used to perform edge extraction on the feature map of the bottom layer, so that the image size of the feature map of the bottom layer after edge extraction is the same as the image sizes of the feature maps of the other three layers; The three global enhancement modules respectively take the other three layers of feature maps and the bottom layer of feature maps with the same size after edge extraction as input, perform channel attention extraction and spatial attention extraction on the two input feature maps, and multiply the channel attention feature map with the spatial attention feature map to obtain the final three fused feature maps; The three detection head modules are used to process the fused feature map respectively, and output the category and bounding box information of the target respectively through the classification branch and the regression branch; Step 3, iteratively train the constructed target detection model using the acquired data set, and obtain the optimal network model parameters after the detection model converges; Step 4: Use the obtained optimal network model parameters to detect the target remote sensing image and obtain the classification and positioning information of the target in the image.

2. The remote sensing image target detection method for enhancing underlying information according to claim 1, characterized in that: In step 1, the acquired dataset is a public dataset or a self-photographed dataset.

3. The remote sensing image target detection method for enhancing underlying information according to claim 2 is characterized in that: Step 1 also includes: dividing the acquired data into a training set and a test set, with a division ratio of 8:2 or 7:

3.

4. The remote sensing image target detection method for enhancing underlying information according to claim 1, characterized in that: In step 2, the image sizes from bottom to top extracted by the feature extraction module are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image size respectively, and the image sizes after the three edge extraction modules perform edge extraction on the feature map of the bottom layer are 1 / 8, 1 / 16 and 1 / 32 of the input image respectively.

5. The remote sensing image target detection method for enhancing underlying information according to claim 1, characterized in that: Each edge extraction module processes the input feature map as follows: perform global average pooling operations along the height and width directions to obtain two attention feature maps of different sizes; perform splicing operations on the two attention feature maps to obtain a spliced ​​feature map; Perform convolution and nonlinear operations on the concatenated feature map, and then cut the feature map; perform dot multiplication with the input original feature map along the height and width directions respectively; adjust the feature map to the required size through the convolution layer.

6. The remote sensing image target detection method for enhancing underlying information according to claim 1, characterized in that: The processing process of each global enhancement module on the input feature map is as follows: the two feature maps are added point by point to obtain the feature map M i+1 ; Perform depth-wise separable convolution operations; Perform deformable convolution operation; transform the feature map M i+1 Perform global maximum pooling and global average pooling operations along the height and width directions to obtain two feature maps, which are then convolved and added element by element, and the final channel attention feature map S is generated after the sigmoid activation operation; The channel attention feature map S and M i+1 Multiply them to get the attention-enhanced feature map; finally, add it to the feature map obtained by performing depth-wise separable convolution and deformable convolution operations to output the final fused feature map.

7. The remote sensing image target detection method for enhancing underlying information according to claim 1, characterized in that: In step 2, the detection head module uses the SSD detection head, and the loss function includes category loss and positioning loss.

Images