Lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion

Abstract

The present invention discloses a lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion, comprising the following steps: acquiring remote sensing data; inputting images from the remote sensing data into a trained remote sensing target detection model; determining the remote sensing target detection result based on the model output; and constructing and training the model, comprising the following steps: acquiring a dataset, clustering the dataset using a Kmeans clustering algorithm to obtain a set of anchor frames; constructing a backbone network to extract feature maps; constructing a fast spatial pyramid pooling structure after the backbone network for feature fusion; constructing a three-layer enhanced feature extraction network (BiFPN) to fuse the spatial information of shallow feature maps with the semantic information of deep feature maps; constructing a channel attention module (ECA) after the BiFPN; constructing three prediction output heads after the ECA for prediction; and training and evaluating the model using the dataset and anchor frames. The network has a small number of network parameters and a low computational load, improving detection accuracy while meeting real-time requirements.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and specifically but not limited to, to a lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion. Background Art

[0002] Object detection has been a hot research topic in computer vision in recent years. Its task is to identify the class of objects of interest in an image, while also detecting their location and size. Object detection is a cornerstone of image understanding and computer vision. As one of the fundamental problems in computer vision, object detection forms the foundation for many other visual tasks, such as instance segmentation, object tracking, and image description. With the continuous advancement of information technology, object detection technology has developed rapidly in recent years. Its widespread application in fields such as biomedicine, surveillance and security, and aerospace has profoundly impacted people's lives.

[0003] The development of deep learning has promoted the application of object detection technology in remote sensing imagery. Deep learning-based object detection in remote sensing images generally outperforms traditional methods, significantly improving detection accuracy. However, remote sensing images present complex scenes, diverse scales, and small objects, which complicate object detection. While deep learning-based object detection algorithms for remote sensing images have achieved excellent performance in recent years, their complex model structures, large number of parameters, and high computational complexity leave much room for improvement in small object detection performance. Therefore, while improving detection accuracy, research on lightweight models for remote sensing imagery holds great theoretical and practical value.

[0004] In view of this, it is necessary to provide a new detection method to solve at least part of the above problems. Summary of the Invention

[0005] In response to one or more problems in the prior art, the present invention proposes a lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion.

[0006] The technical solutions for achieving the purpose of the present invention are:

[0007] First, a lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion is provided, including:

[0008] Acquiring remote sensing data, wherein the remote sensing data includes at least one image;

[0009] Input the images in the remote sensing data into the trained remote sensing target detection model;

[0010] Determine the remote sensing target detection result according to the output of the remote sensing target detection model;

[0011] The construction and training method of the remote sensing target detection model includes:

[0012] Step 1: Get the dataset;

[0013] Step 2: Use the Kmeans clustering algorithm to cluster the data set and obtain a set of anchor boxes;

[0014] Step 3: Build a backbone network and extract features to obtain feature maps;

[0015] Step 4: Construct a fast spatial pyramid pooling structure SPPF after the backbone network to perform feature fusion and obtain a fused feature map;

[0016] Step 5: After the fast spatial pyramid pooling structure SPPF, a three-layer enhanced feature extraction network BiFPN is constructed to fuse the spatial information of the shallow feature map with the semantic information of the deep feature map to reduce information loss;

[0017] Step 6: After strengthening the feature extraction network BiFPN, a three-channel attention module ECA is constructed to make the network pay more attention to the details of the target and improve the detection accuracy;

[0018] Step 7: Build a prediction output head after each channel attention module ECA to predict the output features;

[0019] Step 8: Use the dataset and anchor frames to train and evaluate the remote sensing target detection model, and use the best model for testing to obtain a trained remote sensing target detection model.

[0020] In some embodiments, in step 1, obtaining a data set includes:

[0021] The data sets include at least the remote sensing data set DOTA v1.0, the data set VOC12, the remote sensing data set RSOD, and the remote sensing data set DIOR;

[0022] The training set, validation set, and test set of the VOC12 and RSOD datasets are randomly divided according to the ratio of 7:1:2, and the DIOR dataset is randomly divided according to the official ratio of 1:1:2.

[0023] The remote sensing dataset DOTA v1.0 needs to be preprocessed first, including:

[0024] Step 1-1: Select the horizontally annotated images in the remote sensing dataset DOTA v1.0; cut each selected image into multiple fixed-size images, and generate corresponding annotation information XML files for the cut images;

[0025] Step 1-2: Process the xml files obtained in Step 1-1, delete the xml files that do not meet the requirements and the corresponding images, and obtain the processed DOTA dataset;

[0026] Among them, the xml files that do not meet the requirements are of the following three situations: the labeled target is empty; the difficult value of all labeled targets is 1; there is a problem that the labeled target is out of bounds. There are six situations where the labeled target is out of bounds: xmin < 0, ymin < 0, xmax > width, ymax > height, xmax < xmin, ymax < ymin; where xmin: the minimum abscissa value of the annotation box; ymin: the minimum ordinate value of the annotation box; xmax: the maximum abscissa value of the annotation box; ymax: the maximum ordinate value of the annotation box; width: the width of the image; height: the height of the image;

[0027] Step 1-3: Divide the processed DOTA dataset obtained in Step 1-2 into four files: train, val, trainval, and test. The training set, validation set, and test set are randomly divided according to 7:1:2.

[0028] In some embodiments, in Step 2, the Kmeans clustering algorithm is used to perform clustering on the dataset, including:

[0029] Step 2-1: Randomly select K from all the bounding boxes as the cluster centers according to the dataset;

[0030] Step 2-2: Calculate the distance D between each bounding box and each cluster center. The formula is as follows:

[0031]

[0032] D = 1 - IOU

[0033] In the formula, A and B are the bounding box and the cluster center respectively, and IoU is the intersection over union of A and B;

[0034] Step 2-3: Calculate the cluster center closest to each bounding box and assign it to the cluster closest to it;

[0035] Step 2-4: Recalculate the cluster center according to the bounding boxes in each cluster, and by default, use the median to update the cluster center;

[0036] Step 2-5: Repeat 2-3 to 2-4 until the elements in each cluster no longer change, then the clustering is completed, and K cluster centers are obtained, which are the required anchor boxes;

[0037] Step 2-6: Considering that a larger K value in the prediction network will increase the prediction time of the neural network, the number of cluster centers K is set to 9; the anchor boxes used in each dataset are as follows:

[0038] DOTA v1.0: (11,10)(20,22)(29,41)(43,25)(44,98)(47,45)(78,63)(100,106)(169,202)

[0039] RSOD: (14,15)(22,23)(30,29)(36,39)(47,48)(55,62)(68,74)(80,92)(223,275)

[0040] DIOR: (7,7)(8,16)(14,28)(19,11)(26,47)(36,20)(57,60)(107,128)(278,280)

[0041] VOC12: (21,40)(52,68)(62,141)(97,272)(159,155)(177,380)(303,495)(372,267)(549,556).

[0042] In some embodiments, in step 3, the method for constructing the backbone network includes:

[0043] Step 3-1: The image input size is set to 416*416. First, two convolution modules DarknetConv2D_BN_Leaky are constructed to obtain feature maps. The convolution module includes a 3*3 two-dimensional convolution, normalization and activation function Leaky ReLu, and the step size is set to 2. The first convolution module reduces the image size to 208*208, and the second convolution module further reduces the image size to 104*104. The Leaky ReLu function expression is as follows, where x represents the input value:

[0044]

[0045] Step 3-2: After the two convolution modules, three residual network reductions are constructed in series. The residual network is divided into two branches. The first branch performs a 1*1 convolution on the input feature map, and the feature map size remains unchanged. Then a 3*3 convolution is performed to reduce the size of the feature map by half. Finally, a 1*1 convolution is performed, and the size of the feature map remains unchanged. The second branch performs a 2*2 maximum pooling on the input feature map to reduce the size of the feature map by half. Then a 1*1 convolution is performed, and the size of the feature map remains unchanged. The feature maps output by these two branches are stacked and spliced to finally obtain the feature map with half the size. The feature map sizes output by the first residual network, the second residual network, and the third residual network are: 52*52, 26*26, and 13*13, respectively.

[0046] Step 3-3: After the last residual network, a convolution module DarknetConv2D_BN_Leaky is constructed for feature integration. The feature map size of the final output of the backbone network is 13*13.

[0047] In some embodiments, in step 4, the method for constructing a fast spatial pyramid pooling structure SPPF includes:

[0048] Step 4-1: Input the feature map output by the backbone network into the fast spatial pyramid pooling structure SPPF, perform a convolution on the input feature map to obtain the convolved feature map, and reduce the number of channels by half;

[0049] Step 4-2: Construct three maximum pooling layers with a pooling kernel size of 5*5. The convolved feature map is first passed through the first maximum pooling layer for maximum pooling. The output result is then passed through the second and third maximum pooling layers for maximum pooling. The results of the first, second and third maximum pooling layers are stacked and spliced with the convolved feature map. The stacked and spliced feature map is convolved once, and finally the output size is still 13*13 fused feature map.

[0050] In some embodiments, in step 5, the method for constructing the enhanced feature extraction network BiFPN includes:

[0051] Step 5-1: Perform 1*1 convolution and upsampling on the 13*13 fusion feature map output by the fast spatial pyramid pooling structure SPPF to reduce the feature map size to 26*26; then stack and splice it with the 26*26 feature map output by the second residual network in the backbone network;

[0052] Step 5-2: Perform 1*1 convolution and upsampling on the 26*26 feature map after stacking and splicing in step 5-1, so that the feature map size becomes 52*52; then stack and splice it with the 52*52 feature map output by the first residual network in the backbone network, and output the stacked and spliced feature map to the first channel attention module ECA;

[0053] Step 5-3: Downsample the 52*52 feature map after stacking and splicing in step 5-2 to a size of 26*26, and then stack and splice it with the 26*26 feature map after stacking and splicing in step 5-1 and the 26*26 feature map output by the second residual network in the backbone network, and output the stacked and spliced feature map to the second channel attention module ECA;

[0054] Step 5-4: Downsample the 26*26 feature map after stacking and splicing in step 5-3 to make the size of the feature map 13*13, and then stack and splice it with the 13*13 fusion feature map output by SPPF, and output the stacked and spliced feature map to the third channel attention module ECA.

[0055] In some embodiments, in step 6, the method for constructing the channel attention module ECA includes:

[0056] Step 6-1: Perform global average pooling on the feature map output by the enhanced feature extraction network, and convert the feature map from matrix form to vector form;

[0057] Step 6-2: Avoid dimensionality reduction and use 1D convolution with a kernel size of k to achieve local cross-channel interaction, extract the dependencies between channels, and obtain the weight ω of each channel through the Sigmoid activation function. The formula is as follows:

[0058] ω=σ(C1D k (y)), where C1D represents one-dimensional convolution, σ represents the Sigmoid activation function, and y represents the feature map in vector form output by step 6-1;

[0059] Step 6-3: Multiply the weight ω by the input feature map processed by the enhanced feature extraction network channel by channel to generate a weighted feature map.

[0060] In some embodiments, step 7 includes:

[0061] To perform regression prediction of feature maps, a prediction output head is constructed after each channel attention module ECA; in the prediction output head, a 3*3 convolution module is first constructed for feature integration, and the activation function uses LeakyReLU with a step size of 1; then a 1*1 convolution module is constructed to adjust the number of channels with a step size of 1; each prediction output head obtains the prediction result of one feature map, which corresponds to the three prediction box positions on the grid of different sizes for each image; finally, the prediction result is decoded to obtain the final prediction box position on the image.

[0062] In some embodiments, step 8 includes:

[0063] Step 8-1: Configure the relevant information of the training dataset and model, import the training data and train. The training settings for each dataset are as follows:

[0064] VOC12: 300 rounds, 64 batches, and 640*640 image size.

[0065] DOTA v1.0: 500 rounds, 64 batches, and 640x640 image size.

[0066] RSOD: 1000 rounds, 32 batches, and 640*640 image size.

[0067] DIOR: 300 rounds, 64 batches, and 640*640 image size.

[0068] Step 8-2: Calculate the precision P, recall R, mAP@0.5, and mAP@0.5:0.95 values for each round of training. Plot the graph with the round as the horizontal axis and P / R / map@0.5 / map@0.5:0.95 as the vertical axis. The formula is as follows:

[0069]

[0070] N tp :The positive sample is predicted as a positive sample;

[0071] N fp : Negative samples are predicted as positive samples;

[0072] N fn : Positive samples are predicted as negative samples;

[0073] AP: Plot the PR curve with P as the ordinate and R as the abscissa. The AP value is the area under the PR curve.

[0074] k: number of pictures;

[0075] p(k): Identify k pictures as the value of P;

[0076] Δr(k): The change in recall rate when the number of recognized images increases from k-1 to k;

[0077] mAP: mean average precision;

[0078] c: total number of categories;

[0079] mAP@0.5: mAP when the intersection-over-union ratio is 0.5;

[0080] mAP@0.5:0.95: average mAP at different intersection-over-union (IoU) thresholds from 0.5 to 0.95, with a step size of 0.05.

[0081] Step 8-3: Use the best weights obtained from training to test the test set, set the batch size to 1, and the image size to 640*640. Finally, output the precision P, recall R, mAP@0.5, mAP@0.5:0.95 values, and the inference time T for each image. Calculate the frame rate FPS, FPS = 1000 / T.

[0082] In a second aspect, the present invention provides a lightweight remote sensing target detection device based on channel attention and multi-scale feature fusion, including a processor and a storage medium;

[0083] The storage medium is used to store instructions;

[0084] The processor is configured to operate according to the instructions to execute the steps of the method according to the first aspect.

[0085] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0086] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0087] 1. Use the Kmeans clustering algorithm to cluster the data set and obtain a set of anchor boxes, which speeds up the model convergence and improves the detection accuracy of the model.

[0088] 2. The residual network Reduction is used in the backbone network to reduce the amount of calculation and improve the running speed of the model.

[0089] 3. Using the fast spatial pyramid pooling structure SPPF for feature fusion strengthens the expression of small target information and improves the problem of difficult detection of small targets in remote sensing images.

[0090] 4. The use of enhanced feature extraction network BiFPN increases the spatial detail information of the prediction layer feature map, reduces information loss, improves the model's multi-scale feature fusion capability, and improves the fusion speed and detection accuracy.

[0091] 5. Using the channel attention mechanism ECA, the network pays more attention to the detailed information of the target in the remote sensing image, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings are used to provide a further understanding of the present invention and, together with the description, to explain the embodiments of the present invention, but do not constitute a limitation of the present invention. In the accompanying drawings:

[0093] Figure 1 The overall network structure diagram of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention is shown.

[0094] Figure 2 The figure shows the residual network reduction structure diagram of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention.

[0095] Figure 3 The channel attention ECA structure diagram of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention is shown.

[0096] Figure 4 A schematic diagram shows the detection results of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention on the remote sensing dataset DOTA v1.0.

[0097] Figure 5 A schematic diagram shows the detection results of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention on the remote sensing dataset DIOR.

[0098] Figure 6 A schematic diagram shows the detection results of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention on the remote sensing dataset RSOD.

[0099] Figure 7 A schematic diagram shows the detection results of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion of the present invention on the non-remote sensing dataset VOC12. DETAILED DESCRIPTION

[0100] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0101] The description in this section focuses on a few typical embodiments only, and the present invention is not limited to the scope of the embodiments described. Combinations of different embodiments, replacement of certain technical features in different embodiments, and replacement of certain technical features in the embodiments with the same or similar prior art methods are also within the scope of the present invention.

[0102] In the description of the present invention, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0103] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the exemplary expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0104] Example 1

[0105] A lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion, including:

[0106] Acquiring remote sensing data, wherein the remote sensing data includes at least one image;

[0107] Input the images in the remote sensing data into the trained remote sensing target detection model;

[0108] Determine the remote sensing target detection result according to the output of the remote sensing target detection model;

[0109] The construction and training method of the remote sensing target detection model includes:

[0110] Step 1: Get the dataset;

[0111] Step 2: Use the Kmeans clustering algorithm to cluster the data set and obtain a set of anchor boxes;

[0112] Step 3: Build a backbone network and extract features to obtain feature maps;

[0113] Step 4: Build a fast spatial pyramid pooling structure SPPF after the backbone network for feature fusion to obtain a fused feature map;

[0114] Step 5: Build a three-layer enhanced feature extraction network BiFPN after the fast spatial pyramid pooling structure SPPF to fuse the spatial information of the shallow feature map and the semantic information of the deep feature map, reducing information loss;

[0115] Step 6: Build three channel attention modules ECA after the enhanced feature extraction network BiFPN to make the network pay more attention to the detailed information of the target and improve the detection accuracy;

[0116] Step 7: Build a prediction output head after each channel attention module ECA to predict the output features;

[0117] Step 8: Use the dataset and anchor boxes to train and evaluate the remote sensing target detection model, and use the best model for testing to obtain a trained remote sensing target detection model.

[0118] According to one aspect of the present invention, a lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion, the overall network structure diagram is as Figure 1 shown, including the following steps:

[0119] Step 1: Obtain a dataset and perform cutting and partitioning on the dataset.

[0120] Step 1-1: Obtain the remote sensing dataset DOTA v1.0, select the horizontal annotation part, a total of 2806 images. Since the aspect ratios of some images in the DOTA dataset are too large and cannot be directly used for subsequent training, the DOTA dataset is cut. Each image in the dataset is cut into multiple images of a fixed size of 600*600, a total of 30888 images are obtained, and corresponding annotation information xml files are generated for the cut images.

[0121] Step 1-2: Process the xml files obtained after cutting, and delete the xml files that do not meet the requirements and the corresponding images. There are three cases of xml files that do not meet the requirements: the annotation target is empty; the difficult value of all annotation targets is 1; there is an out-of-bounds problem with the annotation target, and there are six cases of annotation out-of-bounds (xmin<0, ymin<0, xmax>width, ymax>height, xmax<xmin, ymax<ymin). Finally, the dataset has a total of 20889 images.

[0122] xmin: The minimum abscissa value of the annotation box; ymin: The minimum ordinate value of the annotation box

[0123] xmax: the maximum horizontal coordinate value of the label box; ymax: the maximum vertical coordinate value of the label box

[0124] width: width of the image; height: height of the image

[0125] Step 1-3: Divide the processed DOTA dataset into four files: train, val, trainval, and test. The training set, validation set, and test set are randomly divided according to the ratio of 7:1:2.

[0126] Steps 1-4: Obtain the VOC12 dataset, as well as the RSOD and DIOR remote sensing datasets. The training, validation, and test sets of the VOC12 and RSOD datasets are randomly divided into a ratio of 7:1:2, and the DIOR dataset is randomly divided into a ratio of 1:1:2 according to the official ratio.

[0127] Step 2: Use the Kmeans clustering algorithm to cluster the dataset and obtain a set of anchor boxes.

[0128] Step 2-1: According to the dataset, randomly select K cluster centers from all bounding boxes;

[0129] Step 2-2: Calculate the distance D between each bounding box and the center of each cluster using the following formula:

[0130]

[0131] D = 1-IOU

[0132] Where A and B are the bounding box and cluster center respectively, and IoU is the intersection over union of A and B;

[0133] Step 2-3: Calculate the nearest cluster center for each bounding box and assign it to the cluster closest to it;

[0134] Step 2-4: Recalculate the cluster center based on the bounding box in each cluster. By default, the median is used to update the cluster center.

[0135] Step 2-5: Repeat 2-3 to 2-4 until the elements in each cluster no longer change, and the clustering is completed, and K cluster centers are obtained, which are the required anchor boxes;

[0136] Step 2-6: Considering that a large K value in the prediction network will increase the prediction time of the neural network, the cluster center value of the present invention is set to 9. This can accelerate the convergence of the neural network loss function, obtain higher positioning accuracy, and reduce the prediction time of the neural network. The anchor boxes used in the present invention on each data set are as follows:

[0137] DOTAv1.0: (11,10)(20,22)(29,41)(43,25)(44,98)(47,45)(78,63)(100,106)(169,202)

[0138] RSOD: (14,15)(22,23)(30,29)(36,39)(47,48)(55,62)(68,74)(80,92)(223,275)

[0139] DIOR: (7,7)(8,16)(14,28)(19,11)(26,47)(36,20)(57,60)(107,128)(278,280)

[0140] VOC12: (21,40)(52,68)(62,141)(97,272)(159,155)(177,380)(303,495)(372,267)(549,556)

[0141] Step 3: Build a backbone network to extract feature maps.

[0142] Step 3-1: Set the image input size to 416*416. First, construct two convolutional modules DarknetConv2D_BN_Leaky to obtain feature maps. This module includes a 3*3 two-dimensional convolution, normalization, and the activation function Leaky ReLu with a stride of 2. The first module reduces the image size to 208*208, and the second module further reduces the image size to 104*104. The Leaky ReLu function expression is as follows, where x represents the input value:

[0143]

[0144] Step 3-2: Three residual network reductions are constructed serially after the two convolutional modules. This network is divided into two branches. The first branch performs a 1*1 convolution on the input feature map, keeping the feature map size unchanged; then performs a 3*3 convolution to reduce the size of the feature map by half; and finally performs a 1*1 convolution to keep the feature map size unchanged. The second branch performs a 2*2 maximum pooling on the input feature map, reducing the size of the feature map by half; then performs a 1*1 convolution to keep the feature map size unchanged. The feature maps output by these two branches are stacked and concatenated to obtain the feature map with the size reduced by half. The feature map sizes of the three residual networks are: 52*52, 26*26, and 13*13, respectively.

[0145] Step 3-3: Construct the convolution module DarknetConv2D_BN_Leaky after the last residual network for feature integration, and the output feature map size is 13*13.

[0146] Step 4: Construct a fast spatial pyramid pooling structure SPPF after the backbone network for feature fusion.

[0147] Step 4-1: Input the 13*13 feature map output by the last convolution module of the backbone network into the fast spatial pyramid pooling structure SPPF, and perform a convolution on it to reduce the number of channels by half.

[0148] Step 4-2: Construct a maximum pooling layer with a pooling kernel size of 5*5, perform maximum pooling on the convolved feature map, and then perform maximum pooling on the output results in sequence. Repeat twice, stack and splice the result of each time with the feature map that has only undergone convolution, and perform convolution on the stacked and spliced feature map. Finally, the output size is still 13*13 feature map.

[0149] Step 5: Construct a three-layer enhanced feature extraction network BiFPN after SPPF to fuse the spatial information of the shallow feature map with the semantic information of the deep feature map to reduce information loss.

[0150] Step 5-1: Perform a 1x1 convolution and upsample the 13x13 feature map output by the fast spatial pyramid pooling structure (SPPF) to a size of 26x26. This is then concatenated with the 26x26 feature map output by the second residual network in the backbone network.

[0151] Step 5-2: Perform 1x1 convolution and upsampling on the 26x26 feature map from step 5-1, reducing its size to 52x52. This is then stacked and concatenated with the 52x52 feature map output by the first residual network in the backbone network, and the resulting stacked feature map is sent to the channel attention module (ECA).

[0152] Step 5-3: Downsample the 52*52 feature map after stacking and splicing in step 5-2 to a size of 26*26, and then stack and splice it with the 26*26 feature map after stacking and splicing in step 5-1 and the 26*26 feature map output by the second residual network in the backbone network, and output the stacked and spliced feature map to the channel attention module ECA.

[0153] Step 5-4: Downsample the 26*26 feature map after stacking and splicing in step 5-3 to make the size of the feature map 13*13, and then stack and splice it with the 13*13 feature map output by SPPF, and output the stacked and spliced feature map to the channel attention module ECA.

[0154] Step 6: Construct three channel attention modules ECA after the BiFPN network to make the network pay more attention to the details of the target and improve the detection accuracy.

[0155] Step 6-1: Perform global average pooling on the feature map output by the enhanced feature extraction network, and convert the feature map from matrix form to vector form;

[0156] Step 6-2: Avoid dimensionality reduction and use 1D convolution with a kernel size of k to achieve local cross-channel interaction, extract the dependencies between channels, and obtain the weight ω of each channel through the Sigmoid activation function. The formula is as follows:

[0157] ω=σ(C1D k (y)), where C1D represents one-dimensional convolution, σ represents the Sigmoid activation function, and y represents the feature map in vector form output by step 6-1;

[0158] Step 6-3: Multiply the weight ω by the input feature map processed by the enhanced feature extraction network channel by channel to generate a weighted feature map.

[0159] Step 7: Build a prediction output head after each ECA module to predict the output features.

[0160] To perform regression prediction on feature maps, a prediction output head (yolo head) is constructed after each ECA. In the prediction output head, a 3x3 convolutional module is first constructed for feature integration, using the LeakyReLU activation function with a stride of 1. A 1x1 convolutional module is then constructed to adjust the number of channels, with a stride of 1. Each prediction output head obtains a prediction result for one feature map, corresponding to three prediction box positions on a grid of different sizes for each image. Finally, the prediction result is decoded to obtain the final prediction box position on the image.

[0161] Step 8: Train and evaluate the models, and use the best model for testing.

[0162] Step 8-1: Configure the relevant information of the training dataset and model, import the training data and train it. The training settings for each dataset are as follows:

[0163] VOC12: 300 rounds, 64 batches, and 640*640 image size

[0164] DOTA v1.0: 500 rounds, 64 batches, and 640*640 image size.

[0165] RSOD: round number is 1000, batch size is 32, image size is 640*640

[0166] DIOR: 300 rounds, 64 batches, and 640*640 image size

[0167] Step 8-2: Calculate the precision (P), recall (R), mAP@0.5, and mAP@0.5:0.95 values for each round of training. Finally, plot them on a graph with the round as the horizontal axis and P / R / map@0.5 / map@0.5:0.95 as the vertical axis. The formula is as follows:

[0168]

[0169] N tp :The positive sample is predicted as a positive sample;

[0170] N fp : Negative samples are predicted as positive samples;

[0171] N fn : Positive samples are predicted as negative samples;

[0172] AP: Plot the PR curve with P as the ordinate and R as the abscissa. The AP value is the area under the PR curve.

[0173] k: number of pictures;

[0174] p(k): Identify k pictures as the value of P

[0175] Δr(k): The change in recall rate when the number of recognized images increases from k-1 to k;

[0176] mAP: mean average precision;

[0177] c: total number of categories;

[0178] mAP@0.5: mAP when the intersection-over-union ratio is 0.5;

[0179] mAP@0.5:0.95: average mAP at different intersection-over-union thresholds (from 0.5 to 0.95, step size 0.05);

[0180] Step 8-3: Use the best weights obtained from training to test the test set, set the batch size to 1, the image size to 640*640, and finally output the precision (P), recall rate (R), mAP@0.5, mAP@0.5:0.95 values and the inference time T (in milliseconds) for each image. Calculate the frame rate FPS, FPS = 1000 / T.

[0181] This paper has conducted experiments on the datasets DOTA v1.0, DIOR, RSOD, and VOC12 and obtained the test results. Representative test results are selected for display, corresponding to Figure 4 、 Figure 5 、 Figure 6 and Figure 7 To further verify the advantages of the target detection method of the present invention, experiments were conducted on other models under the same conditions and compared with the detection results of the lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion in this embodiment. The evaluation indicators used were mAP@0.5, mAP@0.5:0.95, the time T for detecting an image, and the frame rate FPS, where FPS and T are the average values obtained from 50 tests.

[0182] The experimental results on the RSOD dataset are shown in Table 1:

[0183] Compared with yolov4, the map@0.5 of the proposed method is only reduced by 0.1 percentage points, and the map@0.5:0.95 is only reduced by 7.2 percentage points, but the FPS is increased by 167, which is much faster than yolov4.

[0184] Compared with yolov3, the map@0.5 of the proposed method is improved by 2.7 percentage points, and the map@0.5:0.95 is reduced by 5.5 percentage points, but the FPS is increased by 147, which is much faster than yolov3.

[0185] Compared with yolov4-tiny, the FPS of the proposed method is reduced by 51, but map@0.5 is improved by 7.7 percentage points, and map@0.5:0.95 is improved by 4.9 percentage points, and the detection accuracy is higher.

[0186] Compared with yolov3-tiny, the FPS of the proposed method is reduced by 49, but map@0.5 is improved by 9.5 percentage points, and map@0.5:0.95 is improved by 7.3 percentage points, and the detection accuracy is higher.

[0187] Table 1 Experimental results on the RSOD dataset

[0188]

[0189] The experimental results on the DIOR dataset are shown in Table 2:

[0190] Compared with yolov4, the map@0.5 of the proposed method is reduced by 11.2 percentage points, and the map@0.5:0.95 is reduced by 14.9 percentage points, but the FPS is increased by 160, which is much faster than yolov4.

[0191] Compared with yolov3, the map@0.5 of the proposed method is reduced by 8.2 percentage points, and the map@0.5:0.95 is reduced by 13.4 percentage points, but the FPS is increased by 133, which is much faster than yolov3.

[0192] Compared with yolov4-tiny, the FPS of the proposed method is reduced by 51, but map@0.5 is improved by 10.4 percentage points, and map@0.5:0.95 is improved by 6.4 percentage points, and the detection accuracy is higher.

[0193] Compared with yolov3-tiny, the FPS of the proposed method is reduced by 53, but map@0.5 is improved by 15.2 percentage points, and map@0.5:0.95 is improved by 10.5 percentage points, and the detection accuracy is higher.

[0194] Table 2 Experimental results on the DIOR dataset

[0195]

[0196] The experimental results on the VOC12 dataset are shown in Table 3:

[0197] Compared with yolov4, the map@0.5 of the proposed method is reduced by 12.8 percentage points, and the map@0.5:0.95 is reduced by 16.8 percentage points, but the FPS is increased by 167, which is much faster than yolov4.

[0198] Compared with yolov3, the map@0.5 of the proposed method is reduced by 10.5 percentage points, and the map@0.5:0.95 is reduced by 14.7 percentage points, but the FPS is increased by 160, which is much faster than yolov3.

[0199] Compared with yolov4-tiny, the FPS of the proposed method is reduced by 47, but map@0.5 is improved by 11.8 percentage points, and map@0.5:0.95 is improved by 12.9 percentage points, and the detection accuracy is higher.

[0200] Compared with yolov3-tiny, the FPS of the proposed method is reduced by 45, but map@0.5 is improved by 14.5 percentage points, and map@0.5:0.95 is improved by 15.6 percentage points, and the detection accuracy is higher.

[0201] Table 3 Experimental results on the VOC12 dataset

[0202]

[0203] The experimental results on the DOTA dataset are shown in Table 4:

[0204] Compared with yolov4, the map@0.5 of the proposed method is reduced by 3.8 percentage points, and the map@0.5:0.95 is reduced by 15.9 percentage points, but the FPS is increased by 179, which is much faster than yolov4.

[0205] Compared with yolov3, the map@0.5 of the proposed method is reduced by 1.9 percentage points, and the map@0.5:0.95 is reduced by 14.2 percentage points, but the FPS is increased by 154, which is much faster than yolov3.

[0206] Compared with yolov4-tiny, the FPS of the proposed method is reduced by 23, but map@0.5 is improved by 6 percentage points, and map@0.5:0.95 is improved by 3.9 percentage points, and the detection accuracy is higher.

[0207] Compared with yolov3-tiny, the FPS of the proposed method is reduced by 40, but map@0.5 is improved by 7 percentage points, and map@0.5:0.95 is improved by 9.4 percentage points, and the detection accuracy is higher.

[0208] Table 4 Experimental results on the DOTA v1.0 dataset

[0209]

[0210] To verify the importance of the residual network (M1), the fast spatial pyramid pooling structure (SPPF) (M2), the enhanced feature extraction network (BiFPN) (M3), and the channel attention (ECA) (M4), we conducted ablation experiments on the DIOR dataset: "Baseline," "Baseline + M1," "Baseline + M1 + M2," "Baseline + M1 + M2 + M3," and "Baseline + M1 + M2 + M3 + M4." "Baseline" refers to the YOLOv4-tiny algorithm. The results are shown in Table 5. M1 improves the model's inference speed, while M2, M3, and M4 improve the detection accuracy of YOLOv4-tiny.

[0211] Table 5 Ablation experiments on the DIOR dataset

[0212]

[0213] Example 2

[0214] In a second aspect, this embodiment provides a lightweight remote sensing target detection device based on channel attention and multi-scale feature fusion, including a processor and a storage medium;

[0215] The storage medium is used to store instructions;

[0216] The processor is configured to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0217] Example 3

[0218] In a third aspect, this embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.

[0219] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0220] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0221] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0222] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0223] The description and application of the present invention here are illustrative and are not intended to limit the scope of the present invention to the above-mentioned embodiments. The relevant descriptions of the effects or advantages involved in the specification may not be reflected in the actual experimental examples due to the uncertainty of specific condition parameters or other factors, and the relevant descriptions of the effects or advantages are not used to limit the scope of the invention. Variations and changes to the embodiments disclosed here are possible, and the replacement of the embodiments and various equivalent components are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that, without departing from the spirit or essential characteristics of the present invention, the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts. Without departing from the scope and spirit of the present invention, other variations and changes can be made to the embodiments disclosed here.

Claims

1. A lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion, characterized by: It includes: Obtain remote sensing data, where the remote sensing data includes at least one picture; Input the pictures in the remote sensing data into a trained remote sensing object detection model; Determine the remote sensing object detection result according to the output of the remote sensing object detection model; The construction and training method of the remote sensing object detection model includes: Step 1: Obtain a dataset; Step 2: Use the Kmeans clustering algorithm to perform clustering on the dataset to obtain a set of anchor boxes; Step 3: Construct a backbone network to obtain a feature map through feature extraction; Step 4: Construct a fast spatial pyramid pooling structure SPPF after the backbone network for feature fusion to obtain a fused feature map; Step 5: Construct a three-layer enhanced feature extraction network BiFPN after the fast spatial pyramid pooling structure SPPF to fuse the spatial information of the shallow feature map and the semantic information of the deep feature map, reducing information loss; Step 6: Construct three channel attention modules ECA after the enhanced feature extraction network BiFPN to make the network pay more attention to the detailed information of the target and improve the detection accuracy; Step 7: Construct a prediction output head after each channel attention module ECA to predict the output features; Step 8: Use the dataset and anchor boxes to train and evaluate the remote sensing object detection model, and use the best model for testing to obtain a trained remote sensing object detection model.

2. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that: In the said Step 1, obtaining the dataset includes: The dataset at least includes the remote sensing dataset DOTA v1.0, the dataset VOC12, the remote sensing dataset RSOD, and the remote sensing dataset DIOR; Among them, the training set, validation set, and test set of the dataset VOC12 and RSOD are randomly divided according to 7:1:2, and DIOR is randomly divided according to the ratio of 1:1:2 given by the official; Among them, the remote sensing dataset DOTA v1.0 needs to be preprocessed first, including: Step 1-1: Select the pictures in the horizontal annotation part of the remote sensing dataset DOTA v1.0; Cut each selected picture into multiple pictures of a fixed size, and generate a corresponding annotation information xml file for the cut pictures; Step 1-2: Process the xml files obtained in Step 1-1, delete the xml files that do not meet the requirements and the corresponding pictures, and obtain the processed DOTA dataset; The xml files that do not meet the requirements are of the following three situations: the annotation target is empty; the difficult value of all annotation targets is 1; there is a problem that the annotation target exceeds the boundary. There are six situations where the annotation target exceeds the boundary: xmin < 0, ymin < 0, xmax > width, ymax > height, xmax < xmin, ymax < ymin; where xmin: the minimum abscissa value of the annotation box; ymin: the minimum ordinate value of the annotation box; xmax: the maximum abscissa value of the annotation box; ymax: the maximum ordinate value of the annotation box; width: the width of the picture; height: the height of the picture; Step 1-3: Divide the processed DOTA dataset obtained in step 1-2 into four files: train, val, trainval, and test. The training set, validation set, and test set are randomly divided according to the ratio of 7:1:

2.

3. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that: In step 2, the Kmeans clustering algorithm is used to cluster the data set, including: Step 2-1: According to the dataset, randomly select K cluster centers from all bounding boxes; Step 2-2: Calculate the distance D between each bounding box and the center of each cluster using the following formula: D = 1-IOU Where A and B are the bounding box and cluster center respectively, and IoU is the intersection over union of A and B; Step 2-3: Calculate the nearest cluster center for each bounding box and assign it to the cluster closest to it; Step 2-4: Recalculate the cluster center based on the bounding box in each cluster. By default, the median is used to update the cluster center. Step 2-5: Repeat 2-3 to 2-4 until the elements in each cluster no longer change, and the clustering is completed, and K cluster centers are obtained, which are the required anchor boxes; Step 2-6: Considering that a larger K value in the prediction network will increase the prediction time of the neural network, the number of cluster centers K is set to 9; the anchor boxes used in each dataset are as follows: DOTA v1.0: (11,10)(20,22)(29,41)(43,25)(44,98)(47,45)(78,63)(100,106)(169,202) RSOD: (14,15)(22,23)(30,29)(36,39)(47,48)(55,62)(68,74)(80,92)(223,275) DIOR: (7,7)(8,16)(14,28)(19,11)(26,47)(36,20)(57,60)(107,128)(278,280) VOC12: (21,40)(52,68)(62,141)(97,272)(159,155)(177,380)(303,495)(372,267)(549,556).

4. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that In step 3, the method for constructing the backbone network includes: Step 3-1: The image input size is set to 416*416. First, two convolution modules DarknetConv2D_BN_Leaky are constructed to obtain feature maps. The convolution module includes a 3*3 two-dimensional convolution, normalization and activation function Leaky ReLu, and the step size is set to 2. The first convolution module reduces the image size to 208*208, and the second convolution module further reduces the image size to 104*104. The Leaky ReLu function expression is as follows, where x represents the input value: Step 3-2: After the two convolution modules, three residual network reductions are constructed in series. The residual network is divided into two branches. The first branch performs a 1*1 convolution on the input feature map, and the feature map size remains unchanged. Then a 3*3 convolution is performed to reduce the size of the feature map by half. Finally, a 1*1 convolution is performed, and the size of the feature map remains unchanged. The second branch performs a 2*2 maximum pooling on the input feature map to reduce the size of the feature map by half. Then a 1*1 convolution is performed, and the size of the feature map remains unchanged. The feature maps output by these two branches are stacked and spliced to finally obtain the feature map with half the size. The feature map sizes output by the first residual network, the second residual network, and the third residual network are: 52*52, 26*26, and 13*13, respectively. Step 3-3: After the last residual network, a convolution module DarknetConv2D_BN_Leaky is constructed for feature integration. The feature map size of the final output of the backbone network is 13*13.

5. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that In step 4, the method for constructing the fast spatial pyramid pooling structure SPPF includes: Step 4-1: Input the feature map output by the backbone network into the fast spatial pyramid pooling structure SPPF, perform a convolution on the input feature map to obtain the convolved feature map, and reduce the number of channels by half; Step 4-2: Construct three maximum pooling layers with a pooling kernel size of 5*5. The convolved feature map is first passed through the first maximum pooling layer for maximum pooling. The output result is then passed through the second and third maximum pooling layers for maximum pooling. The results of the first, second and third maximum pooling layers are stacked and spliced with the convolved feature map. The stacked and spliced feature map is convolved once, and finally the output size is still 13*13 fused feature map.

6. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that In step 5, the method for constructing the enhanced feature extraction network BiFPN includes: Step 5-1: Perform 1*1 convolution and upsampling on the 13*13 fusion feature map output by the fast spatial pyramid pooling structure SPPF to reduce the feature map size to 26*26; then stack and splice it with the 26*26 feature map output by the second residual network in the backbone network; Step 5-2: Perform 1*1 convolution and upsampling on the 26*26 feature map after stacking and splicing in step 5-1, so that the feature map size becomes 52*52; then stack and splice it with the 52*52 feature map output by the first residual network in the backbone network, and output the stacked and spliced feature map to the first channel attention module ECA; Step 5-3: Downsample the 52*52 feature map after stacking and splicing in step 5-2 to a size of 26*26, and then stack and splice it with the 26*26 feature map after stacking and splicing in step 5-1 and the 26*26 feature map output by the second residual network in the backbone network, and output the stacked and spliced feature map to the second channel attention module ECA; Step 5-4: Downsample the 26*26 feature map after stacking and splicing in step 5-3 to make the size of the feature map 13*13, and then stack and splice it with the 13*13 fusion feature map output by SPPF, and output the stacked and spliced feature map to the third channel attention module ECA.

7. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that In step 6, the method for constructing the channel attention module ECA includes: Step 6-1: Perform global average pooling on the feature map output by the enhanced feature extraction network, and convert the feature map from matrix form to vector form; Step 6-2: Avoid dimensionality reduction and use 1D convolution with a kernel size of k to achieve local cross-channel interaction, extract the dependencies between channels, and obtain the weight ω of each channel through the Sigmoid activation function. The formula is as follows: ω=σ(C1D k (y)), where C1D represents one-dimensional convolution, σ represents the Sigmoid activation function, and y represents the feature map in vector form output by step 6-1; Step 6-3: Multiply the weight ω by the input feature map processed by the enhanced feature extraction network channel by channel to generate a weighted feature map.

8. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that: The step 7 comprises: To perform regression prediction of feature maps, a prediction output head is constructed after each channel attention module ECA; in the prediction output head, a 3*3 convolution module is first constructed for feature integration, and the activation function uses LeakyReLU with a step size of 1; then a 1*1 convolution module is constructed to adjust the number of channels with a step size of 1; each prediction output head obtains the prediction result of one feature map, which corresponds to the three prediction box positions on the grid of different sizes for each image; finally, the prediction result is decoded to obtain the final prediction box position on the image.

9. The lightweight remote sensing target detection method based on channel attention and multi-scale feature fusion according to claim 1 is characterized in that: The step 8 comprises: Step 8-1: Configure the relevant information of the training dataset and model, import the training data and train it. The training settings for each dataset are as follows: VOC12: 300 rounds, 64 batches, and 640*640 image size. DOTA v1.0: 500 rounds, 64 batches, and 640x640 image size. RSOD: 1000 rounds, 32 batches, and 640*640 image size. DIOR: 300 rounds, 64 batches, and 640*640 image size. Step 8-2: Calculate the precision P, recall R, mAP@0.5, and mAP@0.5:0.95 values for each round of training. Plot the graph with the round as the horizontal axis and P / R / map@0.5 / map@0.5:0.95 as the vertical axis. The formula is as follows: N tp :The positive sample is predicted as a positive sample; N fp : Negative samples are predicted as positive samples; N fn : Positive samples are predicted as negative samples; AP: Plot the PR curve with P as the ordinate and R as the abscissa. The AP value is the area under the PR curve. k: number of pictures; p(k): Identify k pictures as the value of P; Δr(k): The change in recall rate when the number of recognized images increases from k-1 to k; mAP: mean average precision; c: total number of categories; mAP@0.5: mAP when the intersection-over-union ratio is 0.5; mAP@0.5:0.95: average mAP at different intersection-over-union (IoU) thresholds from 0.5 to 0.95, with a step size of 0.

05. Step 8-3: Use the best weights obtained from training to test the test set, set the batch size to 1, and the image size to 640*640. Finally, output the precision P, recall R, mAP@0.5, mAP@0.5:0.95 values, and the inference time T for each image. Calculate the frame rate FPS, FPS = 1000 / T.

10. A lightweight remote sensing target detection device based on channel attention and multi-scale feature fusion, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 9.

Images