High-resolution remote sensing image target detection method and system based on feature fusion

Abstract

The invention provides a high-resolution remote sensing image target detection method and system based on feature fusion, and belongs to the technical field of data processing and application of high-resolution remote sensing images, and the method comprises the steps: carrying out the local feature extraction of a high-resolution remote sensing image through employing an improved OfficientNetv2 network, and obtaining a first local feature and a second local feature; inputting the first local feature into a MobileViT network pair to extract a global feature, and obtaining the global feature; inputting the global features and the second local features into a feature fusion module to obtain fusion features; and inputting the fusion feature to a detection end to obtain a detection result. According to the invention, false detection and missing detection of the target under different scales are effectively reduced, and the high-resolution remote sensing image target detection precision is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of data processing and application of high-resolution remote sensing images, and specifically relates to a method and system for high-resolution remote sensing image target detection based on feature fusion. Background Art

[0002] High-resolution remote sensing imagery (HRRI) has the characteristics of high resolution, strong timeliness, and wide coverage. Using HRRI for target detection is a basic method for automatic extraction, analysis, and understanding of image information, and it plays an important role in geological disaster detection, precision agriculture, urban planning, and other fields. Target detection in HRRI is to determine whether HRRI contains one or more objects belonging to the category of interest and locate the position of each predicted object in the image, which can provide a solid data foundation for many applications. However, HRRI has problems such as large target shape variation, small target object size and dense arrangement, and huge data volume, which makes it face many challenges in image target detection. So far, many detectors have been developed to detect different types of targets in high-resolution remote sensing images. Some researchers use manual features to extract features, such as saliency, scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG), bag of words (BOW), etc. These hand-crafted features perform well in specific target detection tasks. However, the versatility and robustness of manual features in other target detection tasks are poor. Therefore, how to extract deep and highly robust features based on the characteristics of high-resolution remote sensing images is still a hot issue in the current HRRI target detection research.

[0003] Deep learning models have attracted attention in recent years. The most popular deep learning method is the Convolutional Neural Network (CNN) model. CNN does not require hand-crafted features and requires fewer parameters than other networks because it can share the weights of the same filters. CNN can learn the essential features of the input image based on its specific network structure. The target features obtained using CNN have good versatility and robustness for different target detection tasks. CNN has been widely used in target detection in high-resolution remote sensing images. However, only using local features obtained by CNN for detection has limited ability to express multi-scale features in high-resolution remote sensing images, and the detection accuracy is low. Although researchers have effectively improved the detection accuracy of multi-scale targets by fusing deep features of different scales. However, how to build a network model with strong feature extraction capabilities and enhance the effectiveness and applicability of fusion between different features needs to be further addressed. Summary of the invention

[0004] In view of the shortcomings of the existing technology, this application proposes a high-resolution remote sensing image target detection method and system based on feature fusion.

[0005] In the first aspect, the present application proposes a high-resolution remote sensing image target detection method based on feature fusion, comprising:

[0006] Obtain high-resolution remote sensing images to be detected;

[0007] An improved EfficientNetv2 is used to extract local features of the high-resolution remote sensing image to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 sequentially includes: a first convolutional layer, a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer, wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature;

[0008] Input the first local feature into the MobileViT network to extract the global feature and obtain the global feature;

[0009] Input the global feature and the second local feature into the feature fusion module to obtain the fused feature;

[0010] The fusion feature is input to the detection end to obtain the detection result.

[0011] The MobileViT network includes: a first MobileViT block, a second MobileViT block and a third MobileViT block. The output of the second fused mixed convolutional layer is input into the first MobileViT block to obtain a first global feature, the output of the third fused mixed convolutional layer is input into the second MobileViT block to obtain a second global feature, and the output of the first mixed convolutional layer is input into the third MobileViT block to obtain a third global feature.

[0012] The feature fusion module includes: a first upsampling module, a first connection module, a first C3 module, a second upsampling module, a second connection module, a second C3 module, a third upsampling module, a third connection module, a third C3 module, a first convolution module, a fourth connection module, a fourth C3 module, a second convolution module, a fifth connection module, a fifth C3 module, a third convolution module, a sixth connection module, a sixth C3 module, a first ASFF module, a second ASFF module, a third ASFF module and a fourth ASFF module;

[0013] The first upsampling module is connected to the first connection module, the first connection module is connected to the first C3 module, the first C3 module is connected to the second upsampling module, the second upsampling module is connected to the second connection module, the second connection module is connected to the second C3 module, the second C3 module is connected to the third upsampling module, the third upsampling module is connected to the third connection module, the third connection module is connected to the third C3 module, the third C3 module is respectively connected to the first convolution module and the first ASFF module, the first convolution module is connected to the fourth connection module, the fourth connection module is connected to the fourth C3 module, the fourth C3 module is respectively connected to the second convolution module and the second ASFF module, the second convolution module is connected to the fifth connection module, the fifth connection module is connected to the fifth C3 module, the fifth C3 module is respectively connected to the third convolution module and the third ASFF module, the third convolution module is connected to the sixth connection module, the sixth connection module is connected to the sixth C3 module, the sixth C3 module is connected to the fourth ASFF module; the first C3 module is connected to the fifth connection module, and the second C3 module is connected to the fourth connection module.

[0014] The step of inputting the global feature and the second local feature into a feature fusion module to obtain a fused feature includes:

[0015] Inputting the second local features into the first upsampling module and the sixth connection module respectively;

[0016] inputting the first global feature into a third connection module;

[0017] inputting the second global feature into a second connection module;

[0018] inputting the third global feature into the first connection module;

[0019] Inputting the output of the first ASFF module to the detection end for detection to obtain a first detection result;

[0020] Inputting the output of the second ASFF module to the detection end for detection to obtain a second detection result;

[0021] Inputting the output of the third ASFF module to the detection end for detection to obtain a third detection result;

[0022] The output of the fourth ASFF module is input to the detection end for detection to obtain a fourth detection result.

[0023] In the improved EfficientNetv2, the EMA module is used to replace the fused mixed convolution layer and the SE module in the mixed convolution layer, wherein the fused mixed convolution layer includes: a first fused mixed convolution layer, a second fused mixed convolution layer and a third fused mixed convolution layer, and the mixed convolution layer includes: a first mixed convolution layer, a second mixed convolution layer and a third mixed convolution layer.

[0024] The EMA module includes: a feature extraction layer, an attention weight extraction layer, a cross-space learning layer and a feature combination layer;

[0025] The feature extraction layer is connected to the attention weight extraction layer, the attention weight extraction layer is connected to the cross-space learning layer, and the cross-space learning layer is connected to the feature combination layer;

[0026] The feature extraction layer is used to extract the features of the input data of the EMA module using a parallel structure;

[0027] The attention weight extraction layer is used to extract the attention weights of the features of the input data;

[0028] The cross-space learning layer is used to perform cross-space information aggregation on the features of input data with attention weights in different spatial dimensions to obtain an aggregation result;

[0029] The feature combination layer is used to fuse the aggregation results and the features of the input data, and use the fused data as the output of the EMA module.

[0030] The attention weight extraction layer includes: a height-direction average pooling layer, a width-direction average pooling layer, a splicing convolution layer, a first sigmod activation function, a second sigmod activation function, a weight output layer and a second convolution layer;

[0031] The height-direction average pooling layer and the width-direction average pooling layer are respectively connected to the splicing convolution layer, and the splicing convolution layer is respectively connected to the first sigmod activation function and the second sigmod activation function, and the first sigmod activation function and the second sigmod activation function are connected to the weight output layer;

[0032] The height-direction average pooling layer is used to encode the features of the input data in the height direction to obtain a first feature vector;

[0033] The width-wise average pooling layer is used to encode the features of the input data in the width direction to obtain a second feature vector;

[0034] The concatenated convolution layer is used to concatenate the first feature vector and the second feature vector, perform a convolution operation on the concatenated result, and then split the convolution result into a first new feature vector and a second new feature vector according to the height direction and the width direction respectively;

[0035] The first sigmoid activation function is used to calculate a mapping matrix of the first new eigenvector using the sigmoid activation function;

[0036] The second sigmoid activation function is used to calculate a mapping matrix of a second new eigenvector using the sigmoid activation function;

[0037] The weight output layer is used to multiply the features of the input data by the mapping matrix of the first new feature vector and the mapping matrix of the second new feature vector point by point to obtain a feature matrix, and perform average pooling and normalization operations on the feature matrix to obtain a first normalized relationship matrix;

[0038] The second convolutional layer is used to convolve the features of the input data, and then perform average pooling and normalization operations on the convolution results to obtain a second normalized relationship matrix. Each element in the first normalized relationship matrix and the second normalized relationship matrix represents the global weight distribution of the corresponding channel.

[0039] In the second aspect, the present application proposes a high-resolution remote sensing image target detection system based on feature fusion, comprising:

[0040] Image acquisition module, used to acquire high-resolution remote sensing images to be detected;

[0041] A local feature extraction module, used for extracting local features of the high-resolution remote sensing image by using an improved EfficientNetv2 to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 sequentially comprises: a first convolutional layer, a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer, wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature;

[0042] A global feature extraction module, used for inputting the first local feature into the MobileViT network to extract the global feature and obtain the global feature;

[0043] A feature fusion module, used for inputting the global feature and the second local feature into the feature fusion module to obtain a fused feature;

[0044] The detection result output module is used to input the fusion feature into the detection end to obtain the detection result.

[0045] In a third aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the high-resolution remote sensing image target detection method based on feature fusion.

[0046] In a fourth aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the high-resolution remote sensing image target detection method based on feature fusion.

[0047] Beneficial effects:

[0048] This application proposes a high-resolution remote sensing image target detection method and system based on feature fusion, constructs a network model with strong feature extraction capability, and enhances the effectiveness and applicability of fusion between different features. Acquiring more small target feature information greatly improves the small target detection capability, effectively reduces false detection and missed detection of targets at different scales, and improves the target detection accuracy of high-resolution remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the process of high-resolution remote sensing image target detection method based on feature fusion in the embodiment of the present application;

[0050] Figure 2 Flowchart of the high-resolution remote sensing image target detection method based on feature fusion in the embodiment of the present application;

[0051] Figure 3 Schematic diagram of the structure of the Fused-MBConv4 module and the MBConv6 module in the embodiment of the present application;

[0052] Figure 4 The overall structure diagram of the EMA-EfficientNetv2 feature extraction network of the embodiment of the present application;

[0053] Figure 5 Detailed structure diagram of MobileViT block in the embodiment of the present application;

[0054] Figure 6 ASFF structure diagram of the embodiment of the present application;

[0055] Figure 7 The improved detection layer module structure diagram of the embodiment of the present application;

[0056] Figure 8 EMA structure diagram of the embodiment of the present application;

[0057] Fig. 9 Comparison of the detection results of the embodiments of the present application, where (a) SSD detection result (b) Faster-RCNN detection result (c) RetinaNet detection result (d) DETR detection result (e) YOLOv5s detection result (f) detection result of the method of the present embodiment;

[0058] Fig.10 Principle block diagram of a high-resolution remote sensing image target detection system based on feature fusion in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0060] Embodiment 1:

[0061] This embodiment proposes a high-resolution remote sensing image target detection method based on feature fusion, such as Figure 1 , Figure 2 As shown, including:

[0062] Step S1: Acquire a high-resolution remote sensing image to be detected;

[0063] Step S2: using an improved EfficientNetv2 to extract local features from the high-resolution remote sensing image to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 sequentially comprises: a first convolutional layer (Conv 3×3), a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer (SPPF, Spatial PyramidPooling with Features), wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature;

[0064] Step S3: Input the first local feature into the MobileViT network to extract the global feature to obtain the global feature;

[0065] Step S4: inputting the global feature and the second local feature into a feature fusion module to obtain a fusion feature;

[0066] Step S5: input the fusion feature to the detection end to obtain the detection result.

[0067] In this embodiment, the high-resolution remote sensing image has the characteristics of high resolution, strong timeliness, and wide coverage. The general resolution is between 0.6 meters and 5 meters. Due to the above characteristics of the high-resolution remote sensing image, it is not easy to construct the feature network accurately, which ultimately leads to inaccurate feature extraction. In this embodiment, the improved EfficientNetv2, namely the EMA-EfficientNetv2 feature extraction network, is an EfficientNetv2-s network with an EMA module. In order to improve the model's ability to extract small target features, this embodiment uses a lightweight EfficientNetv2-s to perform local feature extraction on the high-resolution remote sensing image. EfficientNetv2-s adjusts the depth, width, and resolution of the network in proportion, and introduces a new module Fused-MBConv and a progressive learning strategy to reduce the number of parameters and complexity of the model, thereby greatly improving the model performance. The EfficientNetv2-s network is mainly composed of Fused-MBConv 3×3 (fused mixed convolution module, including: the first fused mixed convolution layer, the second fused mixed convolution layer, the third fused mixed convolution layer) and MBConv 3×3 (mixed convolution module including: the first mixed convolution layer, the second mixed convolution layer, the third mixed convolution layer) modules. Each Fused-MBConv and MBConv module contains SE modules (SE, Squeeze-and-Excitation). The SE module performs a two-dimensional global pooling operation on the feature map, converting the high-dimensional global feature map into a low-dimensional feature vector through compression operation, thereby extracting channel-level global features, and then using a multi-layer perceptron for nonlinear feature transformation. However, this operation only considers the encoding of inter-channel information, but loses the spatial information that plays an important role in remote sensing image target detection, which affects the model's ability to extract remote sensing image features to a certain extent. Therefore, this embodiment introduces the Efficient Multi-Scale Attention (EMA) module to further optimize and improve EfficientNetv2-s to improve the network's ability to extract key small target information and enhance the network's detection performance of small targets in remote sensing images. The SE attention module in the Fused-MBConv4 module and the MBConv6 module in the backbone feature extraction network EfficientNetv2-s is replaced with the EMA module. The structure of the Fused-MBConv4 module and the MBConv6 module that integrate EMA is as follows: Figure 3 As shown in Figure 2, the improved network can focus on the information of each channel of the remote sensing image while retaining the relevant effective features to the maximum extent.

[0068] In this embodiment, the first convolution layer adopts Conv 3×3, the first fused hybrid convolution layer adopts Fused-MBConv1 3×3, the second fused hybrid convolution layer adopts Fused-MBConv4 3×3, the third fused hybrid convolution layer adopts Fused-MBConv43×3, the first hybrid convolution layer adopts MBConv4 3×3, the second hybrid convolution layer adopts MBConv6 3×3, and the third hybrid convolution layer adopts MBConv6 3×3. Figure 4 shown.

[0069] In this embodiment, although the EfficientNetv2 feature extraction backbone network increases the model's ability to extract local features of remote sensing images, it cannot fully consider the global features of the image. In recent years, networks based on the self-attention mechanism, such as ViT (Vision Transformer), can establish global dependencies in the image and better capture the contextual information in the image. However, it has the problems of high model complexity and slow calculation speed. Therefore, this embodiment introduces a lightweight MobileVIT module after the Fused-MBConv4 module and the MBConv4 module to increase the model's ability to extract global features.

[0070] In the Transformer Encoder of the MobileViT (Mobile Vision Transformer) block, self-attention is not performed on all tokens, but only on tokens at the same position between different patches. This processing method can extract global features and reduce the computational burden.

[0071] The MobileViT network, such as Figure 1 As shown, it includes: a first MobileViT block, a second MobileViT block and a third MobileViT block. The output of the second fused mixed convolution layer is input into the first MobileViT block to obtain the first global feature, the output of the third fused mixed convolution layer is input into the second MobileViT block to obtain the second global feature, and the output of the first mixed convolution layer is input into the third MobileViT block to obtain the third global feature.

[0072] MobileViT (Mobile Vision Transformer) uses a visual Transformer based on a multi-head self-attention mechanism to effectively encode local and global information in tensors. The MobileViT model is trained using basic augmentation and is not very sensitive to L2 regularization. It also has the characteristics of a small number of learning representation parameters for spatial induction bias in lightweight convolutional neural networks (CNNs).

[0073] The MobileVIT block structure is as follows Figure 5 As shown. For a given input Y∈R C×W×H , first use a (3×3) standard convolution to encode local spatial information, and then use a 1×1 standard convolution layer to generate Y L ∈R d×H×W , project the input features into a high-dimensional space (or d-dimensional, d>C). Then Y L Expand into N non-overlapping and flat Y U ∈R d×H×P , to perform a global representation of the spatial inductive bias. Where P = ω × h, (P is the number of image blocks, h≤N and w≤N are the height and width of the image blocks respectively). Finally, for all p∈{1,2,…,P}, the relationship between image blocks is encoded into Y through Transformer. G ∈R d ×N×P .

[0074] There are two main layers in the Transformer encoder: multi-head self-attention mechanism layer (MSA) and multi-layer perceptron layer (MLP). The MSA layer is responsible for building global feature information, while the MLP layer is responsible for nonlinear transformation of each vector to better capture information at different positions. At the same time, there is a normalization layer before the MSA and MLP layers. The normalization layer forms a residual connection with each key layer. In MSA, the input feature Y is first U(p) Linear mapping is performed to obtain the query matrix Q, key matrix K, and value matrix V. Then each self-attention mechanism realizes dynamic aggregation of global information by calculating the three vectors Q, K, and V. The correlation between the current feature and other features is queried by dot product of the two vectors Q and K. The larger the dot product result, the stronger the correlation between the two features. V represents the actual feature information, and the specific calculation formula is expressed as:

[0075] Q=Y U(p) W Q K=Y U(p) W K V=Y U(p) W K (1)

[0076]

[0077] head i =Attention(Q i , K i , V i ) (3)

[0078] MSA(Y U(p) )=Concat(head1,head2,...,head h )W O (4)

[0079] Among them, Q i is the query matrix of the ith attention head, K i is the key matrix for each attention head, V i is the value matrix for each attention head, as well as They are all learnable parameter matrices. Each self-attention mechanism realizes dynamic aggregation of global information by calculating the three vectors Q, K, and V. Then the output of the MSA layer is input to the MLP layer for further feature extraction. The calculation process of the MLP layer is expressed as:

[0080] MLP(Y U(p) )=RELU(MSA(Y U(p) )W1+b1)W2+b2 (5)

[0081] Among them, MLP(Y U(p) ) is the input Y U(p) The MLP layer output is , W1 is the first weight matrix, W2 is the second weight matrix, b1 is the first bias vector, and b2 is the second bias vector.

[0082] The output Y of the Transfomer encoder G(p) , the calculation formula is:

[0083] Y G(p) =MLP(Y U(P) )+ YU(P) (6)

[0084] Among them, Y G(p) is the output of the Transfomer encoder.

[0085] After that, the MobileVIT block will be Y G ∈R d×N×P Fold to get Y F ∈R d×N×P , using point-by-point convolution to convert Y F Projected to a lower dimensional C space, and fused with Y through a 1×1 standard convolution cascade operation. U(p) Convolution is used to encode local information of the 3×3 region, and Y G(p) Encode the global information of p patches at the p-th position, so Y GEach pixel in can encode information from all pixels in Y. Therefore, the overall effective receptive field of MobileViT is H×W. The calculation formula is as follows:

[0086] Z=Conv 3×3 {Concat(Y,Conv 1×1 (Y F ))} (7)

[0087] Among them, Conv 1×1 is a 1×1 standard convolution, Concat is a cascade operation, Conv 3×3 is a 3×3 convolution, and Z is the output of the MobileViT block.

[0088] In this embodiment, the adaptive spatial feature fusion method of the fusion module (Active Scale Filter Feature) of low-level detail information and high-level semantic information is a feature fusion technology for target detection, which aims to improve the model's detection ability for targets of different scales. The core idea of ​​ASFF is to dynamically select the optimal feature scale, and its structure is as follows: Figure 6 ASFF improves the adaptability and robustness of the target detection model to targets of different sizes through dynamic optimal selection, especially in scenarios where small and large targets coexist.

[0089] FPN generates feature layers of multiple scales, each with different resolutions and semantic information, namely Level 1, Level 2, and Level 3 in the figure. ASFF adaptively adjusts feature weights and performs spatial filtering on features at other levels, which can effectively solve the conflict problem of features at different levels of FPN. The specific fusion process is as follows:

[0090]

[0091] in, It is the feature vector output by the ASFF network. Represents the input feature vector at the corresponding position. are learnable parameters for the feature maps at three different levels to the lth layer. The three weight parameters are obtained by 1×1 convolution of the feature maps of Level 1, Level 2, and Level 3, and the sum of the weight parameters a, β, and γ is 1. After normalization, the value of the weight parameter is in [0, 1].

[0092] In step S3, the feature fusion module, such as Figure 1As shown, it includes: a first upsampling module, a first connection module, a first C3 module, a second upsampling module, a second connection module, a second C3 module, a third upsampling module, a third connection module, a third C3 module, a first convolution module, a fourth connection module, a fourth C3 module, a second convolution module, a fifth connection module, a fifth C3 module, a third convolution module, a sixth connection module, a sixth C3 module, a first ASFF module, a second ASFF module, a third ASFF module and a fourth ASFF module;

[0093] The first upsampling module is connected to the first connection module, the first connection module is connected to the first C3 module, the first C3 module is connected to the second upsampling module, the second upsampling module is connected to the second connection module, the second connection module is connected to the second C3 module, the second C3 module is connected to the third upsampling module, the third upsampling module is connected to the third connection module, the third connection module is connected to the third C3 module, the third C3 module is respectively connected to the first convolution module and the first ASFF module, the first convolution module is connected to the fourth connection module, the fourth connection module is connected to the fourth C3 module, the fourth C3 module is respectively connected to the second convolution module and the second ASFF module, the second convolution module is connected to the fifth connection module, the fifth connection module is connected to the fifth C3 module, the fifth C3 module is respectively connected to the third convolution module and the third ASFF module, the third convolution module is connected to the sixth connection module, the sixth connection module is connected to the sixth C3 module, the sixth C3 module is connected to the fourth ASFF module; the first C3 module is connected to the fifth connection module, and the second C3 module is connected to the fourth connection module.

[0094] The step of inputting the global feature and the second local feature into a feature fusion module to obtain a fused feature includes:

[0095] Inputting the second local features into the first upsampling module and the sixth connection module respectively;

[0096] inputting the first global feature into a third connection module;

[0097] inputting the second global feature into a second connection module;

[0098] inputting the third global feature into the first connection module;

[0099] Inputting the output of the first ASFF module to the detection end for detection to obtain a first detection result;

[0100] Inputting the output of the second ASFF module to the detection end for detection to obtain a second detection result;

[0101] Input the output of the third ASFF module into the detection end for detection to obtain the third detection result;

[0102] Input the output of the fourth ASFF module into the detection end for detection to obtain the fourth detection result.

[0103] In the improved EfficientNetv2, the EMA module is adopted to replace the fused hybrid convolution layer and the SE module in the hybrid convolution layer. Among them, the fused hybrid convolution layer includes: the first fused hybrid convolution layer, the second fused hybrid convolution layer, and the third fused hybrid convolution layer, and the hybrid convolution layer includes: the first hybrid convolution layer, the second hybrid convolution layer, and the third hybrid convolution layer.

[0104] In this embodiment, the EMA uses a grouped structure to effectively reduce the required number of parameters and computational overhead, and establishes long and short dependencies between features by designing a multi-scale parallel sub-network and a cross-space learning method. Its structure is as Figure 8 shown.

[0105] It can be Figure 8 seen that the EMA consists of four parts. The EMA module includes: a feature extraction layer, an attention weight extraction layer, a cross-space learning layer, and a feature combination layer; the feature extraction layer is connected to the attention weight extraction layer, the attention weight extraction layer is connected to the cross-space learning layer, and the cross-space learning layer is connected to the feature combination layer.

[0106] (1) Feature extraction layer: The feature extraction layer is used to extract the features of the input data of the EMA module by using a parallel structure;

[0107] The EMA uses a parallel sub-structure to avoid performance degradation caused by complex sequential processing and deep convolution, and extracts pixel-level attention features. For any given input feature Figure X ∈R C×H×W , the EMA divides X into G sub-features in the channel dimension, X = [X0, X i ,..., X G-1 , X i ∈R C / / G×H×W , where G << C is used to learn different semantics, and it is assumed that the learned attention weight descriptor will be used to enhance the feature representation of the region of interest in each sub-feature.

[0108] (2) Attention weight extraction layer: The attention weight extraction layer is used to extract the attention weights of the features of the input data;

[0109] The attention weight extraction layer includes: an average pooling layer in height direction, an average pooling layer in width direction, a splicing convolution layer, a first sigmod activation function, a second sigmod activation function, a weight output layer and a second convolution layer; the average pooling layer in height direction and the average pooling layer in width direction are respectively connected to the splicing convolution layer, the splicing convolution layer is respectively connected to the first sigmod activation function and the second sigmod activation function, and the first sigmod activation function and the second sigmod activation function are connected to the weight output layer;

[0110] The height-direction average pooling layer is used to encode the features of the input data in the height direction to obtain a first eigenvector; the width-direction average pooling layer is used to encode the features of the input data in the width direction to obtain a second eigenvector; the concatenated convolution layer is used to concatenate the first eigenvector and the second eigenvector, and perform a convolution operation on the concatenated result, and then split the convolution result into a first new eigenvector and a second new eigenvector in the height direction and the width direction respectively; the first sigmoid activation function is used to calculate the mapping matrix of the first new eigenvector using the sigmoid activation function; the second sigmoid activation function is used to The sigmoid activation function is used to calculate the mapping matrix of the second new eigenvector; the weight output layer is used to multiply the features of the input data with the mapping matrix of the first new eigenvector and the mapping matrix of the second new eigenvector point by point to obtain a feature matrix, and average pooling and normalization operations are performed on the feature matrix to obtain a first normalized relationship matrix; the second convolution layer is used to convolve the features of the input data, and then average pooling and normalization operations are performed on the convolution result to obtain a second normalized relationship matrix, and each element in the first normalized relationship matrix and the second normalized relationship matrix represents the global weight distribution of the corresponding channel.

[0111] EMA uses three routes to extract attention weights, where 1×1 convolution is in the first two routes and 3×3 convolution is in the third route. EMA models the cross-channel interaction information in both directions of the channel. In the branch of 1×1 convolution, a global average pooling operation is added to encode the channels in both directions of the channel. First, input the feature X∈R C / / G×H×W After pooling operations in two directions, the feature vectors X_h∈R are obtained respectively C / / G×1×W and X_w∈

[0112] R C / / G×H×1 The calculation formula for pooling is as follows:

[0113]

[0114]

[0115] Among them, W is the width dimension, H is the height dimension, X(H, i) is the vector in the width direction, and X(j, w) is the vector in the height dimension.

[0116] The concat function concatenates the feature vectors in the height direction and the width direction to obtain the feature vector hw, and then uses 1x1 convolution to adjust the number of channels of the concatenated feature hw, and learns weights to adjust the relationship between different channels, and then splits hw along the height direction and width direction to obtain new feature vectors X_h and X_w. After the sigmod activation function, the mapping matrix of X_h and X_w is obtained, and then X is multiplied point by point with the output results of the two sigmods to obtain the prominent important feature X_hw. The calculation formula is as follows:

[0117]

[0118] Finally, the GN group normalized feature X is processed by 2D global average pooling and nonlinear Softmax function. 1 Perform operations to obtain the normalized relationship matrix The probability value in the relationship matrix represents the global weight distribution of the channels in each group, which is used for subsequent weighted operations. The specific operation is shown in the formula:

[0119]

[0120] Among them, X 1 (i, j) is the first eigenvector after normalization. In the 3×3 convolution branch, the GN normalization and average pooling operations are omitted to extract multi-scale feature representation. First, the input feature X∈R V / / G×H×W After 3x3 convolution operation, we get X 2 ∈R C / / G×H×W , and then use 2D global average pooling and nonlinear Softmax function to transform feature X 2 Perform row operations to obtain the normalized relationship matrix The calculation formula is as follows:

[0121]

[0122] Among them, X 2 (i, j) is the normalized second eigenvector.

[0123] (3) Cross-spatial learning layer: The cross-spatial learning layer is used to perform cross-spatial information aggregation on the features of the input data with attention weights in different spatial dimensions to obtain an aggregation result. EMA proposes a cross-spatial information aggregation method in different spatial dimensions to achieve richer feature aggregation. Two tensors are introduced, namely the output of the 1×1 branch and the output of the 3×3 branch. The global spatial information output by the 1×1 branch and the 3×3 branch is encoded using 2D global average pooling. The channel features of the output of the smallest branch are converted to the corresponding dimensional shape, i.e., R 1×C / / G ×R C / / G×HW Cross-space learning expands the feature space, efficiently extracts the dependencies between the three channels, preserves the spatial structure information in the channels, and reduces computational overhead.

[0124] The cross-space learning layer and the feature combination layer include: a normalization function layer, a first average pooling layer, a third activation function layer, a first dot product function layer, a second average pooling layer, a fourth activation function layer, and a second dot product function layer;

[0125] The output of the attention weight extraction layer is input into the normalization function layer, the output of the normalization function layer is input into the first average pooling layer and the second dot multiplication function layer respectively, the output layer of the first average pooling layer is input into the third activation function layer, the output of the third activation function layer and the output of the second convolution layer are input into the first dot multiplication function layer respectively, the output of the second convolution layer is input into the second average pooling layer, the output of the second average pooling layer is input into the fourth activation function layer, the output of the fourth activation function layer is input into the second dot multiplication function layer, the output of the first dot multiplication function layer is added to the output of the second dot multiplication function layer to obtain the output of the cross-space learning layer.

[0126] (4) Feature combination layer: The feature combination layer is used to fuse the aggregation results and the features of the input data. The fused data is used as the output of the EMA module. The output features of the three routes are calculated as the aggregation of two spatial attention weight values. The Sigmoid activation function highlights the context pixels of all pixels. The final output Y∈R C×W×H With input features Figure X ∈R C×W×H The dimensions are the same. The calculation formula is as follows:

[0127]

[0128] Among them, weights is the weight of the feature combination layer, Reshape() is the transposition operation, and Y is the output of the feature combination layer.

[0129] In this embodiment, in the high-resolution remote sensing image target detection data set, the aggregation of small targets in some images leads to serious mutual occlusion, the scale differences between the target instances to be detected are large, large targets occupy most of the area, and some small targets are less than 10% of the image size, and the feature information is easily lost. The original Yolov5s detection layer outputs three feature maps of different sizes, namely 20×20, 40×40, and 80×80. When the target is too small, it cannot be detected. This paper adds a 160×160 small target detection head on the basis of the original model to enhance the sensitivity to smaller targets. Features are initially extracted from the sixth layer of the backbone network, and Concat splicing is used to fuse the shallow features extracted from the neck network structure with the contextual information extracted by the EAM attention mechanism, and the fourth detection head is output as the small target detection head. This improvement slightly increases the computational complexity of the model, but greatly improves the ability to detect small targets by obtaining more small target feature information, effectively reducing false detection and missed detection of targets at different scales. The improved detection layer is as follows Figure 7 shown.

[0130] Test results:

[0131] In this embodiment, the number and size of convolution kernels in the convolutional neural network, the learning rate, the number of network iterations epoch, the number of batch samples batch size, and the window size window size will affect the detection effect. When the patch size in the MobileVIT block is 2, the learning rate value is 0.01, when the epoch value is 250, the batchsize = 32, and the window size = 15×15, the classification accuracy is optimal.

[0132] In this embodiment, a comparison of the detection of the DIOR dataset using the method of this embodiment, the Faster R-CNN method, the RetinaNet method, the DETR method, the YOLOv5s method, and the SSD method is also given. The detection accuracy (AP) and overall detection accuracy (mAP) of each method are shown in Table 1. Among them, the mAP of the method of this embodiment is about 71.8%, which is improved compared with the detection results of other methods, and the detection accuracy of the 20 best categories of objects is obtained. The detection results of other methods and the method of this embodiment are compared as follows: Fig. 9 shown.

[0133] Table 1 Test results (%)

[0134]

[0135] Embodiment 2:

[0136] This embodiment proposes a high-resolution remote sensing image target detection system based on feature fusion, such as Fig.10 As shown, including:

[0137] Image acquisition module, used to acquire high-resolution remote sensing images to be detected;

[0138] A local feature extraction module is used to extract local features of the high-resolution remote sensing image by using an improved EfficientNetv2 to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 sequentially comprises: a first convolutional layer, a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer (SPPF, Spatial PyramidPooling with Features), wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature;

[0139] A global feature extraction module, used for inputting the first local feature into the MobileViT network to extract the global feature and obtain the global feature;

[0140] A feature fusion module, used for inputting the global feature and the second local feature into the feature fusion module to obtain a fused feature;

[0141] The detection result output module is used to input the fusion feature into the detection end to obtain the detection result.

[0142] Embodiment 3:

[0143] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the high-resolution remote sensing image target detection method based on feature fusion.

[0144] The electronic device may be a mobile phone, a computer or a tablet computer, etc., including a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the high-resolution remote sensing image target detection method based on feature fusion as described in the embodiment is implemented. It is understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0145] The processor is used to execute all or part of the steps in the high-resolution remote sensing image target detection method based on feature fusion as described in the above embodiment. The memory is used to store various types of data, which may include instructions of any application or method in the electronic device, as well as data related to the application.

[0146] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the high-resolution remote sensing image target detection method based on feature fusion described in the above embodiments.

[0147] Embodiment 4:

[0148] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed and implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0149] The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the high-resolution remote sensing image target detection method based on feature fusion described in each embodiment of the present application.

[0150] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which computer programs are stored. When the computer program is executed by the processor, the various steps of the high-resolution remote sensing image target detection method based on feature fusion described above can be implemented.

[0151] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0152] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A high-resolution remote sensing image target detection method based on feature fusion, characterized in that: include: Obtain high-resolution remote sensing images to be detected; An improved EfficientNetv2 network is used to extract local features of the high-resolution remote sensing image to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 network sequentially comprises: a first convolutional layer, a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer, wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature; Input the first local feature into the MobileViT network to extract the global feature and obtain the global feature; Input the global feature and the second local feature into the feature fusion module to obtain the fused feature; The fusion feature is input to the detection end to obtain the detection result.

2. The high-resolution remote sensing image target detection method based on feature fusion according to claim 1 is characterized in that: The MobileViT network includes: a first MobileViT block, a second MobileViT block and a third MobileViT block. The output of the second fused mixed convolutional layer is input into the first MobileViT block to obtain a first global feature, the output of the third fused mixed convolutional layer is input into the second MobileViT block to obtain a second global feature, and the output of the first mixed convolutional layer is input into the third MobileViT block to obtain a third global feature.

3. The high-resolution remote sensing image target detection method based on feature fusion according to claim 1 is characterized in that: The feature fusion module includes: a first upsampling module, a first connection module, a first C3 module, a second upsampling module, a second connection module, a second C3 module, a third upsampling module, a third connection module, a third C3 module, a first convolution module, a fourth connection module, a fourth C3 module, a second convolution module, a fifth connection module, a fifth C3 module, a third convolution module, a sixth connection module, a sixth C3 module, a first ASFF module, a second ASFF module, a third ASFF module and a fourth ASFF module; The first upsampling module is connected to the first connection module, the first connection module is connected to the first C3 module, the first C3 module is connected to the second upsampling module, the second upsampling module is connected to the second connection module, the second connection module is connected to the second C3 module, the second C3 module is connected to the third upsampling module, the third upsampling module is connected to the third connection module, the third connection module is connected to the third C3 module, the third C3 module is respectively connected to the first convolution module and the first ASFF module, the first convolution module is connected to the fourth connection module, the fourth connection module is connected to the fourth C3 module, the fourth C3 module is respectively connected to the second convolution module and the second ASFF module, the second convolution module is connected to the fifth connection module, the fifth connection module is connected to the fifth C3 module, the fifth C3 module is respectively connected to the third convolution module and the third ASFF module, the third convolution module is connected to the sixth connection module, the sixth connection module is connected to the sixth C3 module, the sixth C3 module is connected to the fourth ASFF module; the first C3 module is connected to the fifth connection module, and the second C3 module is connected to the fourth connection module.

4. The high-resolution remote sensing image target detection method based on feature fusion according to claim 3 is characterized in that: The step of inputting the global feature and the second local feature into a feature fusion module to obtain a fused feature includes: Inputting the second local features into the first upsampling module and the sixth connection module respectively; inputting the first global feature into a third connection module; inputting the second global feature into a second connection module; inputting the third global feature into the first connection module; Inputting the output of the first ASFF module to the detection end for detection to obtain a first detection result; Inputting the output of the second ASFF module to the detection end for detection to obtain a second detection result; Inputting the output of the third ASFF module to the detection end for detection to obtain a third detection result; The output of the fourth ASFF module is input to the detection end for detection to obtain a fourth detection result.

5. The high-resolution remote sensing image target detection method based on feature fusion according to claim 1 is characterized in that: In the improved EfficientNetv2 network, the EMA module is used to replace the fused mixed convolution layer and the SE module in the mixed convolution layer, wherein the fused mixed convolution layer includes: a first fused mixed convolution layer, a second fused mixed convolution layer and a third fused mixed convolution layer, and the mixed convolution layer includes: a first mixed convolution layer, a second mixed convolution layer and a third mixed convolution layer.

6. The high-resolution remote sensing image target detection method based on feature fusion according to claim 5 is characterized in that: The EMA module includes: a feature extraction layer, an attention weight extraction layer, a cross-space learning layer and a feature combination layer; The feature extraction layer is connected to the attention weight extraction layer, the attention weight extraction layer is connected to the cross-space learning layer, and the cross-space learning layer is connected to the feature combination layer; The feature extraction layer is used to extract the features of the input data of the EMA module using a parallel structure; The attention weight extraction layer is used to extract the attention weights of the features of the input data; The cross-space learning layer is used to perform cross-space information aggregation on the features of input data with attention weights in different spatial dimensions to obtain an aggregation result; The feature combination layer is used to fuse the aggregation results and the features of the input data, and use the fused data as the output of the EMA module.

7. The high-resolution remote sensing image target detection method based on feature fusion according to claim 6 is characterized in that: The attention weight extraction layer includes: a height-direction average pooling layer, a width-direction average pooling layer, a splicing convolution layer, a first sigmod activation function, a second sigmod activation function, a weight output layer and a second convolution layer; The height-direction average pooling layer and the width-direction average pooling layer are respectively connected to the splicing convolution layer, and the splicing convolution layer is respectively connected to the first sigmod activation function and the second sigmod activation function, and the first sigmod activation function and the second sigmod activation function are connected to the weight output layer; The height-direction average pooling layer is used to encode the features of the input data in the height direction to obtain a first feature vector; The width-wise average pooling layer is used to encode the features of the input data in the width direction to obtain a second feature vector; The concatenated convolution layer is used to concatenate the first feature vector and the second feature vector, perform a convolution operation on the concatenated result, and then split the convolution result into a first new feature vector and a second new feature vector according to the height direction and the width direction respectively; The first sigmoid activation function is used to calculate a mapping matrix of the first new eigenvector using the sigmoid activation function; The second sigmoid activation function is used to calculate a mapping matrix of a second new eigenvector using the sigmoid activation function; The weight output layer is used to multiply the features of the input data by the mapping matrix of the first new feature vector and the mapping matrix of the second new feature vector point by point to obtain a feature matrix, and perform average pooling and normalization operations on the feature matrix to obtain a first normalized relationship matrix; The second convolutional layer is used to convolve the features of the input data, and then perform average pooling and normalization operations on the convolution results to obtain a second normalized relationship matrix. Each element in the first normalized relationship matrix and the second normalized relationship matrix represents the global weight distribution of the corresponding channel.

8. A high-resolution remote sensing image target detection system based on feature fusion, characterized in that: include: Image acquisition module, used to acquire high-resolution remote sensing images to be detected; A local feature extraction module, used for extracting local features of the high-resolution remote sensing image by using an improved EfficientNetv2 network to obtain a first local feature and a second local feature, wherein the improved EfficientNetv2 network sequentially comprises: a first convolutional layer, a first fused mixed convolutional layer, a second fused mixed convolutional layer, a third fused mixed convolutional layer, a first mixed convolutional layer, a second mixed convolutional layer, a third mixed convolutional layer and a spatial pyramid pooling layer, wherein the outputs of the second fused mixed convolutional layer, the third fused mixed convolutional layer and the first mixed convolutional layer are used as the first local feature, and the output of the spatial pyramid pooling layer is used as the second local feature; A global feature extraction module, used for inputting the first local feature into the MobileViT network to extract the global feature and obtain the global feature; A feature fusion module, used for inputting the global feature and the second local feature into the feature fusion module to obtain a fused feature; The detection result output module is used to input the fusion feature into the detection end to obtain the detection result.

9. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the high-resolution remote sensing image target detection method based on feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores executable instructions, which, when executed, enable the processor to execute the high-resolution remote sensing image target detection method based on feature fusion as described in any one of claims 1 to 7.

Images