Remote sensing target detection method and system based on multi-scale feature fusion, and medium

Abstract

The invention belongs to the technical field of computer vision, and relates to a remote sensing target detection method and system based on multi-scale feature fusion and a medium, and the method comprises the steps: obtaining a remote sensing image, inputting the remote sensing image into a bottom-to-top feature extraction backbone network, and obtaining a multi-level feature map Ck; inputting the Ck into a multidirectional feature strengthening network to obtain a multi-scale feature map Yk; inputting the Yk into a feature extraction backbone network from top to bottom to obtain a multi-scale fusion feature map Pk; inputting the Pk into a target detection network for target detection to obtain a detection result; the multidirectional feature enhancement network performs all-directional frequency domain information enhancement on each level of feature map, performs feature enhancement of different scales on frequency domain enhancement features, and performs bidirectional space coding and feature weight optimization on H and W dimensions by using a coordinate attention mechanism. Multiple information of frequency domains and space domains of low dimensions and high latitudes of the feature layer is comprehensively concerned, the detection capability of small targets is effectively improved, and the capability of coping with complex backgrounds is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a remote sensing target detection method, system and medium based on multi-scale feature fusion. Background Art

[0002] Remote Sensing Object Detection aims to accurately identify and locate various target objects from remote sensing images, and plays a key role in many fields such as geographic surveying and mapping, environmental monitoring, military reconnaissance, and urban planning. Remote sensing images can provide large-area, macroscopic surface information, including a rich variety of targets, such as buildings, roads, vegetation, water bodies, and various means of transportation. However, in practical applications, due to the characteristics of remote sensing images such as large changes in target scale, different resolutions, and complex backgrounds, how to effectively extract and fuse multi-scale features becomes the key to improving detection performance. This has prompted related research to become the focus of this field.

[0003] At present, using deep learning methods to extract effective features from remote sensing images to achieve target detection has become the mainstream. However, due to the complex characteristics of remote sensing images, such as large changes in target scale, uncertain direction, and complex background, existing detection methods often have limitations. For example, the Feature Pyramid Network (FPN) and its improved model have low utilization when processing shallow features, and small target features are easily lost during the transmission of shallow features to deep layers. At the same time, the traditional backbone network lacks global information dependencies, resulting in a lack of feature expression capabilities under complex backgrounds. These factors combined have caused the irrationality of the feature extraction structure of the existing methods, and they are unable to fully obtain the feature information of the targets in remote sensing images.

[0004] In summary, many existing studies focus on improving spatial domain models while ignoring the importance of frequency domain information. In addition, small target features are easily lost when shallow features are transmitted to deep layers. At the same time, there is a lack of global information dependency, which leads to a lack of feature expression capabilities in complex backgrounds. Summary of the invention

[0005] In order to solve the above-mentioned prior art problems, in the first aspect, the present invention adopts a remote sensing target detection method based on multi-scale feature fusion, including: acquiring a remote sensing image, inputting the remote sensing image into a trained remote sensing target detection model, and obtaining a detection result; the remote sensing target detection model includes: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-directional feature enhancement network, and a target detection network; the training process of the remote sensing target detection model includes:

[0006] S1: Obtain remote sensing images and input them into a bottom-up feature extraction backbone network to obtain multi-level feature maps;

[0007] S2: Input the multi-level feature map into the multi-directional feature enhancement network to obtain a multi-scale feature map;

[0008] S3: Input the multi-scale feature map into the top-down feature extraction backbone network to obtain a multi-scale fusion feature map;

[0009] S4: Input the multi-scale fusion feature map into the target detection network to perform target detection and obtain the detection result;

[0010] S5: Calculate the loss function value according to the detection results, and update the model parameters according to the loss function value. When the loss function value is minimum, the trained remote sensing target detection model is obtained.

[0011] The bottom-up feature extraction backbone network includes: n layers of series-connected feature extraction layers; the bottom-up feature extraction backbone network processes the remote sensing image I including:

[0012] S11, inputting the remote sensing image I into the first feature extraction layer, and inputting the output C1 of the first feature extraction layer into the second feature extraction layer;

[0013] S12, the output C of the k-1th feature extraction layer k-1 Input the kth feature extraction layer; where k is the index of the feature extraction layer of the bottom-up feature extraction backbone network;

[0014] S13, repeat step S12 until the output C of the last feature extraction layer is obtained n .

[0015] The feature extraction layer includes: convolution module, residual module and global pooling module; the first feature extraction layer processes the remote sensing image I including:

[0016] S111: Input the remote sensing image I into the convolution module to obtain a feature map Z;

[0017] S112: Perform a pooling operation on the feature map Z to obtain a feature map Z′;

[0018] S113: Input the pooled feature map Z′ into the residual module to obtain the feature map X;

[0019] S114: Input the feature map X into the global pooling module to obtain the feature map C1.

[0020] The multi-directional feature enhancement network includes: a feature enhancement module and a coordinate attention module; the multi-directional feature enhancement network processes the hierarchical feature map including:

[0021] S21: for the level feature map C k Perform convolution to obtain the hierarchical feature map C′ with enhanced channel dimension k ;

[0022] S22: Hierarchical feature map C′ with enhanced channel dimension k Input the feature enhancement module to obtain the feature enhancement graph F with enhanced channel dimension k ;

[0023] S23: Feature enhancement graph F that enhances the channel dimension k and the hierarchical feature map C′ k Input the coordinate attention module to obtain the multi-scale feature map Y k .

[0024] The feature enhancement module includes: HW dimension branch, CW dimension branch, CH dimension branch, local branch and splicer; the feature enhancement module enhances the hierarchical feature map C′ of the channel dimension k Processing includes:

[0025] S221: The hierarchical feature map C′ k Split along the channel dimension to obtain the split feature x k,s ,s=[1,4]; s is the index of the feature after splitting;

[0026] S222: feature x k,1 Input the HW dimension branch to obtain the hierarchical features of the HW dimension

[0027] S223: feature x k,2 Input the CW dimension branch to obtain the hierarchical features of the CW dimension

[0028] S224: feature x k,3 Input the CH dimension branch to obtain the hierarchical features of the CH dimension

[0029] S225: feature x k,4 Input the local branch to obtain the hierarchical features of local information

[0030] S226: Features Input the splicer to obtain the feature enhancement graph F with channel dimension enhancement k ; Where H is the height, W is the width, and C is the number of channels.

[0031] The HW dimension branch includes: Fourier and inverse Fourier transform modules and a dilated convolution pyramid; the HW dimension branch transforms the multi-level feature map x k,1Processing includes:

[0032] S2211: x k,1 Input Fourier and inverse Fourier transform modules to obtain the hierarchical features x containing boundary information k,1 ′;

[0033] S2212: x k,1 ' Input the dilated convolutional pyramid to obtain the hierarchical features of HW dimension

[0034] The dilated convolution pyramid includes: 1×1 convolution module, 3×3 convolution module with dilation rate of 3, 3×3 convolution module with dilation rate of 6, and 3×3 convolution module with dilation rate of 9; the dilated convolution pyramid has a large number of convolution modules, and has a large number of convolution modules with dilation rate of 9. k,1 ′The dilated convolution operation includes: k,1 ' Input the 1×1 convolution module, the 3×3 convolution module with a void rate of 3, the 3×3 convolution module with a void rate of 6, and the 3×3 convolution module with a void rate of 9 respectively, add the outputs of the 1×1 convolution module, the 3×3 convolution module with a void rate of 3, the 3×3 convolution module with a void rate of 6, and the 3×3 convolution module with a void rate of 9, and perform a 1×1 convolution on the added result to obtain the output feature

[0035] The coordinate attention module includes an H-dimension branch and a W-dimension branch; the feature enhancement diagram F of the coordinate attention module for channel dimension enhancement k Processing includes:

[0036] S231: Enhance the feature map F k Input the H-dimensional branch for H-dimensional encoding to obtain the encoding features Among them, H is the height, h is the feature enhancement map F k The index of the height;

[0037] S232: Enhance the feature map F k Input the W-dimensional branch for W-dimensional encoding to obtain the encoded features Among them, W is the width, w is the feature enhancement map F k The index of the width;

[0038] S233: Yes and Combine to get the feature weight z k , perform convolution operation on the feature weight z to obtain the feature weight f k ;

[0039] S234: f by channel dimension k Split and get the feature weight f hk and f w k ;

[0040] S235: f h k Perform channel convolution F h Get the channel attention weight g h k ;

[0041] S236: f w k Perform channel convolution F h Get the channel attention weight g w k ;

[0042] S237: According to the channel attention weight g h k and g w k And the hierarchical feature map C′ k Compute multi-scale feature maps.

[0043] In a second aspect, the present invention adopts a system of a remote sensing target detection method based on the multi-scale feature fusion, comprising:

[0044] A data acquisition module, used for acquiring remote sensing images;

[0045] A bottom-up feature extraction backbone module is used to extract multi-level feature maps of remote sensing images;

[0046] The multi-directional feature enhancement module is used to process the multi-level feature maps of remote sensing images to obtain multi-scale feature maps;

[0047] The top-down feature extraction backbone module is used to process the multi-scale feature map to obtain the multi-scale fusion feature map;

[0048] The target detection module is used to classify the multi-scale fusion feature map to obtain the remote sensing image target detection category.

[0049] In a third aspect, the present invention adopts a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned remote sensing target detection method based on multi-scale feature fusion are implemented.

[0050] Beneficial effects:

[0051] 1. After feature extraction, the present invention performs Fourier transform on the three axes of the input features, assigns external weights in the frequency domain, and then performs inverse Fourier transform to transform the features back to the spatial domain; by focusing on the frequency domain information, the model's edge recognition and detection is enhanced, and then the coordinate attention mechanism is used to perform bidirectional spatial encoding and feature weight optimization on the H and W dimensions of the frequency domain enhanced feature map to generate a multi-scale feature map, which fully pays attention to the multiple information in the frequency domain and spatial domain of the feature layer, better copes with the challenges brought by the different scales and directions of targets in remote sensing images, effectively improves the detection capability of small targets and enhances the ability to cope with the difficult detection of edges in complex backgrounds; 2. The ACMP of the present invention adopts the multi-scale idea in the frequency domain to Rich multi-scale feature information is further extracted from the features of domain enhancement, and multi-scale semantic information is well preserved while obtaining sufficient boundary information, effectively improving the detection capability of small targets and enhancing the ability to cope with complex backgrounds; 3. The ACMP of the present invention removes the pooling layer of the dilated spatial pyramid pooling ASPP to better adapt to tiny targets in remote sensing images, and sets the dilated convolution rate rate to 3, 6, and 9 respectively to be more suitable for small target detection in remote sensing images. At the same time, the output of the dilated convolution is fused by adding an add operation instead of a concat operation to better retain more hierarchical feature information, effectively improving the detection capability of small targets and enhancing the ability to cope with complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flow chart of a remote sensing target detection method based on multi-scale feature fusion in an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the structure of a remote sensing target detection model in an embodiment of the present invention;

[0054] Figure 3 It is a schematic diagram of the structure of a multi-directional feature enhancement network in an embodiment of the present invention;

[0055] Figure 4 Schematic diagram of the structure of a dilated convolutional pyramid device in an embodiment of the present invention;

[0056] Figure 5 Schematic diagram of the structure of the coordinate attention module in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, one aspect of the present invention provides a remote sensing target detection method based on multi-scale feature fusion, including: inputting a remote sensing image in a complex scene into a trained remote sensing target detection model, utilizing the multi-scale features extracted by the model and fusing them, and finally realizing remote sensing image target detection and classification; the remote sensing target detection model includes: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-directional feature enhancement network, and a target detection network;

[0059] The training process of the remote sensing target detection model includes:

[0060] S1: Obtain a remote sensing image dataset, input the remote sensing images in the remote sensing image dataset into a bottom-up feature extraction backbone network, and obtain a multi-level feature map;

[0061] In one embodiment, the low-resolution remote sensing image can be obtained from an existing public data set, or can be obtained by photographing with a camera, a video camera, a mobile phone or other device, and the present invention does not make any specific limitation to this.

[0062] In one embodiment, when training a remote sensing target detection model, 2420 images are obtained from the UCAS-AOD dataset, and the model sample set is divided into a training set and a test set. The training set uses 2000 images from the UCAS-AOD dataset, and the test set uses 420 images from the UCAS-AOD dataset.

[0063] like Figure 2 As shown, both the bottom-up feature extraction backbone network and the top-down feature extraction backbone network include: n layers of feature extraction layers connected in series; the bottom-up feature extraction backbone network processes the remote sensing image I by:

[0064] S11, inputting the remote sensing image I into the first feature extraction layer, and inputting the output C1 of the first feature extraction layer into the second feature extraction layer;

[0065] S12, the output C of the k-1th feature extraction layer k-1 Input the kth feature extraction layer; where k is the index of the feature extraction layer of the bottom-up feature extraction backbone network;

[0066] S13, repeat step S12 until the output C of the last feature extraction layer is obtained n .

[0067] Each feature extraction layer includes: a convolution module, a residual module and a global pooling module; the first feature extraction layer processes the remote sensing image I including:

[0068] S111: Input the remote sensing image I into the convolution module to obtain the feature map Z. The calculation formula is as follows:

[0069]

[0070] Among them, Z i,j is the pixel value at position (i, j) in the output feature map, P i+m-1,j+n-1 is the pixel value of the corresponding position of the remote sensing image P, W m,n is the convolution kernel parameter, b is the bias vector, m and n are indexes, and k is the size of the convolution kernel.

[0071] S112: Perform a pooling operation on the feature map Z so that the feature map Z is downsampled. The calculation formula is as follows:

[0072] Z′ i,j =max m,n (Z i+m,j+n )

[0073] Among them, (m,n) represents the pooling window size.

[0074] S113: Input the pooled feature map Z′ into the residual module to obtain the feature map X. The calculation formula is as follows:

[0075] X=F(Z′,{W′})+Z′

[0076] Among them, F(Z′,{W′}) represents the convolution operation on the feature map Z′, and {X′} is the convolution kernel weight of the residual module.

[0077] S114: Input the feature map X into the global pooling module to obtain the feature map C1. The calculation formula is as follows:

[0078]

[0079] Among them, x i,j is the pixel value at position (i, j) in the feature map X, H X and W X are the height and width of the feature map X, respectively.

[0080] S2: Input the multi-level feature map into the multi-directional feature enhancement network to obtain a multi-scale feature map;

[0081] like Figure 3As shown, the multi-directional feature enhancement network includes a feature enhancement module and a coordinate attention module; the feature enhancement module includes: an HW dimension branch, a CW dimension branch, a CH dimension branch, a local branch and a concatenator (Concatenate).

[0082] The multi-directional feature enhancement network processes multi-level feature maps including:

[0083] S21: Multi-level feature map C k Perform convolution operation to obtain a multi-level feature map C′ with enhanced channel dimension k , the calculation formula is as follows:

[0084] C′ k =Con(C k ,ck1)

[0085] Among them, Con() represents the multi-level feature map C k Perform convolution operation, ck1 is the convolution kernel size, which is a 1-dimensional convolution;

[0086] S22: Multi-level feature map C′ with enhanced channel dimension k Input the feature enhancement module to enhance the features and obtain the feature enhancement graph F with enhanced channel dimension. k ;

[0087] The hierarchical feature map C′ enhanced by the feature enhancement module for channel dimension k Processing includes:

[0088] S221: The hierarchical feature map C′ k Split along the channel dimension to obtain the feature x k,s ,s=[1,4]; s is the index of the feature;

[0089] S222: feature x k,1 Input the HW dimension branch to obtain the hierarchical features of the HW dimension

[0090] S223: feature x k,2 Input the CW dimension branch to obtain the hierarchical features of the CW dimension

[0091] S224: feature x k,3 Input the CH dimension branch to obtain the hierarchical features of the CH dimension

[0092] S225: feature x k,4 Input the local branch to obtain the hierarchical features of local information

[0093] S226: Features Input the splicer to obtain the feature enhancement graph F with channel dimension enhancement k .

[0094] The local branch includes: k,4 Input the deep convolution module (DW conv), input the result of deep convolution into the dilated convolution pyramid to obtain the feature

[0095] The HW dimension branch includes: Fourier and inverse Fourier transform modules and a dilated convolution pyramid; the HW dimension branch transforms the hierarchical feature map x k,1 Processing includes:

[0096] S2211: x k,1 Input Fourier and inverse Fourier transform modules to obtain the hierarchical features x containing boundary information k,1 ′;

[0097] S2212: x k,1 ' Input the dilated convolutional pyramid to obtain the hierarchical features of HW dimension

[0098] The hollow space pyramid pooling ASPP solves the deficiency of convolution using only a single scale feature in the mask generation process. In ASPP, in order to obtain a sufficiently large receptive field, a larger hollow rate needs to be adopted. However, with the increase of the hollow rate, the effect of the hollow convolution will become worse and worse, and even gradually lose the modeling ability. In order to more effectively utilize the fusion information of the shallow high-resolution feature map and the deep low-resolution feature map to better complete the feature extraction, the hollow convolution pyramid ACMP of the present invention removes the pooling layer of ASPP to better adapt to the tiny targets in the remote sensing image, and adopts a hollow rate rate that is more suitable for the detection of small targets in remote sensing images, which is set to 3, 6, and 9 respectively. At the same time, the addition add operation is adopted instead of the splicing concat operation to better retain the feature information of more levels. The structure adopts the multi-scale idea to further extract the rich multi-scale feature information of the shallow high resolution, and well retains the multi-scale semantic information while obtaining a sufficiently large receptive field.

[0099] Specifically, Figure 4 As shown in the figure, the dilated convolution pyramid includes: a 1×1 convolution module, a 3×3 convolution module with a dilation rate of 3, a 3×3 convolution module with a dilation rate of 6, and a 3×3 convolution module with a dilation rate of 9; the dilated convolution pyramid has a large number of convolutions for x k,1 ′The dilated convolution operation includes: k,1' Input the 1×1 convolution module, the 3×3 convolution module with a void rate of 3, the 3×3 convolution module with a void rate of 6, and the 3×3 convolution module with a void rate of 9 respectively, add the outputs of the 1×1 convolution module, the 3×3 convolution module with a void rate of 3, the 3×3 convolution module with a void rate of 6, and the 3×3 convolution module with a void rate of 9, and perform a 1×1 convolution on the added result to obtain the output feature

[0100] The calculation formula of the dilated convolution is as follows:

[0101] z=i×s h -p h +u×r′

[0102] y=j×s w -p w +v×r′

[0103]

[0104] Among them, i and j are The position in h is the padding in the height direction, s h is the step length in the height direction, p w is the padding in the width direction, s w is the step size in the width direction, u and v are the convolution kernel positions, and k h , k w is the size of the convolution kernel, z and y are the receptive field positions in the dilated convolution pyramid, and W u,v is the convolution kernel weight, r′ is the dilation rate, is the output of the dilated convolution.

[0105] The processing process of the CW dimension branch and the CH dimension branch is the same as that of the HW dimension branch, except that the transformation axes and weights of the Fourier and inverse Fourier transform modules are different.

[0106] Specifically, the Fourier and inverse Fourier transform module formulas are:

[0107] x k,i(I,J) =W (I,J) ⊙F (I,J) [x k,i ],i∈{1,2,3}

[0108] x k ' ,i =F -1 (I,J) [x k,i(I,J) ],i∈{1,2,3}

[0109] Among them, W (I,J) 、F(I,J) denote the learnable external weights and Fourier transform of the corresponding axes, F -1 (I,J) represents inverse Fourier transform, when i=1, (I,J) represents the height H-width W axis.

[0110] S23: Feature enhancement graph F that enhances the channel dimension k and the hierarchical feature map C′ k Input the coordinate attention module to obtain the multi-scale feature map Y k .

[0111] like Figure 5 As shown in the figure, the coordinate attention module consists of an H-dimensional branch and a W-dimensional branch. The coordinate attention module encodes the H and W dimensions of the feature map respectively to generate two feature weight maps, which can reduce the loss of spatial information. The two parallel feature weight maps represent information in a specific direction, and are then encoded into two attention maps, which capture the long-distance dependency information of the input feature along a certain direction and encode the coordinates in the vertical direction.

[0112] Feature enhancement graph F of the channel dimension enhancement by the coordinate attention module k Processing includes:

[0113] S231: Multi-directional feature enhancement graph F k Input the H-dimensional branch for H-dimensional encoding to obtain the encoding features The calculation formula is as follows:

[0114]

[0115] Among them, H is the height, h is the feature enhancement map F k The index of the height;

[0116] S232: Multi-directional feature enhancement graph F′ k Input the W-dimensional branch for W-dimensional encoding to obtain the encoded features The calculation formula is as follows:

[0117]

[0118] Among them, W is the width, w is the feature enhancement map F k The index of the width;

[0119] S233: Yes and Combine to get the feature weight z k , perform convolution operation on the feature weight z to obtain the feature weight f k :

[0120] f k=δ(F1(z k ))

[0121] Among them, F1() represents a 1D convolution operation, and δ is a nonlinear activation function;

[0122] S234: f by channel dimension k Split and get the feature weight f h k and f w k ;f k The dimension is C / r×(H+W), r represents the reduced weight of the coordinate attention module, f h k The dimension is C / r×H, f w k The dimension is C / r×W;

[0123] S235: f h k Perform channel convolution F h Get the channel attention weight g h k , the calculation formula is as follows:

[0124] g h k =σ(F h (f h k ))

[0125] S236: f w k Perform channel convolution F h Get the channel attention weight g w k , the calculation formula is as follows:

[0126] g w k =σ(F h (f w k ))

[0127] Among them, σ represents the sigmoid activation function.

[0128] S237: According to the channel attention weight g h k and g w k And the hierarchical feature map C′ k Calculate the multi-scale feature map Y output by the coordinate attention mechanism k , the output value at the (i,j)th position on the cth channel is:

[0129]

[0130] S3: Multi-scale feature map Y k,c (i, j) Input the top-down feature extraction backbone network to obtain a multi-scale fusion feature map;

[0131] The top-down feature extraction backbone network includes: n layers of series-connected feature extraction layers; the top-down feature extraction backbone network processes the output of the multi-directional feature enhancement fusion network including:

[0132] S31, feature map Y n Input the first feature extraction layer, and compare the output P1 of the first feature extraction layer with the feature map Y n-1 Perform splicing and input the splicing result into the second feature extraction layer;

[0133] S32, the output P of the k-1th feature extraction layer k-1 With feature map Y n-k+1 Perform splicing and input the splicing result into the kth feature extraction layer;

[0134] S33, repeat step S32 until the output P of the last feature extraction layer is obtained n .

[0135] S4: Multi-scale fusion feature map P k Input the target detection network to perform target detection and obtain the detection result;

[0136] The object detection network includes an Oriented Region Proposal Network (Oriented RPN) and an Oriented R-CNN detection head (Oriented R-CNN Head); the multi-scale feature map is input into the Oriented RPN to obtain a detection frame, and the output of the Oriented RPN is input into the Oriented R-CNN detection head to obtain a classification prediction value, and the category with the highest classification prediction value is used as the category of the object in the detection frame; among them, R-CNN is a region-based convolutional neural network.

[0137] S5: Calculate the loss function value according to the detection results, and update the model parameters according to the loss function value. When the loss function value is minimum, the trained remote sensing target detection model is obtained.

[0138] In one embodiment, the loss function value may be calculated based on the detected detection box and the classification prediction value using a cross entropy loss function for classification and a Smooth L1 loss function for positioning.

[0139] In one embodiment, during model training, the Adam optimizer is used to update the parameters, where the exponential decay rate β1 of the first-order moment estimate is 0.9, the exponential decay rate β2 of the second-order moment estimate is 0.99, and the initial learning rate is set to 1×10 -4 , multiply it by 0.2 every 60 epochs, train for a total of 200 epochs, and set the batch size to 1.

[0140] Another aspect of the present invention provides a remote sensing target detection system based on multi-scale feature fusion, wherein the system is applied to the remote sensing target detection method based on multi-scale feature fusion, comprising:

[0141] A data acquisition module, used for acquiring remote sensing images;

[0142] A bottom-up feature extraction backbone module is used to extract multi-level feature maps of remote sensing images;

[0143] The multi-directional feature enhancement module is used to perform feature enhancement, bidirectional spatial encoding and feature weight optimization on the multi-level feature maps of remote sensing images to obtain multi-scale feature maps;

[0144] The top-down feature extraction backbone module is used to fuse multi-scale feature maps to obtain multi-scale fused feature maps;

[0145] The target detection module is used to classify the multi-scale fusion feature map to obtain the remote sensing image target detection category.

[0146] Another aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the remote sensing target detection method based on multi-scale feature fusion.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0148] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing target detection method based on multi-scale feature fusion, characterized in that: include: Acquire remote sensing images, input the remote sensing images into the trained remote sensing target detection model, and obtain detection results; The remote sensing target detection model includes: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-directional feature enhancement network, and a target detection network; the training process of the remote sensing target detection model includes: S1: Obtain remote sensing images and input them into a bottom-up feature extraction backbone network to obtain multi-level feature maps; S2: Input the multi-level feature map into the multi-directional feature enhancement network to obtain a multi-scale feature map; S3: Input the multi-scale feature map into the top-down feature extraction backbone network to obtain a multi-scale fusion feature map; S4: Input the multi-scale fusion feature map into the target detection network to perform target detection and obtain the detection result; S5: Calculate the loss function value according to the detection results, and update the model parameters according to the loss function value. When the loss function value is minimum, the trained remote sensing target detection model is obtained.

2. The remote sensing target detection method based on multi-scale feature fusion according to claim 1 is characterized in that: The bottom-up feature extraction backbone network includes: n layers of series-connected feature extraction layers; the bottom-up feature extraction backbone network processes the remote sensing image I including: S11, inputting the remote sensing image I into the first feature extraction layer, and inputting the output C1 of the first feature extraction layer into the second feature extraction layer; S12, the output C of the k-1th feature extraction layer k-1 Input the kth feature extraction layer; where k is the index of the feature extraction layer of the bottom-up feature extraction backbone network; S13, repeat step S12 until the output C of the last feature extraction layer is obtained n .

3. The remote sensing target detection method based on multi-scale feature fusion according to claim 2 is characterized in that: The feature extraction layer includes: convolution module, residual module and global pooling module; the first feature extraction layer processes the remote sensing image I including: S111: Input the remote sensing image I into the convolution module to obtain a feature map Z; S112: Perform a pooling operation on the feature map Z to obtain the feature map Z ′ ; S113: Pooled feature map Z ′ Input the residual module to obtain the feature map X; S114: Input the feature map X into the global pooling module to obtain the feature map C1.

4. The remote sensing target detection method based on multi-scale feature fusion according to claim 1, characterized in that: The multi-directional feature enhancement network includes: a feature enhancement module and a coordinate attention module; the multi-directional feature enhancement network processes the hierarchical feature map including: S21: for the level feature map C k Perform convolution to obtain the hierarchical feature map C′ with enhanced channel dimension k ; S22: Hierarchical feature map C′ with enhanced channel dimension k Input the feature enhancement module to obtain the feature enhancement graph F with enhanced channel dimension k ; S23: Feature enhancement graph F that enhances the channel dimension k and the hierarchical feature map C′ k Input the coordinate attention module to obtain the multi-scale feature map Y k .

5. The remote sensing target detection method based on multi-scale feature fusion according to claim 4 is characterized in that: The feature enhancement module includes: HW dimension branch, CW dimension branch, CH dimension branch, local branch and splicer; the feature enhancement module enhances the hierarchical feature map C′ of the channel dimension k Processing includes: S221: The hierarchical feature map C′ k Split along the channel dimension to obtain the split feature x k,s ,s=[1,4]; s is the index of the feature after splitting; S222: feature x k,1 Input the HW dimension branch to obtain the hierarchical features of the HW dimension S223: feature x k,2 Input the CW dimension branch to obtain the hierarchical features of the CW dimension S224: feature x k,3 Input the CH dimension branch to obtain the hierarchical features of the CH dimension S225: feature x k,4 Input the local branch to obtain the hierarchical features of local information S226: Features Input the splicer to obtain the feature enhancement graph F with channel dimension enhancement k ; Where H is the height, W is the width, and C is the channel.

6. The remote sensing target detection method based on multi-scale feature fusion according to claim 5 is characterized in that: The HW dimension branch includes: Fourier and inverse Fourier transform modules and a dilated convolution pyramid; the HW dimension branch transforms the multi-level feature map x k,1 Processing includes: S2211: x k,1 Input Fourier and inverse Fourier transform modules to obtain the hierarchical features x containing boundary information k,1 ′ ; S2212: x k,1 ′ Input the dilated convolutional pyramid to obtain the hierarchical features of the HW dimension 7. The remote sensing target detection method based on multi-scale feature fusion according to claim 6 is characterized in that: The dilated convolution pyramid includes: 1×1 convolution module, 3×3 convolution module with dilation rate of 3, 3×3 convolution module with dilation rate of 6, and 3×3 convolution module with dilation rate of 9; the dilated convolution pyramid has a large number of convolution modules, and has a large number of convolution modules with dilation rate of 9. k,1 ′ The dilated convolution operation includes: k,1 ′ The 1×1 convolution module, the 3×3 convolution module with a void ratio of 3, the 3×3 convolution module with a void ratio of 6, and the 3×3 convolution module with a void ratio of 9 are input respectively, and the outputs of the 1×1 convolution module, the 3×3 convolution module with a void ratio of 3, the 3×3 convolution module with a void ratio of 6, and the 3×3 convolution module with a void ratio of 9 are added, and the added result is convolved by 1×1 to obtain the output feature.

8. The remote sensing target detection method based on multi-scale feature fusion according to claim 4 is characterized in that: The coordinate attention module includes an H-dimension branch and a W-dimension branch; Feature enhancement graph F of the channel dimension enhancement by the coordinate attention module k Processing includes: S231: Enhance the feature map F k Input the H-dimensional branch for H-dimensional encoding to obtain the encoding features Among them, H is the height, h is the feature enhancement map F k The index of the height; S232: Enhance the feature map F k Input the W-dimensional branch for W-dimensional encoding to obtain the encoded features Among them, W is the width, w is the feature enhancement map F k The index of the width; S233: Yes and Combine to get the feature weight z k , perform convolution operation on the feature weight z to obtain the feature weight f k ; S234: f by channel dimension k Split and get the feature weight f h k and f w k ; S235: f h k Perform channel convolution F h Get the channel attention weight g h k ; S236: f w k Perform channel convolution F h Get the channel attention weight g w k ; S237: According to the channel attention weight g h k and g w k And the hierarchical feature map C′ k Compute multi-scale feature maps.

9. A system based on the remote sensing target detection method based on multi-scale feature fusion according to claims 1 to 8, characterized in that: include: A data acquisition module, used for acquiring remote sensing images; A bottom-up feature extraction backbone module is used to extract multi-level feature maps of remote sensing images; The multi-directional feature enhancement module is used to process the multi-level feature maps of remote sensing images to obtain multi-scale feature maps; The top-down feature extraction backbone module is used to process the multi-scale feature map to obtain the multi-scale fusion feature map; The target detection module is used to classify the multi-scale fusion feature map to obtain the remote sensing image target detection category.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the steps of the remote sensing target detection method based on multi-scale feature fusion as described in any one of claims 1 to 8 are implemented.

Images