Remote sensing target detection method and system based on multi-scale feature fusion and medium

Abstract

The application belongs to the technical field of computer vision, and relates to a remote sensing target detection method and system based on multi-scale feature fusion and a medium, which comprises the following steps: acquiring a remote sensing image and inputting the remote sensing image into a bottom-up feature extraction backbone network to obtain a multi-level feature map C k ; inputting C k into a multi-direction feature enhancement network to obtain a multi-scale feature map Y k ; inputting Y k into a top-down feature extraction backbone network to obtain a multi-scale fusion feature map P k ; inputting P k into a target detection network for target detection to obtain a detection result; the multi-direction feature enhancement network of the application performs omnidirectional frequency domain information enhancement on the feature map of each level, performs feature enhancement on the frequency domain enhanced features in different scales, and performs bidirectional spatial coding and feature weight optimization on the H and W dimensions by using a coordinate attention mechanism, thereby comprehensively paying attention to the multi-information of the frequency domain and the spatial domain of the low dimension and the high latitude of the feature layer, effectively improving the detection capability for small targets and enhancing the capability of coping with complex backgrounds.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and relates to a remote sensing target detection method and system based on multi-scale feature fusion and a medium. BACKGROUND

[0002] Remote Sensing Object Detection aims to accurately identify and locate various target objects from remote sensing images, and plays a key role in many fields such as geographic mapping, environmental monitoring, military reconnaissance, and urban planning. Remote sensing images can provide large-area, macroscopic ground information, containing a variety of targets such as buildings, roads, vegetation, water bodies, and various vehicles. However, in practical applications, due to the characteristics of remote sensing images such as large target scale variation, different resolutions, and complex backgrounds, how to effectively extract and fuse multi-scale features becomes the key to improving detection performance. This has prompted related research to become a focus in the field.

[0003] Currently, using deep learning methods to extract effective features from remote sensing images for target detection has become the mainstream, but due to the complex characteristics of remote sensing images, such as large target scale variation, uncertain direction, and complex background, existing detection methods often have limitations. For example, the Feature Pyramid Network (FPN) and its improved models have low utilization rate when processing shallow features, and small target features are easily lost during the transfer of shallow features to deep layers. At the same time, traditional backbone networks lack global information dependency, resulting in a lack of feature expression ability in complex backgrounds. These factors combined result in the irrationality of existing methods in feature extraction structure, which cannot fully obtain the feature information of targets in remote sensing images.

[0004] In summary, many existing studies focus on improving spatial domain models, ignoring the importance of frequency domain information, and shallow features are easily lost during the transfer to deep layers, while lacking global information dependency, resulting in a lack of feature expression ability in complex backgrounds. SUMMARY

[0005] To solve the above technical problems, in a first aspect, the present application adopts a remote sensing target detection method based on multi-scale feature fusion, which includes: obtaining a remote sensing image, inputting the remote sensing image into a trained remote sensing target detection model, and obtaining a detection result; the remote sensing target detection model includes: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-directional feature enhancement network, and a target detection network; the training process of the remote sensing target detection model includes:

[0006] S1: acquire a remote sensing image, input the remote sensing image into a bottom-up feature extraction backbone network, and obtain a multi-level feature map;

[0007] S2: input the multi-level feature map into a multi-direction feature enhancement network, and obtain a multi-scale feature map;

[0008] S3: input the multi-scale feature map into a top-down feature extraction backbone network, and obtain a multi-scale fusion feature map;

[0009] S4: input the multi-scale fusion feature map into a target detection network for target detection, and obtain a detection result;

[0010] S5: calculate a loss function value according to the detection result, update model parameters according to the loss function value, and obtain a trained remote sensing target detection model when the loss function value is minimum.

[0011] The bottom-up feature extraction backbone network comprises: n layers of feature extraction layers connected in series; the bottom-up feature extraction backbone network comprises:

[0012] S11, input the remote sensing image I into the first layer feature extraction layer, and input the output C1 of the first layer feature extraction layer into the second layer feature extraction layer;

[0013] S12, input the output C k-1 of the k-1 layer feature extraction layer into the k layer feature extraction layer; wherein k is the index of the feature extraction layer of the bottom-up feature extraction backbone network;

[0014] S13, repeat step S12 until the output C n of the last layer feature extraction layer is obtained.

[0015] The feature extraction layer comprises: a convolution module, a residual module and a global pooling module; the first layer feature extraction layer comprises:

[0016] S111: input the remote sensing image I into the convolution module to obtain a feature map Z;

[0017] S112: perform a pooling operation on the feature map Z to obtain a feature map Z';

[0018] S113: input the pooled feature map Z' into the residual module to obtain a feature map X;

[0019] S114: input the feature map X into the global pooling module to obtain a feature map C1.

[0020] The multi-direction feature enhancement network comprises: a feature enhancement module and a coordinate attention module; the multi-direction feature enhancement network comprises:

[0021] S21: input the hierarchical feature map C k to a convolution operation to obtain a channel-dimensionally reinforced hierarchical feature map C' k ;

[0022] S22: input the channel-dimensionally reinforced hierarchical feature map C' k to a feature enhancement module to obtain a channel-dimensionally reinforced feature enhancement map F k ;

[0023] S23: input the channel-dimensionally reinforced feature enhancement map F k and the hierarchical feature map C' k to a coordinate attention module to obtain a multi-scale feature map Y k .

[0024] The feature enhancement module comprises an H-W dimension branch, a C-W dimension branch, a C-H dimension branch, a local branch, and a splicer; the feature enhancement module processes the channel-dimensionally reinforced hierarchical feature map C' k in a manner comprising:

[0025] S221: split the hierarchical feature map C' k along a channel dimension to obtain split features x k,s , s = [1, 4]; s is an index of the split features;

[0026] S222: input the features x k,1 to the H-W dimension branch to obtain H-W dimension hierarchical features

[0027] S223: input the features x k,2 to the C-W dimension branch to obtain C-W dimension hierarchical features

[0028] S224: input the features x k,3 to the C-H dimension branch to obtain C-H dimension hierarchical features

[0029] S225: input the features x k,4 to the local branch to obtain local information hierarchical features

[0030] S226: input the features to the splicer to obtain the channel-dimensionally reinforced feature enhancement map F k ; wherein H is a height, W is a width, and C is a channel number.

[0031] The H-W dimension branch comprises a Fourier and inverse Fourier transform module and a dilated convolution pyramid; the H-W dimension branch processes the multi-level feature map x k,1The processing includes:

[0032] S2211: inputting the x k,1 into a Fourier transform and inverse Fourier transform module to obtain a hierarchical feature x k,1 containing boundary information

[0033] S2212: inputting the x k,1 into a dilated convolution pyramid to obtain a hierarchical feature of H-W dimensions

[0034] The dilated convolution pyramid includes a 1x1 convolution module, a 3x3 convolution module with a dilated rate of 3, a 3x3 convolution module with a dilated rate of 6, and a 3x3 convolution module with a dilated rate of 9; the dilated convolution pyramid performs a dilated convolution operation on the x k,1 to obtain an output feature k,1 The dilated convolution operation includes inputting the x k into the 1x1 convolution module, the 3x3 convolution module with the dilated rate of 3, the 3x3 convolution module with the dilated rate of 6, and the 3x3 convolution module with the dilated rate of 9 respectively, adding outputs of the 1x1 convolution module, the 3x3 convolution module with the dilated rate of 3, the 3x3 convolution module with the dilated rate of 6, and the 3x3 convolution module with the dilated rate of 9, and performing 1x1 convolution on the added result to obtain the output feature

[0035] The coordinate attention module includes an H-dimension branch and a W-dimension branch; the coordinate attention module strengthens the feature enhancement map F k

[0036] S231: inputting the feature enhancement map F k into the H-dimension branch to perform H-dimension encoding to obtain an encoded feature wherein H is a height, and h is an index of the height of the feature enhancement map F k

[0037] S232: inputting the feature enhancement map F k into the W-dimension branch to perform W-dimension encoding to obtain an encoded feature wherein W is a width, and w is an index of the width of the feature enhancement map F k

[0038] S233: combining the and to obtain a feature weight z k

[0039] S234: splitting the f k in the channel dimension to obtain a feature weight f h ​​​​k and f w k ;

[0040] S235: performing channel convolution F h k to obtain channel attention weight g h h k ;

[0041] S236: performing channel convolution F w k to obtain channel attention weight g h w k ;

[0042] S237: calculating multi-scale feature map according to channel attention weight g h k and g w k and hierarchical feature map C' k .

[0043] In a second aspect, the present application adopts a system of a remote sensing target detection method based on the above multi-scale feature fusion, comprising:

[0044] a data acquisition module for acquiring a remote sensing image;

[0045] a bottom-up feature extraction backbone module for extracting a multi-level feature map of the remote sensing image;

[0046] a multi-directional feature enhancement module for processing the multi-level feature map of the remote sensing image to obtain a multi-scale feature map;

[0047] a top-down feature extraction backbone module for processing the multi-scale feature map to obtain a multi-scale fusion feature map;

[0048] a target detection module for classifying the multi-scale fusion feature map to obtain a remote sensing image target detection category.

[0049] In a third aspect, the present application adopts a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of the above remote sensing target detection method based on multi-scale feature fusion.

[0050] Advantages:

[0051] ​​1、The application carries out Fourier transform on three axes of input features after feature extraction, and allocates external weight in frequency domain, and then carries out inverse Fourier transform to change the features back to spatial domain; by paying attention to frequency domain information, the model's recognition and detection of edges are enhanced, and then the H and W dimensions of the frequency domain enhanced feature map are bidirectionally spatially coded and feature weight optimized by using the coordinate attention mechanism to generate a multi-scale feature map, comprehensively paying attention to the multiple information of the feature layer frequency domain and spatial domain, better coping with the challenges brought by different target scales and various direction changes in remote sensing images, effectively improving the detection ability of small targets and enhancing the ability to cope with complex background edges that are difficult to detect; 2、The ACMP of the application further extracts rich multi-scale feature information in the frequency domain enhanced features, effectively improving the detection ability of small targets and enhancing the ability to cope with complex backgrounds while retaining the multi-scale semantic information; 3、The ACMP removes the pooling layer of the ASPP to better adapt to the small targets in remote sensing images, and sets the hole rate of the dilated convolution to 3, 6 and 9 respectively to better suit the small target detection of remote sensing images, and uses addition operation instead of concatenation operation to fuse the output of dilated convolution to better retain more hierarchical feature information, effectively improving the detection ability of small targets and enhancing the ability to cope with complex backgrounds. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The flowchart of the remote sensing target detection method based on multi-scale feature fusion in the embodiment of the application;

[0053] Figure 2 The structural diagram of the remote sensing target detection model in the embodiment of the application;

[0054] Figure 3 The structural diagram of the multi-directional feature enhancement network in the embodiment of the application;

[0055] Figure 4 The structural diagram of the dilated convolution pyramid in the embodiment of the application;

[0056] Figure 5 The structural diagram of the coordinate attention module in the embodiment of the application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0058] As Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 indicated, one aspect of the present application provides a remote sensing target detection method based on multi-scale feature fusion, comprising: inputting a remote sensing image under a complex scene into a trained remote sensing target detection model, using multi-scale features extracted by the model and fusion, and finally realizing remote sensing image target detection and classification; the remote sensing target detection model comprises: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-direction feature enhancement network and a target detection network;

[0059] The training process of the remote sensing target detection model comprises:

[0060] S1: obtaining a remote sensing image dataset, inputting a remote sensing image in the remote sensing image dataset into a bottom-up feature extraction backbone network to obtain a multi-level feature map;

[0061] In one embodiment, the low-resolution remote sensing image can be obtained from an existing public dataset, or can be obtained by shooting with a camera, a video camera, a mobile phone or the like, and the present application does not make specific limitations thereon.

[0062] In one embodiment, when training the remote sensing target detection model, 2420 images are obtained from the UCAS-AOD dataset, the model sample set is divided into a training set and a test set, the training set uses 2000 images in the UCAS-AOD dataset, and the test set uses 420 images in the UCAS-AOD dataset.

[0063] As Figure 2 indicated, the bottom-up feature extraction backbone network and the top-down feature extraction backbone network each comprise: n layers of feature extraction layers connected in series; processing of the remote sensing image I by the bottom-up feature extraction backbone network comprises:

[0064] S11, inputting the remote sensing image I into the first layer of feature extraction layers, and inputting the output C1 of the first layer of feature extraction layers into the second layer of feature extraction layers;

[0065] S12, inputting the output C k-1 of the k-1 layer of feature extraction layers into the k layer of feature extraction layers; wherein k is the index of the feature extraction layer of the bottom-up feature extraction backbone network;

[0066] S13, repeating step S12 until the output C n of the last layer of feature extraction layers is obtained.

[0067] Each layer of the feature extraction layer comprises a convolution module, a residual module and a global pooling module; processing of the remote sensing image I by the first layer of the feature extraction layer comprises:

[0068] S111: inputting the remote sensing image I into the convolution module to obtain a feature map Z, and the calculation formula is as follows:

[0069]

[0070] wherein, Z i,j is a pixel value of a position (i, j) in an output feature map, P i+m-1,j+n-1 is a pixel value of a corresponding position of the remote sensing image P, W m,n is a convolution kernel parameter, b is a bias vector, m and n are indexes, and k is a size of the convolution kernel.

[0071] S112: performing a pooling operation on the feature map Z to make the feature map Z perform a down-sampling operation, and the calculation formula is as follows:

[0072] Z′ i,j =max m,n (Z i+m,j+n )

[0073] wherein, (m, n) represents a size of a pooling window.

[0074] S113: inputting the pooled feature map Z' into the residual module to obtain a feature map X, and the calculation formula is as follows:

[0075] X=F(Z′,{W′})+Z′

[0076] wherein, F(Z', {W'}) represents a convolution operation on the feature map Z', and {X'} is a convolution kernel weight of the residual module.

[0077] S114: inputting the feature map X into the global pooling module to obtain a feature map C1, and the calculation formula is as follows:

[0078]

[0079] wherein, x i,j is a pixel value of a position (i, j) in the feature map X, H X and W X are a height and a width of the feature map X respectively.

[0080] S2: inputting the multi-level feature map into the multi-directional feature enhancement network to obtain a multi-scale feature map;

[0081] For example, Figure 3As shown, the multi-directional feature enhancement network comprises a feature enhancement module and a coordinate attention module; the feature enhancement module comprises an H-W dimensional branch, a C-W dimensional branch, a C-H dimensional branch, a local branch, and a concatenator (Concatenate).

[0082] The multi-directional feature enhancement network processes the multi-level feature map, which comprises:

[0083] S21: performing convolution operation on the multi-level feature map C k to obtain a channel-dimension-enhanced multi-level feature map C' k , and the calculation formula is as follows:

[0084] C' k = Con(C k ,ck1)

[0085] wherein Con() represents performing convolution operation on the multi-level feature map C k , and ck1 is a convolution kernel size, which is a 1-dimensional convolution;

[0086] S22: inputting the channel-dimension-enhanced multi-level feature map C' k into the feature enhancement module to perform feature enhancement, to obtain a channel-dimension-enhanced feature enhancement map F k ;

[0087] The feature enhancement module processes the channel-dimension-enhanced multi-level feature map C' k , which comprises:

[0088] S221: splitting the multi-level feature map C' k along the channel dimension respectively to obtain features x k,s , s = [1, 4]; s is an index of the features;

[0089] S222: inputting the features x k,1 into the H-W dimensional branch to obtain H-W dimensional multi-level features

[0090] S223: inputting the features x k,2 into the C-W dimensional branch to obtain C-W dimensional multi-level features

[0091] S224: inputting the features x k,3 into the C-H dimensional branch to obtain C-H dimensional multi-level features

[0092] S225: inputting the features x k,4 into the local branch to obtain local information multi-level features

[0093] S226: obtaining the feature inputting the splicer to obtain the channel-dimension enhanced feature k .

[0094] The local branch includes: inputting x k,4 inputting a deep convolution module (DW conv), and inputting the result of the deep convolution into a dilated convolution pyramid to obtain the feature

[0095] The H-W dimension branch includes: a Fourier and inverse Fourier transform module and a dilated convolution pyramid; the H-W dimension branch processes the hierarchical feature map x k,1 includes:

[0096] S2211: inputting x k,1 into the Fourier and inverse Fourier transform module to obtain the hierarchical feature x k,1 ′ containing boundary information;

[0097] S2212: inputting x k,1 ′ into the dilated convolution pyramid to obtain the hierarchical feature of the H-W dimension

[0098] The dilated spatial pyramid pooling (ASPP) solves the problem that only single scale features are used in the mask generation process. In the ASPP, a larger dilated rate needs to be used to obtain a large enough receptive field. However, with the increase of the dilated rate, the effect of the dilated convolution becomes worse and worse, and even gradually loses the modeling ability. In order to more effectively utilize the fusion information of the shallow high-resolution feature map and the deep low-resolution feature map to better complete feature extraction, the dilated convolution pyramid (ACMP) of the present application removes the pooling layer of the ASPP to better adapt to the small target in the remote sensing image, and uses a dilated rate rate more suitable for small target detection of the remote sensing image, which is set to 3, 6 and 9 respectively. At the same time, an addition operation add is used instead of a splicing operation concat to better preserve the feature information of more levels. The structure further extracts the rich multi-scale feature information of the shallow high-resolution multi-scale by using the multi-scale idea, and obtains a large enough receptive field while well preserving the multi-scale semantic information.

[0099] Specifically, as shown in Figure 4 the dilated convolution pyramid includes: a 1x1 convolution module, a 3x3 dilated rate of 3 convolution module, a 3x3 dilated rate of 6 convolution module, and a 3x3 dilated rate of 9 convolution module; and the dilated convolution pyramid performs a dilated convolution operation on x k,1 ′ includes: inputting x k,1Input 1×1 convolutional modules, 3×3 convolutional modules with a dilation rate of 3, 3×3 convolutional modules with a dilation rate of 6, and 3×3 convolutional modules with a dilation rate of 9 respectively. Add the outputs of these modules together, and then perform a 1×1 convolution on the summed result to obtain the output features.

[0100] The formula for calculating dilated convolution is as follows:

[0101] z = i × s h -p h +u×r′

[0102] y = j × s w -p w +v×r′

[0103]

[0104] Where i and j are The position of p in the middle h It is fill in the height direction, s h It is the step size in the height direction, p w It is the padding in the width direction, s w The stride is the width direction, u and v are the kernel positions, and k is the width direction. h k w Let W be the kernel size, z and y be the receptive field locations in the dilated convolution within the dilated convolution pyramid, and W be the kernel size. u,v These are the convolution kernel weights, and r′ is the dilation rate. This is the output of dilated convolution.

[0105] The processing of CW and CH dimension branches is the same as that of HW dimension branches, except that the transformation axes and weights of the Fourier and inverse Fourier transform modules are different.

[0106] Specifically, the formulas for the Fourier and inverse Fourier transform modules are as follows:

[0107] x k,i(I,J) =W (I,J) ⊙F (I,J) [x k,i ], i∈{1,2,3}

[0108] x k ′ ,i =F -1 (I,J) [x k,i(I,J) ], i∈{1,2,3}

[0109] Among them, W (I,J) F(I,J) respectively represent the learnable external weights and the Fourier transform of the corresponding axis, F -1 (I,J) denotes the inverse Fourier transform, and (I, J) represents the height H-width W axis when i = 1.

[0110] S23: The channel dimension enhanced feature enhancement map F k and the hierarchical feature map C' k input coordinate attention module to obtain a multi-scale feature map Y k .

[0111] As shown in Figure 5 , the coordinate attention module is composed of an H dimension branch and a W dimension branch; the coordinate attention module encodes the H and W dimensions of the feature map respectively to generate two feature weight maps, which can reduce the loss of spatial information. The two parallel feature weight maps represent information in a specific direction, and are respectively encoded into two attention maps to capture the long-distance dependence information of the input feature in a certain direction and the coordinates in the vertical direction.

[0112] The coordinate attention module processes the channel dimension enhanced feature enhancement map F k , which includes:

[0113] S231: The multi-directional feature enhancement map F k is input into the H dimension branch for H dimension encoding to obtain an encoded feature The calculation formula is as follows:

[0114]

[0115] wherein H is the height, and h is the index of the height of the feature enhancement map F k .

[0116] S232: The multi-directional feature enhancement map F' is input into the W dimension branch for W dimension encoding to obtain an encoded feature k . The calculation formula is as follows:

[0117]

[0118] wherein W is the width, and w is the index of the width of the feature enhancement map F k .

[0119] S233: The and are combined to obtain a feature weight z k , and the feature weight z is subjected to a convolution operation to obtain a feature weight f k .

[0120] f k=δ(F1(z) k ))

[0121] Where F1() represents a 1D convolution operation and δ is a non-linear activation function;

[0122] S234: F by channel dimension k The feature weights f are obtained by splitting the feature into its components. h k and f w k ;f k The dimension is C / r×(H+W), where r represents the reduced weight of the coordinate attention module, and f h k The dimension is C / r×H, f w k The dimension is C / r×W;

[0123] S235: For f h k Perform channel convolution F h Obtain the channel attention weights g h k The calculation formula is as follows:

[0124] g h k =σ(F h (f h k ))

[0125] S236: For f w k Perform channel convolution F h Obtain the channel attention weights g w k The calculation formula is as follows:

[0126] g w k =σ(F h (f w k ))

[0127] Where σ represents the sigmoid activation function.

[0128] S237: Based on the channel attention weight g h k and g w k and hierarchical feature map C′ k The multi-scale feature map Y output by the computation coordinate attention mechanism k Its output value at position (i,j) on channel c is:

[0129]

[0130] S3: input the multi-scale feature map Y k,c from top to bottom, to obtain a multi-scale fusion feature map P;

[0131] The top-down feature extraction backbone network comprises: n layers of feature extraction layers connected in series; processing the output of the multi-directional feature enhancement fusion network by the top-down feature extraction backbone network comprises:

[0132] S31, input the feature map Y n to the first layer of feature extraction layer, splice the output P1 of the first layer of feature extraction layer with the feature map Y n-1 , and input the spliced result to the second layer of feature extraction layer;

[0133] S32, input the output P k-1 of the k-1 layer of feature extraction layer to the k layer of feature extraction layer, and splice the output P n-k+1 of the k layer of feature extraction layer with the feature map Y

[0134] S33, repeat step S32 until the output P n of the last layer of feature extraction layer is obtained.

[0135] S4: input the multi-scale fusion feature map P k to the target detection network for target detection, to obtain a detection result;

[0136] The target detection network comprises an oriented region proposal network (Oriented RPN) and an oriented R-CNN detection head (Oriented R-CNN Head); input the multi-scale feature map to the Oriented RPN to obtain a detection frame, input the output of the Oriented RPN to the oriented R-CNN detection head to obtain a classification prediction value, and take the class with the highest classification prediction value as the class of the object in the detection frame; wherein, R-CNN is a region-based convolutional neural network.

[0137] S5: calculate a loss function value according to the detection result, update the model parameters according to the loss function value, and obtain a trained remote sensing target detection model when the loss function value is the smallest.

[0138] In one embodiment, the loss function value can be calculated by using a cross-entropy loss function for classification and a Smooth L1 loss function for positioning according to the detected detection frame and the classification prediction value.

[0139] In one embodiment, during model training, the parameters are updated using an Adam optimizer, wherein the exponential decay rate of the first moment estimate is 0.9, the exponential decay rate of the second moment estimate is 0.99, the initial learning rate is set to 1x10 -4 and it is multiplied by 0.2 every 60 epochs, a total of 200 epochs are trained, and the batch size is set to 1.

[0140] Another aspect of the present application provides a remote sensing target detection system based on multi-scale feature fusion, which is applied to the remote sensing target detection method based on multi-scale feature fusion, and includes:

[0141] A data acquisition module is configured to acquire a remote sensing image.

[0142] A bottom-up feature extraction backbone module is configured to extract a multi-level feature map of the remote sensing image.

[0143] A multi-directional feature enhancement module is configured to perform feature enhancement, bidirectional spatial encoding and feature weight optimization on the multi-level feature map of the remote sensing image to obtain a multi-scale feature map.

[0144] A top-down feature extraction backbone module is configured to fuse the multi-scale feature map to obtain a multi-scale fusion feature map.

[0145] A target detection module is configured to classify the multi-scale fusion feature map to obtain a remote sensing image target detection category.

[0146] Still another aspect of the present application provides a computer readable storage medium storing a computer program, wherein the program is executed by a processor to implement the remote sensing target detection method based on multi-scale feature fusion.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0148] The above-mentioned embodiments further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above-mentioned embodiments are only preferred embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A remote sensing target detection method based on multi-scale feature fusion, characterized in that, include: Acquire remote sensing images, input the remote sensing images into a trained remote sensing target detection model, and obtain the detection results; The remote sensing target detection model includes: a bottom-up feature extraction backbone network, a top-down feature extraction backbone network, a multi-directional feature enhancement network, and a target detection network; the training process of the remote sensing target detection model includes: S1: Acquire remote sensing images and input them into a bottom-up feature extraction backbone network to obtain multi-level feature maps; S2: Input the multi-level feature maps into the multi-directional feature enhancement network to obtain multi-scale feature maps; S3: Input the multi-scale feature map into the top-down feature extraction backbone network to obtain the multi-scale fused feature map; S4: Input the multi-scale fused feature map into the target detection network to perform target detection and obtain the detection results; S5: Calculate the loss function value based on the detection results, update the model parameters based on the loss function value, and obtain the trained remote sensing target detection model when the loss function value is minimized; The multi-directional feature enhancement network includes a feature enhancement module and a coordinate attention module; the multi-directional feature enhancement network processes hierarchical feature maps in the following ways: S21: For the hierarchical feature map C k Perform convolution to obtain a channel-dimensional enhanced hierarchical feature map C′. k ; S22: Hierarchical feature map C′ enhanced with channel dimension k The input feature enhancement module produces a channel-dimensional enhanced feature map F. k ; S23: Feature enhancement map F that enhances the channel dimension k and hierarchical feature map C′ k Input coordinate attention module to obtain multi-scale feature map Y k ; The feature enhancement module includes: HW dimension branch, CW dimension branch, CH dimension branch, local branch, and stitcher; the feature enhancement module enhances the hierarchical feature map C′ of the channel dimension. k The processing includes: S221: Transfer the hierarchical feature map C′ k The feature x is obtained by splitting along the channel dimension. k,s ,s = [1,4]; s is the index of the split feature; S222: Feature x k,1 Input the HW dimension branch to obtain the hierarchical features of the HW dimension. S223: [The following appears to be a separate, unrelated sentence: "Imagine feature x..."] k,2 Input the CW dimension branch to obtain the hierarchical features of the CW dimension. S224: Feature x k,3 Input the CH dimension branch to obtain the hierarchical features of the CH dimension. S225: Feature x k,4 Input local branches to obtain hierarchical features of local information. S226: Features Input the splicer to obtain the channel-dimensional enhanced feature map F. k Where H is the height, W is the width, and C is the channel; The HW dimensional branch includes: Fourier and inverse Fourier transform modules and a dilated convolutional pyramid; the HW dimensional branch combines multi-level feature maps x k,1 The processing includes: S2211: x k,1 Input the Fourier and inverse Fourier transform modules to obtain the hierarchical features x containing boundary information. k,1 ′; S2212: x k,1 Input a dilated convolutional pyramid to obtain hierarchical features in the HW dimension. The processing of CW dimension branches and CH dimension branches is the same as that of HW dimension branches, except that the transformation axes and weights of the Fourier and inverse Fourier transform modules are different. The dilated convolution pyramid includes: a 1×1 convolutional module, a 3×3 convolutional module with a dilation rate of 3, a 3×3 convolutional module with a dilation rate of 6, and a 3×3 convolutional module with a dilation rate of 9; the dilated convolution pyramid is used for x... k,1 Performing dilated convolution operations includes: converting x... k,1 Input 1×1 convolutional modules, 3×3 convolutional modules with a dilation rate of 3, 3×3 convolutional modules with a dilation rate of 6, and 3×3 convolutional modules with a dilation rate of 9 respectively. Add the outputs of these modules together, and then perform a 1×1 convolution on the summed result to obtain the output features. The coordinate attention module includes H-dimensional branches and W-dimensional branches; the coordinate attention module enhances the feature enhancement map F in the channel dimension. k The processing includes: S231: Enhance the feature map F k Input the H-dimensional branch for H-dimensional encoding to obtain the encoded features. Where H is the height and h is the feature enhancement map F. k The height index; S232: Enhance the feature map F k Inputting the W-dimensional branch and performing W-dimensional encoding yields the encoded features. Where W is the width and w is the feature enhancement map F k The width index; S233: Yes and The feature weights z are obtained by combining them. k Perform a convolution operation on the feature weights z to obtain the feature weights f. k ; S234: F by channel dimension k The feature weights f are obtained by splitting the feature into its components. h k and f w k ; S235: For f h k Perform channel convolution F h Obtain the channel attention weights g h k ; S236: For f w k Perform channel convolution F h Obtain the channel attention weights g w k ; S237: Based on the channel attention weight g h k and g w k and hierarchical feature map C′ k Calculate multi-scale feature maps; The value of the multi-scale feature map at position (i,j) on channel c.

2. The remote sensing target detection method based on multi-scale feature fusion according to claim 1, characterized in that, The bottom-up feature extraction backbone network consists of n cascaded feature extraction layers; the bottom-up feature extraction backbone network processes the remote sensing image I through the following steps: S11. Input the remote sensing image I into the first feature extraction layer, and input the output C1 of the first feature extraction layer into the second feature extraction layer; S12, convert the output C of the (k-1)th feature extraction layer. k-1 Input the k-th feature extraction layer; where k is the index of the feature extraction layer in the bottom-up feature extraction backbone network; S13. Repeat step S12 until the output C of the last feature extraction layer is obtained. n .

3. The remote sensing target detection method based on multi-scale feature fusion according to claim 2, characterized in that, The feature extraction layer includes: a convolution module, a residual module, and a global pooling module; the first feature extraction layer processes the remote sensing image I by: S111: Input the remote sensing image I into the convolution module to obtain the feature map Z; S112: Perform pooling operation on feature map Z to obtain feature map Z′; S113: Pool the feature map Z ′ Input the residual module to obtain the feature map X; S114: Input the feature map X into the global pooling module to obtain the feature map C1.

4. A system based on the remote sensing target detection method based on multi-scale feature fusion according to any one of claims 1 to 3, characterized in that, include: The data acquisition module is used to acquire remote sensing images; The bottom-up feature extraction backbone module is used to extract multi-level feature maps from remote sensing images; The multi-directional feature enhancement module is used to process multi-level feature maps of remote sensing images to obtain multi-scale feature maps. The top-down feature extraction backbone module is used to process multi-scale feature maps to obtain multi-scale fused feature maps. The target detection module is used to classify multi-scale fused feature maps to obtain target detection categories in remote sensing images.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the remote sensing target detection method based on multi-scale feature fusion as described in any one of claims 1 to 3.

Images