A remote sensing image target detection method and device based on two-stage frequency domain enhancement

Abstract

This invention discloses a remote sensing image target detection method based on two-stage frequency domain enhancement. The method includes: acquiring remote sensing image data, annotating and preprocessing the remote sensing images, and constructing recognition training and testing datasets; embedding remote sensing enhancement modules after the res3, res4, and res5 stages of the backbone network ResNet50 to achieve the first stage of adaptive frequency domain enhancement; replacing the standard multi-head self-attention module with a linear attention module in the attention-based intra-scale feature interaction module of the neck network hybrid encoder to achieve efficient global semantic modeling; applying a multi-scale frequency domain fusion module to the highest-level features to achieve the second stage of fine frequency domain processing; and feeding the finally enhanced multi-scale features into a Transformer decoder after cross-scale fusion and query selection to output the target category and bounding box coordinates, thus completing the remote sensing target detection. The device includes a processor and a memory. This significantly improves the detection accuracy and inference efficiency of small and dense targets in remote sensing scenes.

Description

Technical Field

[0001] This invention relates to the field of remote sensing images, and in particular to a method and apparatus for target detection in remote sensing images based on dual-stage frequency domain enhancement. Background Technology

[0002] Target detection in remote sensing images is one of the core tasks of intelligent interpretation of remote sensing information, and it is widely used in urban planning, traffic monitoring, disaster assessment, and other fields. Unlike natural scene images, remote sensing images are characterized by high shooting angles, large target scale variations, arbitrary orientations, and complex backgrounds, posing significant challenges to target detection methods.

[0003] In recent years, end-to-end detection frameworks based on Transformers have demonstrated great potential in the field of object detection due to their global modeling capabilities. Among them, RT-DETR (Real-Time Detection Transformer), as a real-time detection Transformer, consists of a ResNet (Residual Network) backbone, a hybrid encoder neck network, and a Transformer decoder, achieving a good balance between accuracy and speed. However, existing RT-DETR frameworks still have the following shortcomings when applied to remote sensing images:

[0004] Firstly, as the number of layers in the backbone network increases, the resolution of the feature map gradually decreases, and high-frequency information such as the edges and textures of small targets are lost in multiple downsampling processes. Existing methods lack targeted enhancement of high-frequency details from the frequency domain perspective.

[0005] Secondly, the standard multi-head self-attention computational complexity of the neck network is O(N²d), which results in extremely high computational overhead and memory consumption when dealing with high-resolution remote sensing feature maps, thus limiting real-time inference capabilities.

[0006] Thirdly, existing feature pyramids use simple upsampling and downsampling and element-by-element addition for multi-scale fusion, lacking adaptive frequency domain processing capabilities for targets of different scales and orientations in remote sensing scenes. Summary of the Invention

[0007] This invention provides a remote sensing image target detection method and apparatus based on two-stage frequency domain enhancement. The invention achieves adaptive frequency domain enhancement by embedding a Remote Sensing Enhancement Module (RSEM) in the backbone network, integrates a Linear Attention Module (LMSA) in the neck network to achieve linear complexity global semantic modeling, and fuses a Multi-Scale Frequency Domain Fusion Module (MSFD) to achieve fine frequency domain processing through multi-scale, multi-directional Gabor filtering, small target enhancement, and three-scale adaptive fusion. The two-stage frequency domain enhancement is synergistic and complementary, significantly improving the detection accuracy and inference efficiency of small and dense targets in remote sensing scenes. See the description below for details:

[0008] Firstly, a remote sensing image target detection method based on two-stage frequency domain enhancement, the method comprising:

[0009] Collect remote sensing image data, annotate and preprocess the remote sensing images, and construct recognition training and testing datasets;

[0010] Remote sensing enhancement modules are embedded after the three stages of ResNet50 backbone network ResNet50 (res3, res4, res5) to achieve adaptive frequency domain enhancement in the first stage.

[0011] In the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling;

[0012] A multi-scale frequency domain fusion module is applied to the highest-level features to achieve the second-stage fine frequency domain processing;

[0013] The final enhanced multi-scale features are then fed into the Transformer decoder after cross-scale fusion and query selection, outputting the target category and bounding box coordinates to complete remote sensing target detection.

[0014] The first stage of adaptive frequency domain enhancement is as follows:

[0015] Perform a two-dimensional real-valued Fast Fourier Transform (FFT) along the spatial dimension on the feature maps output from each stage, using an orthogonal normalization mode to transform the spatial domain features into a frequency domain representation; calculate the normalized frequency radius at each frequency point in the frequency domain, using a fixed threshold. As a boundary, the frequency radius is smaller than The region is divided into low-frequency regions and a low-frequency mask is generated. The remaining area is divided into high-frequency regions and a high-frequency mask is generated. Learnable scalar weights are applied respectively. and A composite filter is constructed to suppress low frequencies and enhance high frequencies in the spectrum;

[0016] The enhanced spectrum is then converted back to spatial features using a two-dimensional inverse real-number fast Fourier transform. The spatially restored features are then input into a bottleneck structure for feature refinement. This bottleneck structure consists of the first... Convolutional layer, batch normalization layer, ReLU activation function, second The system consists of convolutional layers and batch normalization layers in sequence; finally, the refined features are processed through learnable fusion parameters. Residual fusion with the original input features, i.e., output ,in For the original input, Output for the bottleneck structure.

[0017] Specifically, in the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling as follows:

[0018] Through a unified Convolutional layers reduce the number of channels in the input features from Expand to Generate a joint representation of the query matrix Q, key matrix K, and value matrix V in one step, and then... Depthwise separable convolutions enhance the ability to perceive local spatial features;

[0019] Divide the matrix along the channel dimension into three matrices: Q, K, and V; reshape Q, K, and V into... The format of each attention head is [format missing], and the feature dimension of each head is [format missing]. The total number of spatial locations is L2 normalize Q and K along the feature dimension so that the feature vector at each spatial location has a magnitude of 1.

[0020] Learnable temperature parameters for each attention point right The matrix is ​​scaled to control the degree of concentration of attention distribution. Make the distribution of attention smoother and smaller This allows for greater focus on a few key areas; Convolutional output projection and residual connections generate global semantic enhancement features.

[0021] The multi-scale frequency domain fusion module is as follows:

[0022] The adaptive frequency domain filter extracts multi-directional frequency domain features through 32 Gabor filters in 4 scales × 8 directions; the small target enhancer adopts a three-branch collaborative structure of high-frequency enhancement, low-frequency suppression and small target attention; the multi-scale frequency domain fusion unit extracts features of different scales in parallel through three depths of separable convolution (3×3, 5×5, 7×7), and then fuses them with adaptive weights.

[0023] The composite filter enhances and suppresses the spectrum.

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] In the formula: and , respectively, are the normalized frequency values ​​in the height and width directions, and R is the Euclidean distance from each frequency point to the origin of the frequency domain; The region with a low-frequency threshold of 0.2 or higher is classified as a high-frequency region. and These are low-frequency masks and high-frequency masks, which are complementary and have a dimension of [1, 1, H, W / 2+1]. These are learnable low-frequency weight scalar parameters that control the degree to which low-frequency components are preserved. When the value is less than 1, low frequencies are suppressed; These are learnable high-frequency weight scalar parameters that control the amplification of high-frequency components. > High frequencies are enhanced at time 1; The weights of the composite filter are set to a value in the low-frequency region. In the high-frequency region, the value is... ; This represents element-wise multiplication; Let be the filtered frequency domain complex tensor, with dimensions equal to . Consistent.

[0030] The spatial domain recovery and feature refinement are as follows:

[0031] The enhanced spectrum is converted back to spatial domain features via a two-dimensional inverse real-number fast Fourier transform, then input into the bottleneck structure for feature refinement, and finally residual fusion is performed with the original input features using learnable fusion parameters.

[0032]

[0033]

[0034]

[0035] In the formula: This represents the two-dimensional inverse real fast Fourier transform, where [H, W] specifies the output space size to ensure consistency with the input. The enhanced feature map after spatial domain restoration has dimensions [B, C, H, W]. For the first Convolutional layers reduce the number of channels from Compress to ,in This refers to the compression ratio; This is the first batch of normalized layers; To modify the activation function of the linear unit, ; For the second Convolutional layers reduce the number of channels from Restore to ; This is the second batch of normalized layers; Refined features output by the bottleneck structure; This is the original input feature map for the RSEM module; These are learnable fusion parameters; This is the final output feature map of the RSEM module.

[0036] In a second aspect, a remote sensing image target detection device based on dual-stage frequency domain enhancement is provided. The device includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0037] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0038] The beneficial effects of the technical solution provided by this invention are:

[0039] 1. This invention proposes a remote sensing target detection framework TSF-DETR based on dual-stage frequency domain enhancement. This framework embeds remote sensing enhancement modules (RSEM) after the res3, res4, and res5 stages of the ResNet50 backbone network of RT-DETR. It transforms spatial features to the frequency domain through fast Fourier transform and uses learnable high-frequency and low-frequency weights to enhance high-frequency components and suppress low-frequency background, respectively. Different enhancement strategies are adopted for different network depths (conservative enhancement for res3, moderate enhancement for res4, and aggressive enhancement for res5), which effectively solves the problem of high-frequency information such as edge texture of small targets being gradually lost during multi-layer downsampling.

[0040] 2. Linear Attention Global Modeling: The TSF-DETR framework proposed in this invention integrates a linear attention module (LMSA) in the neck network hybrid encoder. By performing L2 normalization on the feature dimension of the query matrix Q and the key matrix K, and using the matrix multiplication associative law to first calculate K^T×V to obtain a small matrix of d×d dimension before multiplying it with Q, the generation of a large-scale N×N attention matrix is ​​avoided, and the computational complexity is reduced from O(N²d) to O(Nd²). While maintaining the global semantic modeling capability, the memory footprint is reduced, enabling the model to efficiently process high-resolution feature maps in remote sensing images.

[0041] 3. Multi-scale frequency domain fusion enhances small target detection capability: This invention designs a multi-scale frequency domain fusion module (MSFD), which consists of three sub-modules: an adaptive frequency domain filter, a small target enhancer, and a multi-scale frequency domain fusion unit. The adaptive frequency domain filter generates 32 Gabor filters in 4 scales × 8 directions and performs spatial adaptive weighted fusion through an adaptive weight learning network, effectively addressing the challenges of arbitrary orientation and scale diversity of remote sensing targets. The small target enhancer adopts a three-branch collaborative structure of high-frequency enhancement, low-frequency suppression, and small target attention to specifically enhance small target features and suppress background interference. The multi-scale frequency domain fusion unit fully leverages the complementary advantages of multi-scale features through depthwise separable convolutions at three scales: 3×3, 5×5, and 7×7, and an adaptive weight fusion mechanism.

[0042] 4. Two-stage synergistic complementarity: The two-stage frequency domain enhancement of this invention forms a complete frequency domain enhancement closed loop. The first stage injects coarse-grained global frequency domain features into the backbone network to provide high-quality frequency domain enhancement features for subsequent processing. The second stage combines global semantic modeling with linear attention and fine frequency domain processing with MSFD in the neck network to optimize features for the detection task. The synergistic effect of the two stages enables a dual improvement in accuracy and speed, and is especially suitable for complex detection scenarios in remote sensing images with dense small targets, large scale variations, and arbitrary target orientations. Attached Figure Description

[0043] Figure 1 This is a flowchart of a remote sensing image target detection method based on two-stage frequency domain enhancement;

[0044] Figure 2 This is a diagram showing the overall structure of the TSF-DETR.

[0045] Figure 3 This is a schematic diagram of the Remote Sensing Augmentation Module (RSEM).

[0046] Figure 4 This is a diagram comparing the original Multi-Head Self-Attention (MHSA) with the improved Linear Attention (LMSA).

[0047] Wherein, (a) is standard multi-head self-attention, and (b) is the linear attention of the present invention.

[0048] Figure 5 This is a schematic diagram of a multi-scale frequency domain fusion (MSFD) module.

[0049] Among them, (a) is an adaptive frequency domain filter, (b) is a small target enhancer, and (c) is a multi-scale frequency domain fusion device.

[0050] Figure 6 A schematic diagram of the raw remote sensing image input to the detector;

[0051] Figure 7 This is a schematic diagram of the detection results after identification using the system of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0053] Example 1

[0054] A remote sensing image target detection method based on two-stage frequency domain enhancement, see [link to relevant documentation]. Figures 1-5 This method comprises five core steps: dataset construction, adaptive frequency domain enhancement of the backbone network, linear attention global semantic modeling, multi-scale frequency domain fusion, and object detection, as detailed below:

[0055] 101: Collect remote sensing image data, annotate and preprocess the remote sensing images, and construct recognition training and testing datasets;

[0056] This step includes: collecting multi-source remote sensing data such as drone aerial images and satellite remote sensing images, covering a variety of typical targets such as vehicles, pedestrians, buildings, and ships; uniformly scaling the images to a resolution of 640×640 pixels, annotating the bounding boxes, and converting them to COCO (Common Objects) format; and dividing the training set and the test set at an 8:2 ratio.

[0057] 102: After the three stages of ResNet50 backbone network, res3, res4 and res5, remote sensing augmentation modules (RSEM) are embedded to realize the first stage of adaptive frequency domain augmentation;

[0058] This step includes: inputting the preprocessed remote sensing image into the ResNet50 backbone network, sequentially passing it through the conv1 convolutional layer and max pooling layer for initial feature extraction, and then sequentially passing it through four residual stages: res2, res3, res4, and res5. An RSEM module is inserted at the output of each of the res3, res4, and res5 stages. The adaptive frequency domain enhancement component within this module processes the data according to the following process:

[0059] First, a two-dimensional real-number fast Fourier transform (2D-rFFT) is performed on the feature maps output from each stage along the spatial dimension to transform the spatial domain features into a frequency domain representation; then, the normalized frequency radius of each frequency point in the frequency domain is calculated, with a fixed threshold. As a boundary, the frequency radius is smaller than The region is divided into low-frequency regions and a low-frequency mask is generated. The remaining area is divided into high-frequency regions and a high-frequency mask is generated. Learnable scalar weights are applied respectively. and A composite filter is constructed to suppress low frequencies and enhance high frequencies in the spectrum; then, a two-dimensional inverse real fast Fourier transform (2D-irFFT) is used to convert the enhanced spectrum back to spatial features; finally, the spatially restored features are input into a bottleneck structure for feature refinement. This bottleneck structure consists of a first... Convolutional layers (which reduce the number of channels from...) Compress to , (Compression ratio), batch normalization layer, ReLU activation function, second Convolutional layers (which reduce the number of channels from...) Restore to The process consists of a batch normalization layer and a batch normalization layer, which perform nonlinear transformation and denoising on the frequency domain enhancement features in a low-dimensional space. Finally, the refined features are processed through learnable fusion parameters. Residual fusion with the original input features, i.e., output ,in For the original input, Outputting the bottleneck structure. Different initial enhancement parameters are used for different depths of the backbone network: a conservative enhancement strategy is adopted in the res3 stage (512 channels, 80×80 spatial size), with high-frequency weights. Initial value is 1.2, low frequency weight. Initial value is 0.8, fusion parameter The initial value is 0.05, with only slight adjustments to the frequency domain response to preserve rich spatial details in the shallow layers; the res4 stage (1024 channels, 40×40 spatial size) employs a moderate enhancement strategy. The initial value is 1.5. The initial value is 0.6. The initial value is 0.1; the res5 stage (2048 channels, 20×20 space size) adopts an aggressive enhancement strategy. The initial value is 2.0. The initial value is 0.4. The initial value is 0.15, used to recover high-frequency information of small targets lost due to multiple downsampling in deeper layers with a relatively large gain. The learnable parameters are automatically updated to their optimal values ​​during training via backpropagation. The augmented output is feature maps at three scales. (512 channels, 80×80) (1024 channels, 40×40) (2048 channels, 20×20).

[0060] 103: In the AIFI (Attention-based Intra-scale Feature Interaction) module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module (LMSA) to achieve efficient global semantic modeling;

[0061] This step includes: firstly, through a unified... Convolutional layers reduce the number of channels in the input features from

[0062] Expand to Generate a joint representation of the query matrix Q, key matrix K, and value matrix V in one step, and then...

[0063] Depthwise separable convolution (each channel is executed independently) Spatial convolution enhances the perception of local spatial features, and then it is divided into three matrices Q, K, and V along the channel dimension; Q, K, and V are then reshaped into... The format of each attention head is [format missing], and the feature dimension of each head is [format missing]. The total number of spatial locations is L2 normalize Q and K in the feature dimension direction so that the feature vector at each spatial location has a magnitude of 1.

[0064] In standard multi-head self-attention, the product of the transposes of Q and K needs to be calculated first. The result of this product is Attention weight matrix (e.g., when At that time, the matrix contains (n elements), used to measure pairwise similarity between all spatial locations, with a computational complexity of O(n). The linear attention in this embodiment of the invention employs a different computation order: because L2 normalization replaces the Softmax nonlinear operation, matrix chain multiplication... Satisfies the associative law, that is Therefore, the order of operations can be changed, calculating first... The dimension is , The dimension is The matrix obtained by multiplying the two is The dimension is only ,Include The number of elements is much smaller than that in standard attention. An attention matrix with 1 element; then multiply this by Q. small matrix ,Right now , dimension This yields an output that is mathematically equivalent to standard attention. By changing the order of operations, the computational complexity is reduced from... Due to the number of spatial locations in practical applications Much larger than the feature dimension per head (like Therefore, the amount of computation is greatly reduced.

[0065] Finally, each attention point is assessed using an independent learnable temperature parameter. (Initial value is 0.5) The matrix is ​​scaled to control the degree of concentration of attention distribution—a larger scale... Make the distribution of attention smoother and smaller This allows for greater focus on a few key areas; Convolutional output projection and residual connections generate global semantic enhancement features.

[0066] 104: After linear attention modeling, a multi-scale frequency domain fusion module (MSFD) is applied to the highest-level features to achieve the second-stage fine frequency domain processing;

[0067] The MSFD module comprises three sub-modules: an adaptive frequency domain filter that extracts multi-directional frequency domain features through 32 Gabor filters in 4 scales × 8 directions; a small target enhancer that employs a three-branch collaborative structure of high-frequency enhancement, low-frequency suppression, and small target attention; and a multi-scale frequency domain fusion unit that extracts features at different scales in parallel through three depths of separable convolution (3×3, 5×5, and 7×7) and then fuses them using adaptive weights.

[0068] 105: The enhanced multi-scale features are fed into the Transformer decoder after cross-scale fusion and query selection, and the target category and bounding box coordinates are output to complete the remote sensing target detection.

[0069] In summary, the embodiments of the present invention significantly improve the detection accuracy and inference efficiency of small and dense targets in remote sensing scenarios through steps 101-105.

[0070] Example 2

[0071] The following section combines specific calculation formulas, numerical examples, and... Figures 1-7 The solution in Example 1 will be further described below:

[0072] 201: Data Preprocessing;

[0073] like Figure 1 The diagram shows the algorithm flowchart of an embodiment of the present invention. Remote sensing image data is acquired, and all images are scaled to 640×640 pixels. Taking the VisDrone drone aerial photography dataset as an example... Figure 6 The image shown is the original remote sensing image of an urban traffic scene, containing various targets such as dense vehicles, pedestrians, and buildings. The targets vary greatly in scale, and the background is complex. After the annotation information was converted to COCO format, the dataset was divided in an 8:2 ratio.

[0074] 202: Backbone Network Adaptive Frequency Domain Enhancement (RSEM);

[0075] like Figure 2 The diagram shown is the overall architecture of TSF-DETR. Figure 3 The diagram shows the RSEM module structure. The res3, res4, and res5 stages of the backbone network ResNet50 output feature maps with 512, 1024, and 2048 channels, respectively, and the outputs of each stage are fed into the corresponding RSEM module.

[0076] Taking the res5 stage as an example, the input dimensions are [B, 2048, 20, 20], where B is the batch size. The RSEM processing procedure is as follows:

[0077] (1) Frequency domain transformation: for the input feature map Perform a two-dimensional real fast Fourier transform (2D-rFFT) along the spatial dimension, using an orthogonal normalization mode (dividing by each of the forward and inverse transforms). To maintain L2 norm consistency, the spatial domain features are transformed into a frequency domain representation:

[0078]

[0079] In the formula: Represents the two-dimensional real number fast Fourier transform; The input spatial feature map has dimensions [B,C,H,W] = [B, 2048, 20, 20]. For a frequency domain complex tensor, due to the conjugate symmetry of the real FFT, the width dimension is compressed to... The dimensions are [B, C, H, W / 2+1] = [B, 2048, 20, 11].

[0080] (2) Frequency domain filtering: Calculate the normalized frequency radius at each frequency point in the frequency domain. With a fixed threshold To delineate high and low frequency regions, complementary binary masks are generated. and Construct a composite filter to enhance and suppress the spectrum:

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] In the formula: and These are the normalized frequency values ​​in the height and width directions, respectively. (Fast Fourier Transform frequency generation function, used to generate a normalized frequency sequence in the height direction) and (Real-number Fast Fourier Transform frequency generation function, used to generate a normalized frequency sequence in the width direction) The function generates the frequency with values ​​in the ranges of [-0.5, 0.5) and [0, 0.5], respectively; R is the Euclidean distance from each frequency point to the origin of the frequency domain, representing the spatial frequency magnitude corresponding to that point; As a low-frequency threshold, regions with a normalized frequency radius less than 0.2 are classified as low-frequency regions (corresponding to gradually changing information such as flat backgrounds in the image), while regions with a normalized frequency radius greater than or equal to 0.2 are classified as high-frequency regions (corresponding to detailed information such as target edges and textures). and These are low-frequency masks and high-frequency masks, which are complementary and have a dimension of [1, 1, H, W / 2+1]. These are learnable low-frequency weight scalar parameters that control the degree to which low-frequency components are preserved. When the value is less than 1, low frequencies are suppressed; These are learnable high-frequency weight scalar parameters that control the amplification of high-frequency components. > High frequencies are enhanced at time 1; The weights of the composite filter are set to a value in the low-frequency region. In the high-frequency region, the value is... ; This represents element-wise multiplication; Let be the filtered frequency domain complex tensor, with dimensions equal to . Consistent.

[0087] (3) Spatial domain restoration and feature refinement: The enhanced spectrum is converted back to spatial domain features by two-dimensional inverse real fast Fourier transform (2D-irFFT), and then input into the bottleneck structure for feature refinement. Finally, residual fusion is performed with the original input features through learnable fusion parameters.

[0088]

[0089]

[0090]

[0091] In the formula: This represents the two-dimensional inverse real fast Fourier transform, using the same orthogonal normalization mode as the forward transform. [H, W] specifies the output space size to ensure consistency with the input. The enhanced feature map after spatial domain restoration has dimensions [B, C, H, W]. For the first Convolutional layers (without bias) reduce the number of channels from Compress to ,in This refers to the compression ratio; This is the first batch of normalized layers; To modify the activation function of the linear unit, ; For the second Convolutional layers (without bias) reduce the number of channels from Restore to ; This is the second batch normalization layer; taking the res5 stage as an example, After compression, the number of channels is 2048 / 16=128, meaning it undergoes... The bottleneck structure performs nonlinear transformation and denoising on the frequency domain enhancement features in a low-dimensional space, reducing the spectral leakage artifacts that may be introduced by the frequency domain transformation. Refined features for the bottleneck structure output; This is the original input feature map for the RSEM module; These are learnable fusion parameters, scalar values, that control the proportion of the frequency domain enhancement features that contribute to the final output; This is the final output feature map of the RSEM module, with dimensions [B, C, H, W] identical to the input. Residual connections ensure that original feature information is not lost, while also providing gradient-directed paths for backpropagation.

[0092] For different characteristic stages at different depths of the backbone network, the RSEM module uses differentiated initial enhancement parameters: res3 stage (number of channels) (The space size is 80×80) adopts a conservative reinforcement strategy. The initial value is 1.2 (the high-frequency response is amplified to 1.2 times the original, with only a slight enhancement of edge details). The initial value is 0.8 (low-frequency response is reduced to 80% of its original value, and background is slightly suppressed). The initial value is 0.05 (frequency domain enhancement features account for only 5% of the final output). Shallow features have high spatial resolution and retain rich spatial details; excessive enhancement can easily introduce noise. The res4 stage (number of channels) (40×40) A moderate enhancement strategy was adopted. , , Mid-level features exhibit semantic enhancement but detail begins to diminish; moderate enhancement balances semantics and detail; res5 stage (number of channels) (Spatial dimensions 20×20) adopts an aggressive enhancement strategy. (High-frequency response magnified to 2x) (Low-frequency response reduced to 40%) (Frequency domain enhancement features account for 15%), and the spatial resolution of deep features after four 2x downsampling operations is only a fraction of the input. High-frequency edge and texture information of small targets has been largely lost, so a high gain is used to focus on its recovery. Specific numerical examples using res5: the original spectral amplitude at a certain frequency point is... If the point is located in the high-frequency region, the amplitude after filtering becomes If it is located in the low-frequency region, the amplitude after filtering becomes... The initial values ​​mentioned above are all automatically updated to their optimal values ​​through backpropagation during the training process.

[0093] After the RSEM module performs stage-by-stage frequency domain enhancement as described above, the feature maps output by each stage (res3, res4, res5) are shown. (512 channels, 80×80) (1024 channels, 40×40) High-frequency information of small targets in the (2048 channels, 20×20) array was specifically recovered and enhanced, while low-frequency background interference was suppressed. These three enhanced multi-scale features were then fed into the hybrid encoder of the neck network, processed through three independent... The convolutional layers project the number of channels at each scale from 512, 1024, and 2048 to a uniform scale. The dimensions are calculated to obtain feature maps with dimensions [B, 256, 80, 80], [B, 256, 40, 40], and [B, 256, 20, 20], which are used for subsequent linear attention global semantic modeling and multi-scale frequency domain fusion processing.

[0094] 203: Linear Attention Global Semantic Modeling (LMSA);

[0095] like Figure 4 As shown, Figure 4 Figure (a) shows the standard multi-head self-attention MHSA. Figure 4 Figure (b) shows a linear attentional model (LMSA) according to an embodiment of the present invention.

[0096] Standard MHSA, taking the highest-level features (N=400) as an example, requires calculating an attention matrix of 400×400=160,000 elements, with a complexity of O(N²d). The LMSA implementation of this invention changes the calculation order:

[0097] (1) Q, K, V (total channels 768) are generated by 1×1 convolution, and local perception is enhanced by 3×3 DWConv (depth separable convolution).

[0098] (2) Reshape into an 8-head format and perform L2 normalization on Q and K.

[0099] (3) Use the associative law of matrix multiplication to change the order of operations, and calculate the key matrix first. Then multiply by the query matrix:

[0100]

[0101]

[0102] In the formula: The key matrix is ​​L2 normalized, with dimensions [B, 8, 32, 400]. This is the transpose of the value matrix (by swapping the last two dimensions), with dimensions [B, 8, 400, 32]. It is a key-value matrix. and The matrix multiplication result has dimensions [B, 8, 32, 32] and contains only... One element; The temperature parameter is a learnable parameter with dimensions [h, 1, 1] = [8, 1, 1], meaning each attention head has an independent scalar temperature value, initially set to 0.5, used to control the concentration of attention distribution. This is the transpose of the query matrix after L2 normalization, with dimensions [B, 8, 400, 32]. The attention output, with dimensions [B, 8, 400, 32], is transposed and reshaped to restore it to [B, 256, H, W]. Finally, after... Convolutional output projection and residual connections generate global semantic enhancement features.

[0103] The computational complexity is reduced from O(N²d) to O(Nd²), and the memory usage is reduced by about 156 times compared to the standard attention matrix of 400×400 for the intermediate matrix of 32×32; when N=6400 (80×80 feature maps), the memory usage is reduced by about 40,000 times.

[0104] 204: Multi-scale frequency domain fusion (MSFD);

[0105] like Figure 5 As shown, MSFD consists of three sub-modules:

[0106] (1) Adaptive frequency domain filter Figure 5 (a) Figure: Generating 4 scales ( )×8 directions

[0107] ( (j=1,2,...,8, corresponding to 0° to 157.5°) A total of 32 values. Gabor filter core:

[0108]

[0109]

[0110]

[0111] In the formula: and for The spatial coordinates on the filter kernel are all uniformly sampled within the range of [-1, 1]. The direction angle controls the filter's sensitivity to textures in specific directions; and For the Coordinates after angular rotation; Let be the Gaussian envelope function, and its standard deviation be... A fixed value controls the spatial range of the filter; For scale parameters, For spatial frequency, The smaller the value, the higher the spatial frequency, and the more refined the texture and edge information it can capture. It has the highest frequency and is suitable for small targets; (Lowest frequency, suitable for large targets).

[0112] 32 Gabor filters are used in a depthwise separable convolution (groups= That is, each channel executes independently. Convolution (with padding=3 to maintain spatial dimensions) is applied to the input feature map. Simultaneously, an adaptive weight learning network (global average pooling → ...) is used. convolution( →ReLU→ convolution( (→Sigmoid) generates 32 adaptive weights, combined with learnable scale weights. (dimension) ) and direction weight (dimension) The 32 filtered results were weighted and fused.

[0113] (2) Small target enhancer ( Figure 5 (Figure b): The process employs a three-branch collaborative approach: a high-frequency enhancement branch, a low-frequency suppression branch, and a small target detection head.

[0114]

[0115]

[0116] In the formula: The input features are after adaptive frequency domain filtering; For the output of the high-frequency enhancement branch, by Depthwise convolution (groups= Batch Normalization → ReLU → Standard convolution → batch normalization → ReLU construction, extracting high-frequency details such as target edges and textures; For low-frequency suppression masking, global average pooling → convolution( →ReLU→ convolution( →Sigmoid is generated, with a value range of (0,1); 0.5 is the suppression strength coefficient, which suppresses low-frequency background by up to 50% to avoid excessive information loss; These are characteristics after low-frequency suppression; For small-target attention maps, by convolution( →ReLU→ convolution( →Sigmoid, with input as (Using high-frequency features to locate small targets), output dimension [B, 1, H, W], a value close to 1 indicates that the location is highly likely to be a small target area; This is element-wise multiplication; This is the enhanced feature. Taking a specific numerical value as an example: Input... , hour, Low-frequency background is suppressed by 30%; if , (High probability small target area), then Compared to the original input, it is enhanced by 42%; if (Background area) Only slight changes.

[0117] (3) Multi-scale frequency domain fusion unit Figure 5 (C) Figure: Through After reducing the number of input channels from C to C / 3 using convolution, each channel is then connected to... , , Three depthwise separable convolutions (each channel is independently convolved, with padding of 1, 2, and 3 respectively to maintain spatial dimensions) are used in parallel to extract features from different receptive fields, which are then fused using adaptive weighting.

[0118]

[0119] In the formula: For small-scale branch output, after Depthwise separable convolutions extract fine local features of small targets, with dimensions [B, C / 3, H, W]. For mesoscale branch output, after Depthwise separable convolutions capture contextual features of medium-sized target regions; For large-scale branch output, after Depthwise separable convolutions capture global structural features of large targets; C / 3 represents the number of channels in each branch ( hour ); The adaptive fusion weights are calculated using global average pooling. convolution( → Softmax (normalizes along the channel dimension to ensure...) Generates a feature with dimensions [B, 3, 1, 1], and multiplies it element-wise with each branch feature through a broadcast mechanism; To perform the splicing operation along the channel dimension, the three The branch splicing of the channel is aisle; for Fusion convolutional layers ( Following batch normalization and ReLU activation, the number of channels is restored to [normal value]. ; The fused features have dimensions [B, 256, H, W]. The weights, after Softmax normalization, can be adaptively adjusted based on the input content, especially when small targets dominate the detection scene. It can reach 0.5, adaptively enhancing the contribution of small-scale branches.

[0120] Finally, the output projection layer and residual are connected:

[0121]

[0122] In the formula: Output features for the multi-scale fusion unit; To output the projection layer, by convolution( (padding=1) → Batch Normalization → ReLU → convolution( It consists of four operation sequences, which further perform nonlinear transformation and refinement on the fused features; The raw input features of the MSFD module (before input projection); For residual connection paths, an identity mapping is used when the number of input and output channels are the same; otherwise, it is used through... Convolution is used for channel alignment; The final output features of the MSFD module are [B, 256, H, W].

[0123] 205: Detection output.

[0124] The enhanced features are then fused across scales using CCFM and subjected to Uncertainty-minimal Query Selection before being fed into the Transformer decoder, which outputs the target category and bounding box. For example... Figure 7 As shown, the embodiments of the present invention can accurately detect dense vehicles (car 0.97), trucks (truck 0.88) and small pedestrians (pedestrians). Different categories are distinguished by different colored bounding boxes, which verifies the detection effect of the present invention in remote sensing scenes with dense small targets, large scale variations and complex backgrounds.

[0125] Training employs the DINOLoss loss function (which includes classification loss, bounding box L1 loss, and GIoU loss), uses the Hungarian matching algorithm for one-to-one matching between predictions and ground truth labels, and employs the AdamW optimizer for parameter updates.

[0126] Example 3

[0127] A remote sensing image target detection device based on dual-stage frequency domain enhancement includes: a processor, a GPU accelerator, a memory, an input / output interface, and a communication module. The memory stores program instructions, and the processor calls the program instructions stored in the memory to make the device execute the steps in Embodiment 1.

[0128] Collect remote sensing image data, annotate and preprocess the remote sensing images, and construct recognition training and testing datasets;

[0129] Remote sensing enhancement modules are embedded after the three stages of ResNet50 backbone network ResNet50 (res3, res4, res5) to achieve adaptive frequency domain enhancement in the first stage.

[0130] In the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling;

[0131] A multi-scale frequency domain fusion module is applied to the highest-level features to achieve the second-stage fine frequency domain processing;

[0132] The final enhanced multi-scale features are then fed into the Transformer decoder after cross-scale fusion and query selection, outputting the target category and bounding box coordinates to complete remote sensing target detection.

[0133] The first stage of adaptive frequency domain enhancement is as follows:

[0134] Perform a two-dimensional real-valued Fast Fourier Transform (FFT) along the spatial dimension on the feature maps output from each stage, using an orthogonal normalization mode to transform the spatial domain features into a frequency domain representation; calculate the normalized frequency radius at each frequency point in the frequency domain, using a fixed threshold. As a boundary, the frequency radius is smaller than The region is divided into low-frequency regions and a low-frequency mask is generated. The remaining area is divided into high-frequency regions and a high-frequency mask is generated. Learnable scalar weights are applied respectively. and A composite filter is constructed to suppress low frequencies and enhance high frequencies in the spectrum;

[0135] The enhanced spectrum is then converted back to spatial features using a two-dimensional inverse real-number fast Fourier transform. The spatially restored features are then input into a bottleneck structure for feature refinement. This bottleneck structure consists of the first... Convolutional layer, batch normalization layer, ReLU activation function, second The system consists of convolutional layers and batch normalization layers in sequence; finally, the refined features are processed through learnable fusion parameters. Residual fusion with the original input features, i.e., output ,in For the original input, Output for the bottleneck structure.

[0136] Specifically, in the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling as follows:

[0137] Through a unified Convolutional layers reduce the number of channels in the input features from Expand to Generate a joint representation of the query matrix Q, key matrix K, and value matrix V in one step, and then... Depthwise separable convolutions enhance the ability to perceive local spatial features;

[0138] Divide the matrix along the channel dimension into three matrices: Q, K, and V; reshape Q, K, and V into... The format of each attention head is [format missing], and the feature dimension of each head is [format missing]. The total number of spatial locations is L2 normalize Q and K along the feature dimension so that the feature vector at each spatial location has a magnitude of 1.

[0139] Learnable temperature parameters for each attention point right The matrix is ​​scaled to control the degree of concentration of attention distribution. Make the distribution of attention smoother and smaller This allows for greater focus on a few key areas; Convolutional output projection and residual connections generate global semantic enhancement features.

[0140] The multi-scale frequency domain fusion module is as follows:

[0141] The adaptive frequency domain filter extracts multi-directional frequency domain features through 32 Gabor filters in 4 scales × 8 directions; the small target enhancer adopts a three-branch collaborative structure of high-frequency enhancement, low-frequency suppression and small target attention; the multi-scale frequency domain fusion unit extracts features of different scales in parallel through three depths of separable convolution (3×3, 5×5, 7×7), and then fuses them with adaptive weights.

[0142] Among them, the composite filter enhances and suppresses the spectrum:

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] In the formula: and , respectively, are the normalized frequency values ​​in the height and width directions, and R is the Euclidean distance from each frequency point to the origin of the frequency domain; The region with a low-frequency threshold of 0.2 or higher is classified as a high-frequency region. and These are low-frequency masks and high-frequency masks, which are complementary and have a dimension of [1, 1, H, W / 2+1]. These are learnable low-frequency weight scalar parameters that control the degree to which low-frequency components are preserved. When the value is less than 1, low frequencies are suppressed; These are learnable high-frequency weight scalar parameters that control the amplification of high-frequency components. > High frequencies are enhanced at time 1; The weights of the composite filter are set to a value in the low-frequency region. In the high-frequency region, the value is... ; This represents element-wise multiplication; Let be the filtered frequency domain complex tensor, with dimensions equal to . Consistent.

[0149] Among them, airspace restoration and feature refinement are:

[0150] The enhanced spectrum is converted back to spatial domain features via a two-dimensional inverse real-number fast Fourier transform, then input into the bottleneck structure for feature refinement, and finally residual fusion is performed with the original input features using learnable fusion parameters.

[0151]

[0152]

[0153]

[0154] In the formula: This represents the two-dimensional inverse real fast Fourier transform, where [H, W] specifies the output space size to ensure consistency with the input. The enhanced feature map after spatial domain restoration has dimensions [B, C, H, W]. For the first Convolutional layers reduce the number of channels from Compress to ,in This refers to the compression ratio; This is the first batch of normalized layers; To modify the activation function of the linear unit, ; For the second Convolutional layers reduce the number of channels from Restore to ; This is the second batch of normalized layers; Refined features output by the bottleneck structure; This is the original input feature map for the RSEM module; These are learnable fusion parameters; This is the final output feature map of the RSEM module.

[0155] This device can be deployed in the following scenarios:

[0156] Remote sensing satellite ground station: receives large-format remote sensing images transmitted from satellites in real time, performs batch target detection and analysis, and supports applications such as urban planning and disaster assessment;

[0157] Drone edge: Deploy a lightweight version on an airborne embedded computing platform to enable real-time drone inspection and target alarm;

[0158] Cloud-Hybrid Architecture: Image preprocessing and preliminary feature extraction are performed at the edge, while the cloud performs complete two-stage frequency domain enhancement and fine detection, balancing real-time performance and accuracy.

[0159] The entire system supports functions such as hot model updates, access to multiple data sources, and visualization export of detection results, and can be seamlessly integrated into existing remote sensing information processing platforms.

[0160] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0161] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0162] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0163] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0164] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0165] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0166] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0167] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0168] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0169] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing image target detection method based on two-stage frequency domain enhancement, characterized in that, The method includes: Collect remote sensing image data, annotate and preprocess the remote sensing images, and construct recognition training and testing datasets; Remote sensing enhancement modules are embedded after the three stages of ResNet50 backbone network ResNet50 (res3, res4, res5) to achieve adaptive frequency domain enhancement in the first stage. In the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling; A multi-scale frequency domain fusion module is applied to the highest-level features to achieve the second-stage fine frequency domain processing; The final enhanced multi-scale features are then fed into the Transformer decoder after cross-scale fusion and query selection, outputting the target category and bounding box coordinates to complete remote sensing target detection.

2. The remote sensing image target detection method based on dual-stage frequency domain enhancement according to claim 1, characterized in that, The first stage of adaptive frequency domain enhancement is as follows: A two-dimensional real-number fast Fourier transform is performed on the feature maps output at each stage along the spatial dimension, and an orthogonal normalization mode is adopted to transform the spatial domain features into a frequency domain representation. Calculate the normalized frequency radius at each frequency point in the frequency domain, using a fixed threshold. As a boundary, the frequency radius is smaller than The region is divided into low-frequency regions and a low-frequency mask is generated. The remaining area is divided into high-frequency regions and a high-frequency mask is generated. Learnable scalar weights are applied respectively. and A composite filter is constructed to suppress low frequencies and enhance high frequencies in the spectrum; The enhanced spectrum is then converted back to spatial features using a two-dimensional inverse real-number fast Fourier transform. The spatially restored features are then input into a bottleneck structure for feature refinement. This bottleneck structure consists of the first... Convolutional layer, batch normalization layer, ReLU activation function, second The system consists of convolutional layers and batch normalization layers in sequence; finally, the refined features are processed through learnable fusion parameters. Residual fusion with the original input features, i.e., output ,in For the original input, Output for the bottleneck structure.

3. The remote sensing image target detection method based on dual-stage frequency domain enhancement according to claim 1, characterized in that, In the attention-based intra-scale feature interaction module of the neck network hybrid encoder, the standard multi-head self-attention is replaced with a linear attention module to achieve efficient global semantic modeling as follows: Through a unified Convolutional layers reduce the number of channels in the input features from Expand to Generate a joint representation of the query matrix Q, key matrix K, and value matrix V in one step, and then... Depthwise separable convolutions enhance the ability to perceive local spatial features; Divide the matrix along the channel dimension into three matrices: Q, K, and V; reshape Q, K, and V into... The format of each attention head is [format missing], and the feature dimension of each head is [format missing]. The total number of spatial locations is L2 normalize Q and K along the feature dimension so that the feature vector at each spatial location has a magnitude of 1. Learnable temperature parameters for each attention point right The matrix is ​​scaled to control the degree of concentration of attention distribution. Make the distribution of attention smoother and smaller This allows you to focus your attention on a few key areas. through Convolutional output projection and residual connections generate global semantic enhancement features.

4. The remote sensing image target detection method based on dual-stage frequency domain enhancement according to claim 1, characterized in that, The multi-scale frequency domain fusion module is: The adaptive frequency domain filter extracts multi-directional frequency domain features through 32 Gabor filters in 4 scales × 8 directions; the small target enhancer adopts a three-branch collaborative structure of high-frequency enhancement, low-frequency suppression and small target attention; the multi-scale frequency domain fusion unit extracts features of different scales in parallel through three depths of separable convolution (3×3, 5×5, 7×7), and then fuses them with adaptive weights.

5. A remote sensing image target detection method based on dual-stage frequency domain enhancement according to claim 2, characterized in that... Therefore, the composite filter enhances and suppresses the spectrum: ； ； ； ； ； In the formula: and , respectively, are the normalized frequency values ​​in the height and width directions, and R is the Euclidean distance from each frequency point to the origin of the frequency domain; The region with a low-frequency threshold of 0.2 or higher is classified as a high-frequency region. and These are low-frequency masks and high-frequency masks, which are complementary and have a dimension of [1, 1, H, W / 2+1]. These are learnable low-frequency weight scalar parameters that control the degree to which low-frequency components are preserved. When the value is less than 1, low frequencies are suppressed; These are learnable high-frequency weight scalar parameters that control the amplification of high-frequency components. > High frequencies are enhanced at time 1; The weights of the composite filter are set to a value in the low-frequency region. In the high-frequency region, the value is... ; This represents element-wise multiplication; Let be the filtered frequency domain complex tensor, with dimensions equal to . Consistent.

6. The remote sensing image target detection method based on dual-stage frequency domain enhancement according to claim 2, wherein its features are as follows: The characteristic is that the spatial domain recovery and feature refinement are as follows: The enhanced spectrum is converted back to spatial domain features via a two-dimensional inverse real-number fast Fourier transform, then input into the bottleneck structure for feature refinement, and finally residual fusion is performed with the original input features using learnable fusion parameters. ； ； ； In the formula: This represents the two-dimensional inverse real fast Fourier transform, where [H, W] specifies the output space size to ensure consistency with the input. The enhanced feature map after spatial domain restoration has dimensions [B, C, H, W]. For the first Convolutional layers reduce the number of channels from Compress to ,in This refers to the compression ratio; This is the first batch of normalized layers; To modify the activation function of the linear unit, ; For the second Convolutional layers reduce the number of channels from Restore to ; This is the second batch of normalized layers; Refined features output by the bottleneck structure; This is the original input feature map for the RSEM module; These are learnable fusion parameters; This is the final output feature map of the RSEM module.

7. A remote sensing image target detection device based on dual-stage frequency domain enhancement, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.

Images