A remote sensing target detection method and system for low-visibility images

Abstract

The present application relates to the technical field of remote sensing monitoring, and particularly relates to a remote sensing target detection method and system for low-visibility images. The method comprises the following steps: acquiring multi-modal remote sensing image data; performing defogging enhancement processing on a low-visibility input image; performing splicing fusion on the normalized RGB and IR images after defogging; adopting a hybrid backbone structure of fused Transformer, Mamba and CNN to perform layer-by-layer encoding on the multi-modal fusion image; performing frequency domain decomposition on the backbone output features based on two-dimensional wavelet transform; generating an HR feature map by adaptively selecting a key region; and obtaining a detection target frame based on cross-scale aggregation of the HR feature map. Through multi-modal image defogging enhancement and feature distillation mechanism, the present application effectively enhances the definition and contrast of remote sensing images under adverse weather conditions such as fog, haze and rainy days, reduces the shielding interference of environmental degradation on small target detection, and enhances the stability and adaptability of the model in complex weather scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing monitoring, in particular to a remote sensing target detection method and system for low-visibility images. BACKGROUND

[0002] With the development of remote sensing technology, remote sensing images have been widely used in environmental monitoring, disaster assessment, urban planning and other fields. Target detection, as a core task in remote sensing image analysis, aims to automatically identify and accurately locate various ground object targets from high-resolution images. Convolutional Neural Networks (CNN) and Transformers are commonly used in target detection algorithms.

[0003] CNN-based detection methods automatically extract hierarchical features in images by constructing multi-layer convolutional structures, and have good local feature modeling capabilities, performing well in small target detection and edge information extraction. The Transformer model realizes global information interaction between any positions in the image through self-attention mechanism, breaking through the limitation of the local receptive field of CNN, and showing strong advantages in modeling long-distance dependencies and target recognition tasks in complex backgrounds.

[0004] Although the above methods have made certain progress, there are still technical problems in practical applications. In fog, haze, low light, rain, snow and other adverse weather conditions, the quality of remote sensing images decreases significantly, resulting in insufficient clarity, blurred target boundaries, reduced contrast and other problems, making it difficult for detection algorithms to accurately identify ground object targets, especially the detection accuracy of small or low-contrast targets decreases significantly. The Transformer model has high memory usage and slow inference speed when processing high-resolution remote sensing images due to the quadratic growth of the computational complexity of its self-attention mechanism, making it difficult to meet the deployment requirements in real-time and resource-constrained environments.

[0005] Therefore, it is necessary to develop more accurate and efficient remote sensing target detection methods to improve the accuracy and efficiency of remote sensing target detection. SUMMARY

[0006] In order to solve the problems of image clarity decline, target boundary blur, low detection accuracy and poor model robustness of remote sensing target detection methods in fog, haze, low light, rain, snow and other complex weather conditions, the present application provides a remote sensing target detection method and system for low-visibility images.

[0007] In a first aspect, the present application provides a remote sensing target detection method for low-visibility images, which adopts the following technical solution:

[0008] A remote sensing target detection method for low-visibility images, comprising:

[0009] acquiring multi-modal remote sensing image data;

[0010] foggy input image is enhanced by dehazing;

[0011] The dehazed RGB and IR images are normalized and spliced and fused;

[0012] A hybrid backbone structure that fuses Transformer, Mamba and CNN is adopted to encode the multi-modal fusion image layer by layer;

[0013] The backbone output features are decomposed in the frequency domain based on two-dimensional wavelet transform;

[0014] An HR feature map is generated by adaptively selecting a key region;

[0015] The detection target frame is obtained based on cross-scale aggregation of the HR feature map.

[0016] Further, the foggy input image is enhanced by dehazing, which includes using a pre-trained teacher network on clear multi-modal images to guide a student network to perform feature distillation on the foggy input image, and through gradient alignment and degradation response constraint, the student network learns robust representation ability under foggy conditions, thereby enhancing the clarity and contrast of the image, reducing the shielding effect of fog on small target detection, and finally outputting enhanced RGB and IR images for subsequent fusion processing. The pixel-level difference between clear images and foggy images is calculated to obtain a preliminary degradation map, which is then normalized to quantify the degradation degree of different regions in the image. Finally, a degradation weighted response distillation method is adopted, which gives higher weight to the severely degraded regions according to the degradation degree, and in the response map distillation process of classification and positioning, the student network pays more attention to the targets in the severely degraded regions, thereby realizing the targeted improvement of the detection ability of the student network in different degraded regions.

[0017] Further, the degradation weighted response distillation method includes calculating the pixel difference between the foggy image and the clear image according to the formula D represents the difference map, is the remote sensing image under clear conditions, is the input remote sensing image under low visibility conditions; according to the formula D is normalized, wherein max(D) represents the maximum pixel difference value in the difference map D, and min(D) represents the minimum pixel difference value in the difference map D; the degradation degree map obtained after normalization reflects the degradation intensity of each region of the image, and is used in the distillation process of the classification and positioning response map to guide the student network to focus on the severely degraded regions; according to the formula, , the weighted response distillation loss is calculated; wherein C, H and W represent the channel number, height and width of the feature map, respectively, and is the activation value of the c-th channel, h-th row, w-th column in the response feature map of the teacher network; according to the formula, , the defogging enhanced loss function is obtained, wherein is the edge distillation loss, used to constrain the perceptual consistency of the teacher and student networks in the edge structure; is the degradation weighted response distillation loss, used to emphasize the response consistency of the severe degradation area; is the student network regular detection loss, including the classification loss and the bounding box regression loss; represents the weight coefficient of the edge distillation loss; represents the weight coefficient of the degradation weighted response distillation loss; represents the weight coefficient of the regular detection loss.

[0018] Further, the RGB and IR images after defogging are normalized and spliced, including splicing along the channel dimension into a four-channel image (R, G, B, I), wherein according to the formula the fusion image is obtained, wherein, , C is the number of channels, H and W are the height and width of the image, {R, G, B} and {I} represent the RGB image and the IR image, and Concat() represents the connection operation along the channel axis; the RGB and IR images after defogging are normalized to [0, 1], specifically, X is twice sampled to 1 / n size of the original image to complete the SR module and speed up the training process, according to the formula the sampling image is obtained, wherein, D() represents an n-time downsampling operation using bilinear interpolation, and then the twice sampling result is sent to the backbone to generate multi-level features.

[0019] Further, the hybrid backbone structure of fusion Transformer, Mamba and CNN is used to encode the multi-modal fusion image layer by layer, including in the high-resolution stage, using a lightweight CNN module to extract edge, texture local information, adapting to the small target detection task; in the medium-resolution stage, introducing a state space modeling module based on Mamba, using a multi-direction scanning mechanism to capture the global structure of rotating and deforming targets; in the low-resolution stage, combining a Transformer module based on multi-scale sliding window attention, realizing semantic modeling and regional attention of large-scale targets, and the features of each stage are connected through cross-layer connection and multi-scale fusion to form a complete feature pyramid, outputting multi-scale feature maps; wherein, the dynamic convolution module adaptively generates convolution kernel weights according to the input feature content, realizing personalized processing of different image regions, represented as: , wherein, is the weight coefficient generated according to the input feature x, for the corresponding dynamic convolution kernel, * is the convolution operation.

[0020] Further, the two-dimensional wavelet transform-based frequency domain decomposition of the backbone output feature includes first extracting preliminary features through convolution operation on the backbone output feature map, then applying low-pass filter LPF and high-pass filter HPF along the horizontal and vertical directions in turn through two-dimensional Haar wavelet transform to decompose into four sub-band components: low-frequency component , horizontal high-frequency component , vertical high-frequency component and diagonal high-frequency component ; applying Laplace filter to high-frequency components to enhance edge details, and further strengthening target features and suppressing background noise through Squeeze-and-Excitation attention and pixel attention; applying Gaussian filter to smooth the background and suppress high-frequency noise, and combining SE attention to adaptively reduce background redundancy and enhance target contrast; according to the formula , the frequency domain decomposition result of the input feature map after convolution and two-dimensional Haar wavelet transform (HWT) is obtained, wherein F represents the final output feature map of the backbone network, i.e. the multi-modal fusion feature encoded by the hybrid backbone structure combining Transformer, Mamba and CNN; f(·) represents the preliminary convolution operation on the final output feature map F to extract basic features, and HWT represents two-dimensional wavelet transform.

[0021] Further, the HR feature map is generated by adaptively selecting key regions, including selecting multi-scale features from the fusion backbone network, wherein the low-level features are first input into the channel recalibration module to strengthen the key texture channel response related to reconstruction; the high-level features are then upsampled to match the spatial resolution of the low-level features; then the two are connected to fuse into a set of spatially aligned multi-scale features; the fused features are processed through two CR modules in series to enhance the inter-channel dependence and suppress redundant background information; wherein in the training stage, the pixel difference between the reconstructed image and the original input is calculated according to the formula, ; wherein S represents the reconstructed image generated by upsampling on the SR branch, and X represents the original input image; the target detection loss is obtained according to the formula, ; wherein , , are the weights of different layers of the three loss functions, and the weights , , adjust the error emphasis between the bounding box coordinates, the bounding box dimensions, and the objectness, non-objectness and classification; the total loss is obtained according to the formula, ; wherein , is a loss weight factor.

[0022] Further, the generating the HR feature map by adaptively selecting the key region further comprises reducing the calculation amount by adaptively selecting the key region, wherein the calculation amount is reduced by formula C to obtain the coarse attention, wherein Q, K and V are query, key and value generated by 1x1 convolution, K and V of the high-level feature perform cross-layer attention calculation on Q of the bottom layer, and the fine attention is calculated and represented as to obtain the fine attention, wherein and are the fine-grained tokens obtained by screening the top k key-value pair indexes with the most information amount according to the global similarity s and mapping to the bottom feature network, and the fine attention is calculated by formula

[0023] ,

[0024] to obtain the final output, wherein represents convolution, represents up-sampling, represents deep convolution.

[0025] Further, the detecting the target frame based on the cross-scale aggregation of the HR feature map comprises: between each layer of the feature pyramid, the coarse-grained feature fusion is realized by the cross-layer attention mechanism, the adjacent low-level feature map is taken as the query by using the higher layer feature map as the key and the value, the cross-layer attention calculation is performed, the coarse attention map is generated, the coarse attention stage is performed in the high-level semantic space, the context modeling capability can be maintained while the attention calculation amount is significantly reduced; subsequently, the key region with large response intensity is screened from the coarse attention result, which is taken as the candidate region of the fine attention operation, the standard self-attention operation is performed in the region, the local details are finely modeled and the feature is enhanced; by the coarse-to-fine attention strategy, the excessive response of the traditional global attention to the non-key background region is effectively avoided, and the attention ability of the model to small targets and edge details is improved.

[0026] In a second aspect, a remote sensing target detection system for low-visibility images comprises:

[0027] A data acquisition module configured to acquire medical images; and acquire multi-modal remote sensing image data.

[0028] A preprocessing module configured to perform defogging enhancement processing on the low-visibility input image.

[0029] A fusion module configured to normalize and splice the RGB and IR images after defogging.

[0030] The encoding module is configured to encode the multi-modal fusion image layer by layer by adopting a hybrid backbone structure fusing a Transformer, a Mamba and a CNN.

[0031] The decomposition module is configured to perform frequency domain decomposition on the backbone output feature based on a two-dimensional wavelet transform.

[0032] The selection module is configured to generate an HR feature map by adaptively selecting a key region.

[0033] The detection module is configured to obtain a detection target frame based on cross-scale aggregation of the HR feature map.

[0034] In a third aspect, the present application provides a computer readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and to execute the remote sensing target detection method for a low-visibility image.

[0035] In a fourth aspect, the present application provides a terminal device, comprising a processor and a computer readable storage medium, the processor being configured to implement each instruction; and the computer readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded by the processor and to execute the remote sensing target detection method for a low-visibility image.

[0036] In summary, the present application has the following beneficial technical effects:

[0037] Improve the target recognizability and robustness under low-visibility conditions: The present application effectively enhances the clarity and contrast of remote sensing images under adverse weather conditions such as fog, rain, etc., reduces the shielding interference of environmental degradation on small target detection, and enhances the stability and adaptability of the model in complex weather scenarios through multi-modal image dehazing enhancement and feature distillation mechanism.

[0038] Reduce the computational complexity in the model inference stage: The Mamba, CNN and Transformer fusion module are used as the backbone, and combined with dynamic convolution and sparse self-attention mechanism, the calculation area can be adaptively adjusted according to the feature importance, key region modeling is realized, redundant calculation overhead is reduced, detection efficiency is improved, and end-to-end deployment is suitable.

[0039] Strengthen the separation modeling ability of target structure and semantic features: By introducing a two-dimensional wavelet transform after feature extraction, the image features are decoupled into high-frequency structure and low-frequency semantic components, and branch modeling is performed, which effectively improves the independent perception ability of the model to edge contour and semantic region, and enhances the comprehensive utilization ability of the detection head to different types of information. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1is a schematic diagram of a remote sensing target detection method for low-visibility images according to embodiment 1 of the present application;

[0041] Figure 2 is a structural diagram of the model according to embodiment 1 of the present application;

[0042] Figure 3 is a columnar diagram of a comparison of core performances of the model according to embodiment 1 of the present application and other models;

[0043] Figure 4 is a columnar diagram of a comparison of calculation performances of the model according to embodiment 1 of the present application and other models. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below with reference to the accompanying drawings.

[0045] Embodiment 1

[0046] With reference to Figure 1 , the remote sensing target detection method for low-visibility images according to the present embodiment comprises:

[0047] S1: Obtain multi-modal remote sensing image data, perform defogging enhancement processing on the low-visibility input image, improve the definition and contrast of the low-visibility image, and reduce the shielding effect of environmental factors such as fog and haze on small targets;

[0048] S2: Normalize the RGB and IR images after defogging to [0, 1], and concatenate them along the channel dimension to form a four-channel image (R, G, B, I), thereby forming a fusion input;

[0049] S3: Use a hybrid backbone structure that combines a fusion Transformer, a Mamba, and a CNN to perform layer-by-layer encoding on the multi-modal fusion image. This structure combines the global modeling capability of the Transformer, the state space modeling advantage of the Mamba, and the fine capture capability of the CNN for local details, while maintaining efficient modeling, and realizing multi-scale feature expression from bottom to top: low-level features focus on texture and edge preservation, and high-level features focus more on semantic relationship and scene structure modeling.

[0050] To enhance its perception of spatial structure changes and small target textures in remote sensing images, a lightweight dynamic convolution module is introduced at each stage of the backbone network, which adaptively adjusts the convolution kernel weights according to the spatial distribution of the input features, thereby achieving more precise context adjustment and response. Through the synergistic effect of state modeling and dynamic convolution, the network can more accurately perceive the feature differences of different scales and regions, effectively improving the target representation ability under low-visibility conditions.

[0051] S4: To further improve the model's ability to decouple structural and semantic information, a two-dimensional wavelet transform is introduced to decompose the backbone output features in the frequency domain. High-frequency components are rich in texture details and edge contours, suitable for structural enhancement; low-frequency components carry scene structure and semantic backbone information, suitable for global understanding.

[0052] S5: In the training phase, the SR branch is enabled, the encoder fuses the backbone multi-layer features, and the decoder uses three layers of deconvolution upsampling to the input image 2 times the size to generate the HR feature map, guiding the backbone network to learn a clearer target representation. In the inference phase, the SR branch is removed, and only the fused backbone features are used for feature extraction;

[0053] S6: The structure-semantic fusion features are aggregated across scales via the detection head. Sparse self-attention is introduced at each layer of the feature pyramid, reducing computational complexity by adaptively selecting key regions and avoiding excessive response to background noise from global attention. Finally, the detection head outputs target bounding box parameters, confidence, and class probability.

[0054] Specifically,

[0055] In S1, the haze enhancement processing is specifically,

[0056] A target detector that performs well on clear image datasets is used as a teacher network to guide the student network to perform feature distillation on haze input images. During feature distillation, the Sobel operator is used to calculate the gradient of the feature map in the horizontal and vertical directions to highlight edges and semantic information in the image and enhance the contrast of the features.

[0057] According to the formula, , , , Convolve the feature map to calculate the edge gradient, where is the horizontal direction gradient convolution kernel, is the vertical direction gradient convolution kernel; is the feature map output by the teacher network at a certain layer, is the feature map output by the student network at the corresponding layer.

[0058] According to the formula, , , the gradient amplitude is obtained.

[0059] By minimizing the difference between the gradient representations of the teacher network and the student network, the student network's ability to perceive target edges is enhanced, helping the student network to extract clearer semantic information in haze images and improving the quality of feature extraction.

[0060] Meanwhile, combined with the degradation weighted response distillation method, the pixel-level difference between the clear image and the foggy image is calculated first to obtain a preliminary degradation image, and then the degradation degree of different regions in the image is quantified by normalizing the preliminary degradation image. Finally, the degradation weighted response distillation method is adopted, and according to the degradation degree, a higher weight is given to the severely degraded region. In the response map distillation process of classification and positioning, the student network pays more attention to the target in the severely degraded region, so as to realize the targeted improvement of the detection ability of the student network in different degraded regions. The final output is the enhanced RGB and IR images, which are used for subsequent fusion processing.

[0061] According to the formula, The pixel difference between the fog image and the clear image is calculated, where D represents the difference map, is the remote sensing image under clear conditions, is the input remote sensing image under low visibility conditions:

[0062] According to the formula, D is normalized to obtain a degradation degree map, which reflects the degradation intensity of each region of the image: it is used in the distillation process of the classification and positioning response map to guide the student network to focus on the severely degraded region.

[0063] According to the formula, , the weighted response distillation loss is calculated. Where C, H, W represent the channel number (Channel), height (Height), and width (Width) of the feature map, and are the activation values of the teacher network and the student network in the cth channel, the hth row, and the wth column of the response feature map.

[0064] According to the formula, , the dehazing enhancement loss function is obtained. Where is the edge distillation loss, which is used to constrain the perceptual consistency of the teacher and student networks in the edge structure; is the degradation weighted response distillation loss, which is used to emphasize the response consistency of the severely degraded region; is the regular detection loss of the student network, including the classification loss and the bounding box regression loss; represents the weight coefficient of the edge distillation loss; represents the weight coefficient of the degradation weighted response distillation loss; represents the weight coefficient of the regular detection loss.

[0065] In S2, the dehazed RGB and IR images are normalized to [0, 1], specifically,

[0066] For the RGB image, it is usually in 8-bit format, and the pixel value range is between 0 and 2550, so according to the formula,

[0067] ,

[0068] The resulting image is a three-channel image with shape (H,W,3) and values in the range 0-1.

[0069] For the infrared image IR, which is usually a single-channel image, its bit depth can be either 8-bit or 16-bit.

[0070] If it is an 8-bit image, the normalization result is obtained according to the formula,

[0071] ,

[0072] The IR normalization result is obtained.

[0073] If it is a 16-bit image, the normalization result is obtained according to the formula,

[0074]

[0075] The IR normalization result is obtained.

[0076] In S2, the four-channel image (R, G, B, I) is obtained by concatenating along the channel dimension, specifically,

[0077] According to the formula, the fused image is obtained,

[0078] wherein, C is the number of channels, H and W are the height and width of the image, {R, G, B} and {I} represent the RGB image and the IR image, and Concat() represents the concatenation operation along the channel axis.

[0079] The IR image needs to be expanded by one channel dimension, according to the formula,

[0080] ,

[0081] so that its shape is converted from (H,W) to (H,W,1), and then concatenated with the RGB image by channel to obtain the fused image

[0082] After concatenation, the shape of the image is (H,W,4), and the channel order is usually R, G, B, IR.

[0083] In S2, it is normalized to [0,1], specifically,

[0084] In order to accelerate the training process and improve the efficiency of the SR module, the original image is first down-sampled to 1 / n of the original image to complete the SR module and accelerate the training process.

[0085] According to the formula, Downsampled image.

[0086] where D() denotes the operation of downsampling using bilinear interpolation, X is the original image, and n is the downsampling factor. Bilinear interpolation generates each pixel value of the new image by calculating the weighted average of the surrounding four pixels, achieving a smooth downsizing effect. After downsampling, the image is processed through a multi-layer convolutional network to extract and encode features of different scales, forming multi-level feature maps from low to high.

[0087] S3. Multimodal fusion

[0088] A hybrid backbone structure that integrates Transformer, Mamba, and CNN is adopted to encode the multi-modal fusion image layer by layer. Specifically,

[0089] In the high-resolution stage, a lightweight CNN module is used to extract local information such as edges and textures, adapting to small target detection tasks. The input image is preliminarily processed through convolutional layers to extract low-level features such as edges, corners, and other detailed information. Multiple convolutional layers are used to extract features layer by layer to ensure that local texture and edge information are fully captured. Convolution operations with a stride of 1 are used to ensure that the spatial resolution of the feature map does not drop too much, so as to capture the detailed information of small targets. Batch normalization and ReLU activation functions are used to improve feature expression ability and speed up training.

[0090] According to the formula, , the high-resolution stage output feature map is obtained.

[0091] where W is the convolution kernel, b is the bias, is the input feature map.

[0092] In the medium-resolution stage, a state space modeling module based on Mamba is introduced, which uses a multi-directional scanning mechanism to capture the global structure of rotating and deforming targets. The state space model is used to model the spatio-temporal features of the image, which is particularly suitable for handling complex situations such as target rotation and deformation. Through convolution operations in different directions, the target is scanned in all directions to capture its global structure. The image is processed using a rotation-invariant convolution kernel to ensure that the features of the target at different angles can be effectively captured.

[0093] According to the formula, , the medium-resolution stage output feature map is obtained.

[0094] where is the parameter of the state space modeling, is the input feature map.

[0095] In the low-resolution stage, a Transformer module based on multi-scale sliding window attention is used to achieve semantic modeling and region focus for large-scale targets (such as airports and woodlands). A self-attention mechanism is used to weight different regions in the image, with particular attention paid to large-scale target regions. A multi-scale sliding window approach is employed to divide the image into regions, ensuring comprehensive perception of targets of different sizes. Based on the features within each window, their similarity to global features is calculated. By weighted combination of features from different regions, global semantic modeling is achieved.

[0096] According to the formula, This yields the weighted output feature map after the low-resolution stage.

[0097] Here, Attention represents the self-attention operation, with the input being three mappings of the same feature map: query, key, and value, and the output being the weighted feature map.

[0098] Features from each stage are fused across layers and at multiple scales to form a complete feature pyramid. In the feature pyramid network, low-resolution features are fused with high-resolution features to maintain a balance between detailed and semantic information. During feature fusion, cross-scale connections are used to ensure effective representation of multi-level image information at multiple scales. Finally, the output multi-scale feature map is processed by a detection head to predict target bounding box parameters, estimate confidence scores, and output target class probabilities.

[0099] In S3, dynamic convolution can further refine the feature extraction process. By adaptively adjusting the convolution kernel weights, personalized processing can be achieved at different regions and scales, enhancing the model's sensitivity to local image features. Specifically,

[0100] The dynamic convolution module adaptively generates convolution kernel weights based on the input feature content, enabling personalized processing of different image regions.

[0101] According to the formula, ,

[0102] In the formula, These are the weight coefficients generated based on the input feature x. represents the corresponding dynamic convolution kernel, and * represents the convolution operation.

[0103] In S4, the two-dimensional wavelet transform is specifically as follows:

[0104] The feature map output by the backbone, according to the formula, Preliminary features are extracted through convolution operations. Here, F represents the original feature map output by the backbone, f(·) represents the convolution operation, W is the convolution kernel weight, b is the bias term, and * represents the convolution operation.

[0105] Then, the low-pass filter (LPF) and high-pass filter (HPF) are applied in the horizontal and vertical directions in turn by two-dimensional Haar wavelet transform (HWT), and the input feature map is decomposed into four sub-band components: low-frequency component (L) ), horizontal high-frequency component (H ), vertical high-frequency component (V ), and diagonal high-frequency component (D ).

[0106] According to the formula, , the frequency domain decomposition result of the input feature map after convolution and two-dimensional Haar wavelet transform (HWT) is obtained. Wherein, F represents the final output feature map of the backbone network, that is, the multi-modal fusion feature encoded by the hybrid backbone structure of the fusion Transformer, Mamba and CNN; f(·) represents the preliminary convolution operation on the final output feature map F to extract the basic feature. HWT represents two-dimensional wavelet transform. Specifically,

[0107] According to the formula, , the low-frequency component is obtained, which effectively retains the main structural features of the image.

[0108] The high-pass filter is applied in the horizontal direction, and the low-pass filter is applied in the vertical direction. Then, the horizontal high-frequency component is obtained by 2 times down-sampling. Then, the low-pass filter is used in the horizontal direction, and the high-pass filter is used in the vertical direction. Then, the vertical high-frequency component is obtained by 2 times down-sampling. The high-pass filter is applied in the horizontal and vertical directions, and then 2 times down-sampling is performed to obtain the diagonal high-frequency component.

[0109] The low-pass filter is mainly used to retain the low-frequency components in the image, which correspond to the main structural features and background information of the image. The low-pass filter processes the input feature map according to the formula in the horizontal and vertical directions. Wherein D(u,v) represents the radial distance to the spectral center in the frequency domain, is the cut-off frequency.

[0110] The high-pass filter is used to extract the high-frequency components in the image, which contain texture, edge and other detailed information, which is crucial for target detection. The high-pass filter processes the input feature map according to the formula .

[0111] In the S5, the SR branch is enabled, specifically,

[0112] The SR branch is used to guide the backbone network to learn a clearer target representation in the training stage, thereby improving the resolution and detection accuracy of small targets in low-visibility remote sensing images. The branch selects multi-scale features from the fusion backbone network, including a low-level feature and a high-level feature.

[0113] In the encoder, low-level features are first input into a channel re-calibration module (CR module) to enhance the key texture channel responses related to reconstruction. High-level features are upsampled to match the spatial resolution of low-level features. Then, the two are concatenated to fuse into a set of spatially aligned multi-scale features. The fused features are further processed by two serial CR modules to enhance the inter-channel dependency and suppress redundant background information, thus improving the structural integrity and detail fidelity of the super-resolution representation.

[0114] In the decoder, the fused feature map is upsampled by three deconvolution layers to reconstruct a high-resolution feature map with a spatial size twice that of the input image. This high-resolution output serves as the image reconstruction result, which is pixel-level aligned with the original input image.

[0115] In the training phase, the pixel difference between the reconstructed image and the original input is calculated according to the formula,

[0116] The target detection loss is obtained according to the formula, , , are the weights of different layers of the three loss functions, and the weights , , adjust the error emphasis between box coordinates, box dimensions, and objectness, non-objectness, and classification.

[0117] The total loss is obtained according to the formula, , is the loss weight factor.

[0118] In S6, sparse self-attention is introduced at each layer of the feature pyramid. By adaptively selecting key regions, the computational complexity is reduced. Specifically,

[0119] According to the formula, C , the coarse attention is obtained, where Q, K, and V are the query, key, and value generated by 1x1 convolution. K and V of high-level features perform cross-layer attention calculation on Q of the bottom layer.

[0120] The fine attention is obtained by the formula, and are the fine-grained tokens obtained by screening the top k most information-containing key-value pair indexes according to the global similarity s and mapping them into the bottom feature network.

[0121] The final output is obtained by the formula, ​​​​

[0122] to obtain the final output.

[0123] Experimental verification:

[0124] To systematically verify the target detection capability of the method of the present application in low-visibility multi-modal remote sensing images, a self-defined multi-modal low-visibility remote sensing small target detection data set is constructed based on the VEIDAI remote sensing data set, which contains two types of modalities: ① RGB low-visibility modality: based on the original visible light image of VEIDAI, low-visibility images are synthesized using an atmospheric scattering model to simulate real foggy weather conditions; ② infrared modality: using the original infrared imaging image, which has strong ability to penetrate fog. The data set contains a total of 1246 images, with a resolution of 1024x1024 or 512x512, covering typical remote sensing scenes such as grassland, highway, mountain and city. The images contain small targets of different scales.

[0125] The MPE-YOLO, SuperYOLO methods are set up for performance comparison with the present application. The complete framework proposed by the method of the present application integrates Transformer, CNN and Mamba model, and increases the corresponding self-attention mechanism.

[0126] All methods are evaluated under the same training set and test set conditions. The evaluation indicators include: ① precision: the number of samples that actually belong to a certain class among the samples predicted by the model as belonging to that class, divided by the number of samples predicted as belonging to that class. It is an indicator for a specific class, reflecting the prediction accuracy of the model for that class; ② mAP@0.5: represents the value of mAP when the IOU threshold is 0.5; ③ parameter quantity (Prarams): the parameter quantity of the model, which is also used to measure the complexity of the algorithm / model; ④ floating point number (GFLOPs): the number of floating point operations per second.

[0127] Table 1 Comparison of performance data of different methods

[0128] mAP@0.5 Precision Prarams GFLOPs MPE-YOLO 0.5366 0.4023 4.2 17.3 SuperYOLO 0.7025 0.6735 7.7 56.2 The present invention 0.8511 0.8203 8.5 70.3

[0129] Since MPE-YOLO cannot directly process low-visibility remote sensing images and can only process RGB single-modal images, the performance is obviously poor when processing the self-defined data set, and the parameter and calculation amount are very small. Although SuperYOLO cannot directly process low-visibility images, it can be compensated by IR infrared images, which will improve the accuracy to a certain extent, but it still has deficiencies in extreme low-visibility environments. The present application further deepens multi-modal fusion and target detection, and improves the detection accuracy, but the model parameters and calculation complexity are higher.

[0130] Example 2

[0131] The embodiment provides a remote sensing target detection system for a low-visibility image, comprising:

[0132] The data acquisition module is configured to:

[0133] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions are suitable for being loaded by a processor of a terminal device and performing the remote sensing target detection method for a low-visibility image.

[0134] A terminal device, comprising a processor and a computer readable storage medium, the processor is used for implementing instructions; the computer readable storage medium is used for storing a plurality of instructions, the instructions are suitable for being loaded by the processor and performing the remote sensing target detection method for a low-visibility image.

[0135] The above are preferred embodiments of the present application, not limited to the protection scope of the present application, therefore: all equivalent changes made according to the structure, shape, principle of the present application should be covered in the protection scope of the present application.

Claims

1. A method for remote sensing target detection for low-visibility images, characterized in that, The method comprises the following steps: obtaining multi-modal remote sensing image data; performing defogging enhancement processing on the low-visibility input image; normalizing the defogged RGB and IR images and then performing splicing fusion; using a hybrid backbone structure that fuses Transformer, Mamba and CNN to perform layer-by-layer encoding on the multi-modal fusion image; performing frequency domain decomposition on the backbone output features based on two-dimensional wavelet transform; generating an HR feature map by adaptively selecting key regions; obtaining a detection target frame based on cross-scale aggregation of the HR feature map; the hybrid backbone structure that fuses Transformer, Mamba and CNN to perform layer-by-layer encoding on the multi-modal fusion image comprises, in the high-resolution stage, using a lightweight CNN module to extract edge and texture local information to adapt to the small target detection task; in the medium-resolution stage, introducing a state space modeling module based on Mamba and using a multi-direction scanning mechanism to capture the global structure of rotating and deforming targets; in the low-resolution stage, combining a Transformer module based on multi-scale sliding window attention to realize semantic modeling and regional attention of large-scale targets; the features of each stage are connected across layers and fused in multiple scales to form a complete feature pyramid, and multi-scale feature maps are output; wherein a dynamic convolution module adaptively generates convolution kernel weights according to the content of the input features to realize personalized processing of different image regions, and is represented as: ， wherein is a weight coefficient generated according to an input feature x, is a corresponding dynamic convolution kernel, and * is a convolution operation; The HR feature map generated by adaptive selection of key regions comprises multi-scale features selected from the fusion backbone network, wherein low-level features are first input into a channel recalibration module to strengthen key texture channel responses related to reconstruction; high-level features are subjected to up-sampling operation to match the spatial resolution of the low-level features; then the two are connected to fuse into a set of spatially aligned multi-scale features; the fused features are processed by two series-connected CR modules to enhance inter-channel dependence and suppress redundant background information; wherein in the training stage, the pixel difference between the reconstructed image and the original input is calculated according to the formula, the pixel difference between the reconstructed image and the original input is calculated; wherein S represents the reconstructed image generated by up-sampling of the SR branch, and X represents the original input image; according to the formula, ， a target detection loss is obtained; wherein , , is the weight of different layers of the three loss functions, the weight , , , adjusts the error emphasis among the box coordinates, the box dimensions, and the objectiveness, non-objectiveness and classification; according to the formula, a total loss is obtained, wherein , is a loss weight factor; Also included is reducing the amount of calculation by adaptively selecting key areas, wherein the formula C Coarse attention C is obtained , wherein Q, K, and V are queries, keys, and values generated by 1x1 convolution, T is transposition, dk is the dimension of the key vector K, and the K and V of the high-level feature perform cross-layer attention calculation on the bottom layer Q, and fine attention is calculated , denoted as , after obtaining the fine attention, wherein and are fine-grained tokens obtained by screening the top k most information-containing key-value pair indexes according to the global similarity s and mapping them into the bottom feature network, and the fine attention is obtained by the formula ， a final output is obtained, wherein denotes a convolution, denotes an up-sampling, denotes a depthwise convolution.

2. The method for remote sensing target detection for low-visibility images according to claim 1, characterized in that, the defogging enhancement processing on the low-visibility input image comprises using a teacher network pre-trained on clear multi-modal images to guide a student network to perform feature distillation on the foggy input image, and through gradient alignment and degradation response constraint, the student network learns the robust representation ability under foggy conditions, thereby enhancing the clarity and contrast of the image and reducing the shielding effect of fog on small target detection; finally, the output is the enhanced RGB and IR images, which are used for subsequent fusion processing; wherein the pixel-level difference between clear images and foggy images is calculated to obtain a preliminary degradation map, which is then normalized to quantify the degradation degree of different regions in the image; finally, a degradation weighted response distillation method is used to give higher weights to severely degraded regions according to the degradation degree, so that the student network pays more attention to the targets in the severely degraded regions during the response map distillation process of classification and positioning, thereby realizing targeted improvement of the detection ability of the student network in different degraded regions.

3. The method for remote sensing target detection for low-visibility images according to claim 2, characterized in that, The degradation weighted response distillation method comprises calculating pixel differences between the fog image and the clear image according to a formula D represents a difference map, is a remote sensing image under a clear condition, is an input remote sensing image under a low-visibility condition; according to a formula D is normalized, wherein max(D) represents a maximum pixel difference value in the difference map D, and min(D) represents a minimum pixel difference value in the difference map D; a degradation degree map obtained after normalization reflects the degradation intensity of each region of the image and is used in the distillation process of the classification and positioning response map to guide the student network to focus on the seriously degraded region; according to a formula, ， calculating the weighted response distillation loss; where C, H, W represent the channel number, height, and width of the feature map, respectively, and are the activation values of the teacher network and the student network at the c-th channel, h-th row, and w-th column of the response feature map; according to the formula, ， The defogging enhancement loss function is obtained, wherein is an edge distillation loss for constraining the perception consistency of the teacher and student networks on edge structure; is a degradation weighted response distillation loss for emphasizing the response consistency of the severe degradation area; is a student network regular detection loss, including a classification loss and a bounding box regression loss; represents a weight coefficient of the edge distillation loss; represents a weight coefficient of the degradation weighted response distillation loss; represents a weight coefficient of the regular detection loss.

4. The method for remote sensing target detection for low-visibility images according to claim 3, characterized in that, The RGB and IR images after defogging are normalized and spliced and fused, including being spliced into a four-channel image (R, G, B, I) along a channel dimension, wherein according to a formula a fused image is obtained, wherein, C is a channel number, H and W are a height and a width of the image, {R, G, B} and {I} represent the RGB image and the IR image, and Concat() represents a connection operation along a channel axis; the RGB and IR images after defogging are normalized to [0, 1], specifically, X is twice sampled to 1 / n of the original image to complete an SR module and accelerate a training process, according to a formula a sampling image is obtained, wherein, D() represents an n-time downsampling operation using bilinear interpolation, and then the twice sampling result is sent to a backbone to generate multi-level features.

5. The method for remote sensing target detection for low-visibility images according to claim 4, characterized in that, The backbone output feature is decomposed in the frequency domain based on the two-dimensional wavelet transform, including that the backbone output feature map is firstly subjected to a convolution operation to extract preliminary features, then a low-pass filter LPF and a high-pass filter HPF are sequentially applied along horizontal and vertical directions by two-dimensional Haar wavelet transform to decompose the feature map into four sub-band components: a low-frequency component , a horizontal high-frequency component , a vertical high-frequency component , and a diagonal high-frequency component ; a Laplace filter is applied to the high-frequency component to enhance edge details, and then a squeeze-and-excitation attention and a pixel attention are further applied to strengthen target features and suppress background noise; a Gaussian filter is applied to smooth the background and suppress high-frequency noise, and the SE attention is combined to adaptively reduce background redundancy and enhance target contrast; according to the formula , a frequency domain decomposition result of the input feature map after convolution and two-dimensional Haar wavelet transform HWT is obtained, wherein F represents the final output feature map of the backbone network, i.e., a multi-modal fusion feature encoded by a hybrid backbone structure combining a fusion Transformer, a Mamba and a CNN; f(·) represents a preliminary convolution operation on the final output feature map F to extract basic features, and HWT represents a two-dimensional wavelet transform.

6. The method for remote sensing target detection for low-visibility images according to claim 5, characterized in that, the cross-scale aggregation of the HR feature map to obtain the detection target frame comprises, between the layers of the feature pyramid, realizing semantic-guided coarse-grained feature fusion through a cross-layer attention mechanism, using higher layer feature maps as keys and values and adjacent lower layer feature maps as queries to perform cross-layer attention calculation and generate a coarse attention map; the coarse attention map generation stage is performed in the high-level semantic space, which can significantly reduce the attention calculation amount while maintaining the context modeling capability. Subsequently, the key areas with stronger response intensity are screened from the coarse attention map results as candidate areas for fine attention operation. Standard self-attention operation is performed in the candidate areas to model and enhance local details. Through the coarse-to-fine attention strategy, the excessive response of the traditional global attention to non-key background areas is effectively avoided, and the attention ability of the model to small targets and edge details is improved.

7. A remote sensing target detection system for low-visibility images, which performs a remote sensing target detection method for low-visibility images as claimed in claim 1, characterized in that, The method comprises the following steps: a data acquisition module configured to acquire medical images; acquire multi-modal remote sensing image data; a preprocessing module configured to perform defogging enhancement processing on the low-visibility input image; a fusion module configured to normalize and splice the defogged RGB and IR images after fusion; an encoding module configured to use a hybrid backbone structure of fusion Transformer, Mamba and CNN to perform layer-by-layer encoding on the multi-modal fusion image; a decomposition module configured to perform frequency domain decomposition on the backbone output features based on two-dimensional wavelet transform; a selection module configured to generate an HR feature map by adaptively selecting key areas; a detection module configured to obtain a detection target frame based on cross-scale aggregation of the HR feature map.

Images