A cross-modal infrared small target detection method based on degradation memory compensation

Abstract

This invention discloses a cross-modal infrared small target detection method based on degradation memory compensation, belonging to the field of target detection technology. The method includes the following steps: preparing a dataset, constructing a degradation memory compensation network, training the degradation memory compensation network, performing cross-modal image hybrid masking, training a small target detection network, testing and evaluating the model, and outputting detection results and performance metrics. This invention constructs a degradation memory compensation network consisting of a degradation estimation module and a memory compensation network, and designs a training pipeline for the degradation memory compensation network; proposes a novel cross-modal image hybrid masking method; designs a cross-modal small target detection network based on a mask autoencoder, and proposes a novel masking region loss function. This achieves cross-modal small target detection in harsh environments, improves the detection accuracy and speed of infrared small targets, and solves the problems of difficult and controllable learning in cross-modal tasks.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a cross-modal infrared small target detection method based on degradation memory compensation. Background Technology

[0002] Infrared small target detection is an important research direction in the field of image processing. It aims to obtain the location information of small targets in a wide range of backgrounds by using data-driven and deep representation principles. It is widely used in tasks such as military surveillance, UAV navigation and target recognition in complex scenarios. However, existing technologies are usually limited to a single modality (such as infrared modality), and the few cross-modal detection technologies that have emerged have the following obvious defects: (1) The purpose of cross-modal image information fusion is often not clear enough, and the learning process of the network and the cross-modal interpretability are poor, making it impossible to effectively control the impact of the degradation intensity between modalities on the detection accuracy. (2) Due to the "insufficient supervision signal for end-to-end training", existing technologies often experience a sharp drop in cross-modal detection accuracy in real environments. (3) There is a lack of degradation compensation mechanism for cross-modal detection. When the distribution of targets differs significantly between different modalities, existing models cannot give full play to the advantages of modal complementarity. Summary of the Invention

[0003] The technical solution of this invention to solve the above-mentioned technical problems is to provide a cross-modal infrared small target detection method based on degradation memory compensation, comprising the following steps:

[0004] S1: Prepare the datasets, including dataset one for training and dataset two for testing. Dataset one contains registered infrared and visible light image pairs, and dataset two contains a self-made registration test set.

[0005] S2: Construct a degradation memory compensation network, including a degradation estimation module (DEM) and a memory compensation network; the degradation estimation module estimates the blur degradation score (Pblur), noise degradation score (Pnoise), and visibility degradation score (Gvisible) of infrared and visible light images through six metric spaces, and outputs them in the form of heatmaps; the memory compensation network generates weight heatmaps and weight vectors through fully connected layers (FC), softmax layers, and hist layers;

[0006] S3: Train the degradation memory compensation network, generate degradation samples for the bimodal control group (gc) and the consistency enhancement group (gr) through the degradation sample generation model, and optimize the network using the adaptive ranking loss function;

[0007] S4: Cross-modal image region masking processing, which divides the registered infrared and visible light images into image blocks, randomly masks some image blocks and replaces them with mask signals;

[0008] S5: Cross-modal image mixing mask processing, replacing the masked infrared image block with the corresponding visible light image block, and replacing the masked visible light image block with the corresponding infrared image block, generating a mixed mask image pair;

[0009] S6: Construct a small target detection network, including a multi-level Swing Transformer encoder, a fusion strategy, and a decoder; the multi-level Swing Transformer encoder uses the weight heatmap and weight vector of the memory compensation network to modulate and compensate cross-modal features.

[0010] S7: Train the small target detection network using an end-to-end framework, and optimize the network by combining masking region loss and soft-IoU loss;

[0011] S8: Test and evaluate the model, and output detection results and performance metrics.

[0012] Furthermore, in step S1, the first dataset is an Anti-UAV dataset, and the second dataset contains infrared and visible light image pairs of various scenes, all of which have undergone fine-grained registration.

[0013] Further, in step S2, the degradation estimation module (DEM) includes: a fuzzy degradation subnetwork, a noise degradation subnetwork, and a visibility degradation subnetwork. Each subnetwork is composed of a spatial and channel attention module (CBAM) and outputs a degradation score heatmap using a sigmoid function.

[0014] The noise degradation subnetwork and the fuzzy degradation subnetwork are generated based on a probabilistic degradation model, and the generation is expressed by the following formula:

[0015]

[0016] Where n and k represent the size of the noise kernel and the fuzzy kernel, respectively, z n and z k M represents noise and fuzzy kernel, respectively, both following a normal distribution; n and M k These represent the noise and fuzzy generation functions, respectively.

[0017] The visibility degradation subnetwork is composed of brightness and contrast degradation functions;

[0018] The memory compensation network consists of a fully connected layer (FC), a Softmax layer, and a hist layer connected to the end of each DEM.

[0019] Further, in step S3, the adaptation ranking loss function includes:

[0020] For the Margin Ranking Loss of the bimodal control group, the difference between the constraint fuzziness degradation and noise degradation scores is expressed as follows:

[0021]

[0022] For the Margin Ranking Loss of the consistency enhancement group, the score difference between non-degenerate samples and severely degenerate samples is constrained; expressed as:

[0023]

[0024] Furthermore, in step S4, the infrared and visible light images are divided into 3×4 image blocks, and 6 of these image blocks are randomly masked and replaced with mask signals.

[0025] Furthermore, in S5, the infrared mask image after hybrid mask processing is represented as follows:

[0026]

[0027] The visible light mask image is represented as follows:

[0028]

[0029] Where {MASK} represents the mask signal, This represents the image patch at (i,j) in the original visible light image. This represents the image block at (i,j) in the original infrared image.

[0030] Further, in step S6, the small target detection network includes:

[0031] Encoder: 5-level Patch Merging and 5-level Swing Transformer blocks, with each level's output features fused after being modulated by memory-compensated weights;

[0032] Decoder: 5-level Patch Expanding and dual ResBlock blocks for reconstructing the detected image with the target bounding box.

[0033] Furthermore, in step S7, the masking region loss function is:

[0034]

[0035] in, This represents the total number of pixels in the masked area, ensuring a uniform distribution of loss; I f,i I represents the pixel value at the i-th masking position of the reconstructed image; ir / vi,i It is the actual pixel value of the original image at the i-th masking position.

[0036] Furthermore, in step S7,

[0037] The Soft-IoU loss function is expressed as:

[0038]

[0039] Among them, P i,j G represents the predicted probability at position (i,j); i,j ε represents the true label at pixel (i,j); ε represents a smoothing term to prevent division by zero errors, typically a small constant (1e-6) used to ensure numerical stability.

[0040] Furthermore, in step S8, the performance metrics include detection rate, false alarm rate, recall rate, IoU, and F1 score.

[0041] Compared with existing technologies, this invention provides a cross-modal infrared small target detection method based on degradation memory compensation, which has the following beneficial effects:

[0042] 1. This invention innovatively combines metric learning with a hybrid masking strategy, cleverly utilizes image degradation modeling mechanisms and metric space mapping methods, and introduces masking strategies and multi-level autoencoders for target detection tasks, thus developing a novel cross-modal small target detection framework.

[0043] 2. This invention designs a controllable real degradation estimation module (DEM) based on metric learning, which introduces metric learning into cross-modal infrared small target detection tasks for the first time. This invention also proposes a complete metric degradation sample generation model, which trains the DEM to estimate the real degradation score by learning the similarity and difference between samples.

[0044] 3. This invention proposes a multi-level Vit autoencoder cross-modal target detection network based on hybrid masks. First, the traditional direct masking strategy is improved by constructing a new infrared and visible light hybrid mask image pair for training a robust detection model. Then, a multi-level Vit autoencoder detection network is designed, using the estimated weights output by the memory compensation network to guide the network in reconstructing the detection image.

[0045] 4. In response to the aforementioned hybrid masking strategy, this invention proposes a masking area loss (L...) method. mask By constraining the similarity between the reconstructed image and the masked areas in the original infrared and visible light images, and adapting the Soft-IoU loss, the model learns better information representation and accurate cross-modal infrared small detection capabilities. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a cross-modal infrared small target detection method based on degradation memory compensation according to the present invention.

[0048] Figure 2 This is a schematic diagram illustrating the working principle of a cross-modal infrared small target detection method based on degradation memory compensation according to the present invention.

[0049] Figure 3 This is a schematic diagram of the degradation estimation module (DEM) of the present invention;

[0050] Figure 4 This is a structural diagram of the degradation compensation network of the present invention;

[0051] Figure 5 This is a qualitative comparison diagram of the cross-modal small target detection method of the present invention with existing methods;

[0052] Figure 6 This is a schematic diagram illustrating a quantitative comparison between the cross-modal small target detection method of the present invention and existing methods. Detailed Implementation

[0053] This invention proposes a cross-modal infrared small target detection method based on degradation memory compensation, aiming to design a cross-modal infrared small target detection method based on degradation memory compensation.

[0054] The cross-modal infrared small target detection method based on degradation memory compensation proposed in this invention will be described below in specific embodiments:

[0055] Example 1:

[0056] A cross-modal infrared small target detection method based on degradation memory compensation, such as Figure 1 As shown, it includes the following steps:

[0057] S1: Prepare the datasets, including dataset one for training and dataset two for testing. Dataset one contains registered infrared and visible light image pairs, and dataset two contains a self-made registration test set.

[0058] Specifically, the first prepared dataset is the Anti-UAV dataset, used to train the degradation compensation network and the small target detection network. In this embodiment, 15,700 pairs of infrared and visible light image pairs from the Anti-UAV dataset were selected. The selection principle was to ensure the richness of the image background and remove overly similar background image pairs. The second prepared dataset is a self-made infrared and visible light image dataset, used to test the final small target detection model. In this embodiment, the self-made dataset two contains a total of 463 pairs of infrared and visible light image pairs for various scenes such as ground-to-air, air-to-air, buildings, sky, and trees. All image pairs have achieved fine-grained registration.

[0059] S2: Construct a degradation memory compensation network, including a degradation estimation module (DEM) and a memory compensation network; the degradation estimation module estimates the blur degradation score (Pblur), noise degradation score (Pnoise), and visibility degradation score (Gvisible) of infrared and visible light images through six metric spaces, and outputs them in the form of heatmaps; the memory compensation network generates weight heatmaps and weight vectors through fully connected layers (FC), softmax layers, and hist layers;

[0060] Specifically, the degradation memory compensation network includes a degradation estimation module (DEM) and a memory compensation network. The bimodal dataset prepared in step S1 is input into the DEM to obtain two sets of degradation scores. The two sets of degradation scores are then input into the compensation network to generate pseudo-supervision signals (including a set of weight heatmaps and weight vectors).

[0061] like Figure 3 As shown, the input to the degradation estimation module (DEM) is two sets of degradation samples (gr) obtained from the degradation sample generation model: a bimodal control group degradation sample (gc) and a consistency enhancement group degradation sample (gr). Six metric spaces are constructed for these two sets of degradation samples, corresponding to the metric spaces of the infrared and visible light image sub-networks: blur degradation (Pblur), noise degradation (Pnoise), and visibility degradation (Gvisible). Each metric space sub-network consists of one or more spatial and channel attention modules (CBAM) and uses a sigmoid activation function as its output to convert the unquantifiable degradation intensity into a degradation score. Specifically, each set of infrared and visible light images has three prediction branches that predict the Pnoise score, Pblur score, and Gvisible score, which are output as heatmaps. The six constructed metric spaces, along with noise degradation, blur degradation, and visibility degradation, jointly estimate the degree of degradation.

[0062] The noise degradation subnetwork and the fuzzy degradation subnetwork are generated based on a probabilistic degradation model, and the generation is expressed by the following formula:

[0063]

[0064] Where n and k represent the size of the noise kernel and the fuzzy kernel, respectively, z n and z k M represents noise and fuzzy kernel, respectively, both following a normal distribution; n and M k These represent the noise and fuzzy generation functions, respectively.

[0065] The visibility degradation subnetwork consists of brightness and contrast degradation functions;

[0066] Memory compensation networks, such as Figure 2 As shown in b, the memory compensation network consists of a fully connected layer (FC), a softmax layer, and a hist layer connected to the end of each DEM. Each FC layer takes a set of degradation score heatmaps output from the DEM as input, generates a new weight heatmap through the softmax layer, and then generates a weight vector through the hist layer. These two weights jointly guide the compensation as follows: Figure 2 (d) The training process of the fusion network.

[0067] S3: Train the degradation memory compensation network, generate degradation samples for the bimodal control group (gc) and the consistency enhancement group (gr) through the degradation sample generation model, and optimize the network using the adaptive ranking loss function;

[0068] Degenerative memory compensation networks, such as Figure 4 As shown, the degraded sample generation model, the adaptive ranking loss function, and the network training are included. The memory compensation network model is trained using the bimodal dataset prepared by S1, the DEM module, and the metric learning method. The trained memory compensation model can output pseudo-supervisory signals (including a set of weight heatmaps and weight vectors).

[0069] Training for the degenerative memory compensation network includes:

[0070] First, in order to train a highly robust metric space, this embodiment generates two sets of degenerate samples: a bimodal control group (gc) and a consistency enhancement group (gr).

[0071] Then, in order to simulate the noise degradation of infrared images due to insufficient light source or extreme temperature conditions, and the blurring degradation of visible light images due to haze or low contrast conditions, and the degradation factors such as low-temperature boundary blurring of infrared image targets and overexposure or underexposure of visible light images caused by brightness / contrast degradation; this embodiment uses g c Introducing two combined degradation modes, O:σ1 and U:σ2; in g r Introducing H:σ3 as a degradation mode, where σ i Representing the degeneration factor of the combination, respectively generating and Three sets of degraded samples, each set containing several degraded images after infrared and visible light degradation; the acquisition of degraded samples can be described by the formula:

[0072]

[0073] σ3=d no ||d max ;

[0074] in, Compare It exhibits more severe O:σ1 degradation, Compare It exhibits more severe O:σ1 degradation; This pattern also applies; Indicates no degenerate samples. This indicates samples that have suffered the most severe degradation due to the superposition of Pnoise, Pblur, and Gvisible.

[0075] Subsequently, the two sets of degraded samples were input into the DEM to generate degradation scores. This process can be represented as follows:

[0076]

[0077] Among them, f RDEM (.) represents the functional representation of the DEM module.

[0078] Finally, under the influence of the loss function, each group of degradation scores continuously and accurately estimates the most precise degradation score (degree of degradation) for each degradation sample.

[0079] The adapted ranking loss function can be expressed in the following three ways:

[0080] To distinguish between samples with different degrees of degradation, three metric spaces are trained. This embodiment designs a multi-directional MarginRanking Loss as the loss function. This loss function widens the gap between positive and negative samples, constraining a larger distance between severely degraded negative samples (after passing H:σ3) and non-degraded positive samples, and a larger distance between hierarchically degraded samples (after passing O:σ1 and U:σ2). Margin Ranking Loss can be defined as:

[0081]

[0082] in, κ The boundary parameter constrains the distance between two samples; N represents the number of training samples; and This invention represents the predicted score; when using Margin Ranking Loss, it will... and The corresponding true value s i Ranked in s j Therefore, let ξ = 1, and thus, in a certain metric space L MR It can be simplified to:

[0083]

[0084] Where →i represents the i-th training sample. In this embodiment, there are three types of Margin Ranking Loss losses. The first is the bimodal control group g. c The loss for the O:σ1 metric space is expressed as:

[0085]

[0086] The second type is a dual-modal control group g c The loss for the U:σ² metric space is expressed as:

[0087]

[0088] The third type is the consistency reinforcement group g. r The loss function is defined for the H:σ3 metric space. For non-degenerate samples, the loss function should restrict the network from making any transformations or modifications to these samples. For samples that have suffered severe degradation, the network needs a stronger recovery operation. It is defined as:

[0089]

[0090] in, This represents the l2 norm.

[0091] S4: Cross-modal image region masking processing, which divides the registered infrared and visible light images into image blocks, randomly masks some image blocks and replaces them with mask signals;

[0092] Specifically, cross-modal image region masking processing includes cutting bimodal image blocks and randomly masking bimodal image blocks. The infrared and visible light images are divided into 3×4 image blocks respectively, and 6 of the image blocks are randomly masked and replaced with mask signals.

[0093] S5: Cross-modal image mixing mask processing, replacing the masked infrared image block with the corresponding visible light image block, and replacing the masked visible light image block with the corresponding infrared image block, generating a mixed mask image pair;

[0094] Specifically, cross-modal image mixing masking processing: the infrared image block that was blocked in step S4 is replaced by the image block at the corresponding position of the original visible light image; the visible light image block that was blocked in step S4 is replaced by the image block at the corresponding position of the original infrared image; a new dual-modal image pair after mixing masking processing is obtained.

[0095] Existing multimodal feature fusion methods often perform concatenation operations directly in the last or initial layer of the network, which inevitably weakens the powerful feature representation ability of the neural network. In contrast, the cross-modal masking method designed in this embodiment masks and mixes partial information from multiple modalities. The network is forced to learn how to reconstruct complementary information between infrared and visible light images even with incomplete information, solving the imbalance problem between different modal image information and improving the detection accuracy of small targets. Figure 2 (c) shows a schematic diagram of the cross-modal image region masking processing and hybrid mask principle designed in this embodiment.

[0096] The cross-modal image region masking process, at the input end, preprocesses a set of infrared and visible light images and registers them to the same size (640×480), denoted as I. ir and I vi The image was then divided into 3×4 image blocks. and Where H = 640 and W = 480; unlike traditional masking mechanisms, a random image block masking method is designed; randomly selected from infrared image I... IR Six subset image patches were selected. To perform masking, a mask signal [MASK] is used instead; then, random selection is made from the infrared image I. VI Six subset image patches were selected. For shielding, the mask signal [MASK] is also used instead.

[0097] The cross-modal image mixing masking process, from visible light images Replace the 6 subset image blocks in the infrared image with the corresponding image blocks at the appropriate locations. Then from infrared images Replace the six subset image blocks in the visible light image with the corresponding image blocks at the appropriate locations. Final infrared mask image I IR⊙mask It can be represented as:

[0098]

[0099] Visible light mask image I VI⊙mask It can be represented as:

[0100]

[0101] This embodiment uses the mask mixing processing method described above to ensure that the mask infrared and mask visible light images both contain the infrared image I. IR The features are also embedded in the visible light image I. VI The information not only enhances the complementarity of information but also improves the network's reconstruction ability when some information is missing. In addition, the random masking mechanism ensures that the mask position changes in each iteration, further weakening the boundary effect caused by fixed image block division and avoiding the potential artifact risk caused by fixed input patterns. Especially for datasets taken in harsh environments, the masking method in this embodiment plays a crucial role in the face of severe image degradation problems.

[0102] S6: Construct a small target detection network, including a multi-level Swing Transformer encoder, a fusion strategy, and a decoder; the multi-level Swing Transformer encoder uses the weight heatmap and weight vector of the memory compensation network to modulate and compensate cross-modal features.

[0103] Specifically, the small object detection network includes designing an autoencoder, fusion strategy, and autodecoder; it uses the bimodal image obtained from S5 as input to the small object detection network for feature extraction, feature fusion, and image reconstruction; such as... Figure 2 (d) shows the network structure for small target detection. The encoder consists of 5 levels of Patch Merging and 5 levels of Swin Transformer blocks (VitBlock); the decoder consists of 5 levels of Patch Expanding and 5 levels of dual ResBlock blocks.

[0104] The working principle of the small target detection network is as follows:

[0105] Before inputting the encoder, first the masked infrared image I IR⊙mask Visible light image with mask I VI⊙mask Feature mapping is performed through a linear projection layer, and positional information is embedded in the image to generate an initial feature representation of the masked infrared image. Initial feature representation of the visible light image and the mask

[0106] Then, deep feature extraction is performed using an encoder. Each level of the encoder performs hierarchical processing on the input features, with the Patch Merging module used to downsample the feature dimensions of each layer; specifically, and Output after Level 1 Vit Block and The original image is processed by a degradation estimation module (DEM) to generate a degradation score, and then processed by a memory compensation network to generate a weighted heatmap w2 and a weight vector w1. The modulation compensation operation with w2w1 is performed in the following way:

[0107]

[0108] Similarly, the outputs of the 2nd, 3rd, and 4th Vit Blocks are combined with the weights generated by the memory compensation network to perform the above calculation, and then fed into the next level network for further processing; the two outputs after the 5th Vit Block are concatenated using the Cat operation to output the features extracted by the encoder.

[0109] Finally, a decoder is used for feature reconstruction and object detection. Specifically, the Patch Expanding module upsamples the feature maps layer by layer, gradually restoring low-dimensional features to their original resolution; the ResBlock block preserves high-level semantic and detailed features and performs non-linear transformations; the last decoder layer consists of a ReLU activation function and convolutional layers, used to reconstruct the final fused image I with the target bounding box. f .

[0110] S7: Train the small target detection network using an end-to-end framework, and optimize the network by combining masking region loss and soft-IoU loss;

[0111] Specifically, training the small object detection network includes a small object detection network training framework, designing a masking region loss function, and selecting an optimization method. The training framework is an end-to-end approach from input bimodal images, degradation compensation pseudo-supervision to small object detection. In addition to using masking region loss, a joint IoU loss (Soft-IoU loss) is also adapted to improve the accuracy of small object detection. The bimodal image pairs obtained in S5 are used as the training dataset to train the small object detection network constructed in S6.

[0112] The training framework for the small object detection network is an end-to-end Yolov10 framework; the masking region loss is used to constrain the reconstruction error across modal masking regions; the design principle of the masking region loss is as follows:

[0113] First, since the small object detection network is trained by randomly masking parts of the image and mixing image patches from another modality, a masking region loss (L0) is designed for the masked regions. mask The reconstruction error of the masked area is constrained by the Masked Area Loss.

[0114] Assume the set of indices for the masked region is It contains the index locations of all masked image patches. For each masked region... The reconstructed image I is calculated using the mean squared error (MSE). f and the original image I ir / vi Losses in this area, Lmask This can be expressed using the following formula:

[0115]

[0116] in, This represents the total number of pixels in the masked area, ensuring a uniform distribution of loss; I f,i I represents the pixel value at the i-th masking position of the reconstructed image; ir / vi,i It is the actual pixel value of the original image at the i-th masking position.

[0117] To improve cross-modal detection accuracy and ensure the global structural consistency of reconstructed images, a joint IoU loss (Soft-IoU loss) is also applied to further constrain network training. The Soft-IoU loss is expressed as:

[0118]

[0119] Among them, P i,j G represents the predicted probability at position (i,j); i,j ε represents the true label at pixel (i,j); ε represents a smoothing term to prevent division by zero errors, typically a small constant (1e-6) used to ensure numerical stability.

[0120] S8: Test and evaluate the model, and output detection results and performance metrics.

[0121] Specifically, after training and validation in step S7 are completed, the network parameters are fixed and the final small object detection model is saved; the model is tested using dataset 2 from step S1 to obtain the result image with the detection box labeled; and the detection rate (P) is used. d ), false alarm rate (F) a ), recall rate (R) e The performance and accuracy of the detection model are evaluated by the area under the segment curve (IoU) and the F1 score (F).

[0122] This invention constructs a cross-modal infrared small target detection method based on degradation memory compensation. It innovatively combines metric learning with a hybrid masking strategy, addressing the problem of cross-modal small target information loss through controllable degradation estimation space. Simultaneously, it utilizes the self-supervised characteristics of global masks to solve the core problem of insufficient end-to-end supervision in traditional cross-modal small target detection tasks. To our knowledge, the method designed in this invention is one of the few in the field of cross-modal small target detection that combines image fusion, metric space, image degradation, and mask autoencoding, offering a fresh perspective on cross-modal detection and achieving excellent results across various metrics. The proposed cross-modal small target detection method not only achieves controllable degradation estimation but also achieves optimal performance in real-world small target detection tasks. Existing technologies (compare with Method 1, DNANet addresses the problem of infrared small target loss in deep networks through dense nested structures and attention mechanisms. Its core components include densely nested interaction modules (D...) The method employs a multi-layer feature progressive interaction (DNIM) and a cascaded attention module (CSAM). DNIM preserves the target information flow through multi-layer feature progressive interaction, while CSAM uses channel-space dual-path attention to dynamically enhance features. This method effectively suppresses background noise and strengthens small target features through repeated cross-layer feature fusion and mathematically modeled attention weight allocation, achieving stable target detection in low signal-to-noise ratio scenarios. In contrast, method 2, ISNet, detects small targets through Taylor finite difference edge blocks and a bidirectional attention fusion mechanism. TFD edge blocks enhance multi-level edge features through mathematical differential operators, while the TOAA module performs attention weighting in the row and column directions, fusing low-level details with high-level semantics. This method innovatively combines mathematical edge detection principles with deep learning, effectively suppressing complex background interference while maintaining the integrity of the target's geometric structure through a direction-sensitive feature aggregation mechanism, significantly improving the target contour localization accuracy. A qualitative comparison of the target detection qualitative results of the proposed method and the method in this invention on the self-made dataset is shown below. Figure 5 As shown.

[0123] Under the same conditions, the feasibility and superiority of this proposed method were further verified by calculating the correlation index of cross-modal small target detection results obtained by existing methods. A comparative diagram of the evaluation indexes of existing technologies and the method proposed in this invention is shown below. Figure 6 As shown in the figure, the method proposed in this invention has higher detection and recall rates, and lower false alarm and false alarm rates. During the testing phase, the time complexity and space complexity of the detection method proposed in this invention decreased by approximately 7.5% and 16.8% percentage points, respectively.

[0124] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal infrared small target detection method based on degradation memory compensation, characterized in that, Includes the following steps: S1: Prepare the datasets, including dataset one for training and dataset two for testing. Dataset one contains registered infrared and visible light image pairs, and dataset two contains a self-made registration test set. S2: Construct a degradation memory compensation network, including a degradation estimation module and a memory compensation network; the degradation estimation module estimates the blur degradation score, noise degradation score and visibility degradation score of infrared and visible light images through six metric spaces, and outputs them in the form of heatmaps; the memory compensation network generates weight heatmaps and weight vectors through fully connected layers, Softmax layers and hist layers. S3: Train the degenerate memory compensation network, generate degenerate samples for the bimodal control group and the consistency enhancement group through the degenerate sample generation model, and optimize the network using the adaptive ranking loss function; S4: Cross-modal image region masking processing, which divides the registered infrared and visible light images into image blocks, randomly masks some image blocks and replaces them with mask signals; S5: Cross-modal image mixing mask processing, replacing the masked infrared image block with the corresponding visible light image block, and replacing the masked visible light image block with the corresponding infrared image block, generating a mixed mask image pair; S6: Construct a small target detection network, including a multi-level Swing Transformer encoder, a fusion strategy, and a decoder; the multi-level Swing Transformer encoder uses the weight heatmap and weight vector of the memory compensation network to modulate and compensate cross-modal features. S7: Train the small target detection network using an end-to-end framework, and optimize the network by combining masking region loss and soft-IoU loss; S8: Test and evaluate the model, and output detection results and performance metrics.

2. The method according to claim 1, characterized in that, In step S1, the first dataset is the Anti-UAV dataset, and the second dataset contains infrared and visible light image pairs of various scenes. All image pairs have been registered with fine granularity.

3. The method according to claim 1, characterized in that, In step S2, the degradation estimation module includes: a fuzzy degradation subnetwork, a noise degradation subnetwork, and a visibility degradation subnetwork. Each subnetwork is composed of spatial and channel attention modules and outputs a degradation score heatmap using a Sigmoid function. The noise degradation subnetwork and the fuzzy degradation subnetwork are generated based on a probabilistic degradation model, and the generation is expressed by the following formula: Where n and k represent the size of the noise kernel and the fuzzy kernel, respectively, z n and z k M represents noise and fuzzy kernel, respectively, both following a normal distribution; n and M k These represent the noise and fuzzy generation functions, respectively. The visibility degradation subnetwork is composed of brightness and contrast degradation functions; The memory compensation network consists of a fully connected layer, a Softmax layer, and a hist layer connected to the end of each DEM.

4. The method according to claim 1, characterized in that, In step S3, the adaptation ranking loss function includes: For the Margin Ranking Loss of the bimodal control group, the difference between the constraint fuzziness degradation and noise degradation scores is expressed as follows: For the Margin Ranking Loss of the consistency enhancement group, the score difference between non-degenerate samples and severely degenerate samples is constrained; expressed as:

5. The method according to claim 1, characterized in that, In step S4, the infrared and visible light images are divided into 3×4 image blocks, and 6 of these image blocks are randomly masked and replaced with mask signals.

6. The method according to claim 1, characterized in that, In step S5, the infrared mask image after hybrid mask processing is represented as follows: The visible light mask image is represented as follows: Where {MASK} represents the mask signal, This represents the image patch at (i,j) in the original visible light image. This represents the image block at (i,j) in the original infrared image.

7. The method according to claim 1, characterized in that, In step S6, the small target detection network includes: Encoder: 5-level Patch Merging and 5-level Swing Transformer blocks, with each level's output features fused after being modulated by memory-compensated weights; Decoder: 5-level Patch Expanding and dual ResBlock blocks for reconstructing the detected image with the target bounding box.

8. The method according to claim 1, characterized in that, In step S7, the loss function for the masked area is: in, This represents the total number of pixels in the masked area, ensuring a uniform distribution of loss; I f,i I represents the pixel value at the i-th masking position of the reconstructed image; ir / vi,i It is the actual pixel value of the original image at the i-th masking position.

9. The method according to claim 1, characterized in that, In step S7, The Soft-IoU loss function is expressed as: Among them, P i,j G represents the predicted probability at position (i,j); i,j ε represents the true label at pixel (i,j); ε represents the smoothing term to prevent division by zero errors.

10. The method according to claim 1, characterized in that, In step S8, the performance metrics include detection rate, false alarm rate, recall rate, IoU, and F1 score.

Images