Infrared small target detection network with paradigm feature extraction and ode guided feature fusion

Abstract

The application discloses an infrared small target detection network with the same paradigm feature extraction and ODE guided feature fusion and belongs to the technical field of infrared small target detection. In view of the problems of the traditional network in the aspects of feature diversity and adaptability, first, a three-branch architecture of CNN, Transformer and Mamba is adopted: CNN extracts local features, Transformer obtains global information, and Mamba filters out irrelevant information and retains important features of key visual clues in the image by virtue of a unique selection mechanism; secondly, an ODE inspired feature fusion module is designed as an information bottleneck to suppress high-frequency noise, and the gradient is back-propagated to strengthen target features; then, a general architecture is abstracted from the Transformer and the Mamba, a multilayer perceptron is added to enhance the nonlinear problem processing capability, and a gating mechanism is introduced to dynamically adjust the information flow, suppress noise and irrelevant background information; finally, ablation experiments and comparative experiments are carried out on the publicly available SIRST dataset, and the effectiveness of the OFSPNet in improving the detection performance is verified.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of infrared small target detection, and particularly relates to an infrared small target detection network based on the same paradigm feature extraction and ODE guided feature fusion. BACKGROUND

[0002] Infrared small target detection is a key technology for identifying and locating small targets in infrared images, and has important application value in military, security, medical and other fields. However, infrared small targets may be extremely small in size, with only a few pixels, making it difficult to identify their shape and texture features. Moreover, infrared signals are prone to attenuation during long-distance transmission, resulting in reduced brightness of the target in the image, making it difficult to distinguish from the background. These characteristics make the infrared small target detection task extremely challenging. Therefore, improving the detection rate of the infrared small target detection task is still an urgent need in practical applications.

[0003] Traditional infrared small target detection methods are model-driven methods, such as methods based on filtering, human visual system simulation, and low-rank sparse matrix. The filter-based method is to differentiate the original image from the filtered background image to highlight the small target, but the detection performance of the small target decreases and the robustness is poor when the background is complex; the human visual system-based method uses a saliency map to distinguish small targets based on the difference between the local range of the target and the background, which is mainly suitable for scenes where the target brightness is larger and different from the surrounding background; the low-rank sparse matrix-based method uses the low-rank property of the background and the sparse property of the target to improve the accuracy of detection, but some strong clutter signals are as sparse as target signals, resulting in high false positive rate. Traditional methods generally face problems such as noise sensitivity, parameter dependence, and poor adaptability.

[0004] Benefiting from the development of computer vision in many applications, convolutional neural networks (CNNs) and vision transformers (ViTs) have been proven to be effective in handling various visual tasks. The convolution operation of CNNs can effectively extract local features in images, but the perception range is limited, making it difficult to obtain global information. Unlike standard CNN-based methods that process images pixel by pixel, vision transformers (ViTs) treat images as a sequence of patch tokens. At each layer of the network, ViT uses a multi-head self-attention mechanism to process patch tokens based on the relationship between each pair of tokens, enabling the construction of global representations of the entire image, but it is difficult to accurately identify small targets with blurred texture features due to its shortcomings in local feature extraction and detail preservation.

[0005] In addition, with the advent of the Mamba model, the application of selective state space model (SSM) based architectures in the visual domain has garnered significant attention, offering new avenues for infrared small target detection. Mamba enhances the model's ability to capture long-term dependencies through a cleverly designed state space structure. The core operations of Mamba can be highly parallelized, making it more efficient when handling large-scale data. However, Mamba requires flattening spatial data into one-dimensional tokens, which disrupts natural local two-dimensional dependencies and weakens the model's ability to accurately interpret spatial relationships. Although VMamba introduces a 2D scanning technique to address this issue by scanning the image in both horizontal and vertical directions, it still struggles to maintain the proximity of originally adjacent tokens in the scanning sequence, which is crucial for effective local representation modeling.

[0006] Despite their respective strengths, they lack in feature diversity and adaptability, which limits their performance in handling complex or specific scenarios. Moreover, there is limited in-depth research and innovation on the structure of these models. SUMMARY

[0007] To address the lack of feature diversity and adaptability in traditional networks, the present application provides an infrared small target detection network with isomorphic feature extraction and ODE-guided feature fusion.

[0008] To achieve the above purpose, the application adopts the following technical solutions:

[0009] The infrared small target detection network with isomorphic feature extraction and ODE-guided feature fusion uses a CNN, Transformer, and Mamba three-branch network architecture. The structure of Mamba and Transformer is optimized, the MLP module and residual module in the Transformer structure are retained, and the specific attention module is no longer specified. A gating module is introduced as a structure block under the same paradigm. A differential equation guided feature fusion module is designed to integrate visual features from different encoding strategies and enhance the visual representation ability of the network model.

[0010] In this network architecture, the convolutional layer CNN focuses on mining local feature details of the image; the self-attention mechanism in the Transformer is responsible for extracting the global context information of the image, widening the model's field of view; the unique selection mechanism of the Mamba model allows the model to dynamically adjust the parameters of the state space model (SSM) according to the input data, accurately eliminate redundant information, and highlight the key visual features in the image. This multi-dimensional feature extraction method not only greatly improves the accuracy of the detection task, but also enhances the model's ability to perceive subtle details. And the module structure of CNN, Transformer and Mamba is optimized and designed, in which a multi-layer perceptron (MLP) is added to enhance the model's ability to handle nonlinear problems, enabling it to capture more complex feature relationships. At the same time, a gating mechanism is introduced to allow the model to dynamically adjust the information flow, effectively retaining key features while suppressing noise and irrelevant background information.

[0011] Further, in the same paradigm, each structure block includes two residual sub-blocks; the first residual sub-block contains an unspecified module and a gating mechanism; the second residual sub-block consists of a double-layer MLP with nonlinear activation, and the formula of the structure block is as follows:

[0012]

[0013] where x i represents the input of the i-th layer, represents the output of the first residual block, σ represents SiLu, Y i represents the unspecified module, and this module is specified as an attention module, an SSM module, and a multi-scale convolution module, with the formula as follows:

[0014] Y i = Attetnion (x i · W Q , x i · W K , x i · W V ) (3)

[0015] Y i = conv 1×1 (concat(conv 1×1 (x i ), conv 3×3 (x i ), conv 5×5 (x i ))) (4)

[0016]

[0017] where Attention denotes the standard attention, and the calculation formula is Q, K, V denote query, key, value matrix; j e {1, 2, 3, 4} denotes one of the four scanning directions; z j denotes the feature sequence of the extended scanning output; denotes that the four result feature sequences are then processed separately by the S6 block; S6 denotes a state space model processing input feature sequence.

[0018] Further, the design of the ordinary differential equation guided feature fusion module is used to integrate visual features from different encoding strategies, which plays the role of information bottleneck, effectively suppresses high-frequency noise, and strengthens target features through backpropagation gradient, specifically:

[0019] The ordinary differential equation uses the function y(t) as the variable t and its derivative, and is expressed as follows:

[0020]

[0021] where f(t, y(t)) defines a time-varying function with discrete time-varying time, t = t0, t1,..., t0 represents the initial value of the ODE, and the solution at time t1 is expressed as:

[0022]

[0023] Formula (7) is rewritten as:

[0024] y(t+Δt)=y(t)+Δtf(t,y(t)) (8)

[0025] where Δt = t1-t0, and Δtf(t, y(t)) = hg(y t ), y(t+Δt) = y t+1 and y(t) = y t , formula (8) is mapped to Resblock in ResNet, and formula (8) is expressed in formula (9) as:

[0026] y t+1 =y i +hg(y t ) (9)

[0027] The fourth-order Runge-Kutta method is used to replace the first-order Euler method to reduce the local truncation error of formula (8), and the formula is as follows:

[0028]

[0029] Wherein, F1, F2, F3, F4 represent the slope calculated at different intermediate points; g(y t ) describes the rate of change of y with time t; h represents the step size.

[0030] Further, the deep network is trained using a binary cross-entropy loss and a cross-link loss to improve the performance of the network in infrared small target detection, and the formula is as follows:

[0031] l=Bce(y,)+Iou(y,) (11)

[0032] Wherein, Bce(y,) represents the binary cross-entropy loss, and Iou(y,) represents the cross-link loss.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] (1) A three-branch network structure integrating convolutional neural network (CNN), Mamba and Transformer is proposed to fully extract important local features and global context information. Drawing on the U-Net architecture, skip connections are used in the decoding stage to effectively restore image details, thereby improving the detection capability of small targets in infrared images.

[0035] (2) Innovations are made to the structure of Mamba, CNN and transformer, and the structure integrates multilayer perceptron (MLP) sub-blocks and gating mechanisms to enhance the performance and efficiency of the model in processing large-scale data sets and complex tasks.

[0036] (3) A feature fusion module guided by ordinary differential function (ODE) is proposed to adaptively aggregate visual features from different encoding strategies. The features are enhanced and noise is suppressed, effectively dealing with background interference and preserving target details. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a structural module schematic diagram, (a) is a Mamba module, (b) is a Transformer module; (c) is the same paradigm structure;

[0038] Figure 2 is the overall architecture diagram of OFSPNet;

[0039] Figure 3 is a branch structure diagram; (a) is a Transformer Block; (b) is a MambaBlock; (c) is a ConvBlock;

[0040] Figure 4The ode-inspired fusion module graph; (a) is Runge-Kutta Block; (b) is the structure of F1, F2, F3, F4;

[0041] Figure 5 The ROC comparison graph with the latest method;

[0042] Figure 6 The visual comparison graph of the detection results on the SIRST-5K dataset;

[0043] Figure 7 The visual comparison graph of the detection results on the SIRST-AUG dataset;

[0044] Figure 8 The visual comparison graph of the detection results on the NUAA dataset;

[0045] Figure 9 The 3D visualization result graph of different methods on test images. DETAILED DESCRIPTION

[0046] In order to understand the present application, it will be described in detail. However, the present application has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the overall understanding of the disclosure of the present application.

[0047] The infrared small target detection network of the same paradigm feature extraction and ODE guided feature fusion, the deep network adopts CNN, Transformer and Mamba three branch network architecture, optimizes the structure of Mamba and Transformer, such as Figure 1 (c) as shown, the MLP module and the residual module in the Transformer structure are reserved, and the specific attention module is no longer specified, a gating module is introduced as a structure block under the same paradigm; At the same time, the feature fusion module guided by ordinary differential equation is designed to integrate the visual features from different encoding strategies and enhance the visual representation ability of the network model.

[0048] In order to more effectively fuse the features output by each branch and narrow the semantic difference, first, the channel splicing operation is performed on the features generated by different branches. The purpose is to retain the uniqueness and richness of the features captured by each branch. Subsequently, the spliced features will be passed to the feature fusion module inspired by ordinary differential equation (ODE). This module plays a key role in information bottleneck, which can not only effectively filter out high-frequency noise, but also can strengthen the target features by means of the gradient signal of back propagation, so as to improve the quality and representativeness of the features.

[0049] The structure of mamba and Transformer module is analyzed, such asFigure 1 The structure of the Transformer encoder shown in (b) mainly consists of two parts: the first part is the attention module; the second part includes other key modules such as the multi-layer perceptron (MLP) and residual connections. Different from the transformer, Mamba designs the structure by combining two basic designs H3 and gated attention to form Figure 1 (a) The present invention optimizes the structural design of Mamba and transformer blocks and abstracts a structural block under the same paradigm, such as Figure 1 As shown in (c), while retaining the residual connection and MLP structure in the transformer structure, the gating mechanism in the mamba structure is integrated.

[0050] In the structural blocks under the same paradigm, each structural block consists of two residual sub-blocks; the first residual sub-block contains an unspecified module and a gating mechanism; the second residual sub-block consists of a two-layer MLP with nonlinear activation. The formula of the structural block is as follows:

[0051]

[0052] Among them, x i represents the input of the i-th layer, represents the output of the first residual block, σ represents SiLu, Y i Represents an unspecified module. By specifying the specific design of the module, different models can be obtained. Figure 3 As shown, this module is designated as an attention module, an SSM module, and a multi-scale convolution module. The formula is as follows:

[0053] Y i =Attention(x i W Q ,x i W K ,x i W V ) (3)

[0054] Y i =conv 1×1 (concat(conv 1×1 (x i ),conv 3×3 (x i ),conv 5×5 (x i ))) (4)

[0055]

[0056] where Attention denotes the standard attention, and the calculation formula is Q, K, V denote query, key, and value matrices; j e {1, 2, 3, 4} denotes one of the four scanning directions; z j denotes the feature sequence of the extended scanning output; denotes that the four resulting feature sequences are then processed individually by the S6 block; S6 denotes a state space model processing the input feature sequence.

[0057] Further, the design ordinary differential equation guided feature fusion module is used to integrate visual features from different encoding strategies, specifically:

[0058] An ordinary differential equation is an equation that uses a function y(t) as a variable t and its derivative, expressed as follows:

[0059]

[0060] where f(t, y(t)) defines a time-varying function with discrete time-varying time, t = t0, t1,..., t0 denotes the initial value of the ODE, and the solution at time t1 is expressed as:

[0061]

[0062] Formula (7) is rewritten as:

[0063] y(t+Δt)=y(t)+Δtf(t,y(t)) (8)

[0064] where Δt = T1 - T0, and Δtf(T, y(t)) = hg(y t ), y(t+Δt) = y t+1 , and y(t) = y t Formula (8) is mapped to the Resblock in ResNet, and formula (8) is expressed in formula (9) as:

[0065] y t+1 =y i +hg(y t ) (9)

[0066] The fourth-order Runge-Kutta method is used to replace the first-order Euler method to reduce the local truncation error of formula (8), and the formula is as follows:

[0067]

[0068] where F1, F2, F3, and F4 denote the slopes calculated at different intermediate points; g(y t ) describes the rate of change of y with respect to time t; and h denotes the step size.

[0069] The ODE-guided feature fusion module draws on ideas from the mathematical field to improve the network structure. Figure 4 (b) The convolution layer-activation layer-convolution layer-activation layer is used as the basic encoder block to achieve Figure 4 The fourth-order Runge-Kutta method for solving the ODE in (a). This method utilizes the integration process of the ordinary differential equation (ODE) to smoothly process the input data, effectively maintaining the stability of the solution during time stepping, thereby suppressing noise amplification caused by numerical errors. Furthermore, the ODE model's dynamic adjustment capability enables it to adaptively distinguish between signal and noise, enhancing the useful signal. During backpropagation, the ODE model can more effectively propagate gradients and enhance the target features, as the gradients more accurately reflect the contribution of the target features to the final output.

[0070] Furthermore, the binary cross entropy loss and cross-linking loss are used to train the deep network to improve the performance of the network in infrared small target detection. The formula is as follows:

[0071] l=Bce(y,)+Iou(y,) (11)

[0072] Among them, Bce(y,) represents the binary cross entropy loss and Iou(y,) represents the cross-linking loss.

[0073] In order to verify the performance of OFSPNet, experiments were conducted on the NUAA dataset, SIRST-5K dataset, and SIRST-AUG dataset. Figure 2 As shown, given an input image I, the network's end-to-end processing classifies each pixel to distinguish whether it is a target pixel, and ultimately outputs a segmentation result of the same size as I. The ratio of the training set to the test set is 8:2. During training, each image is cropped to 224x224. OFSPNet is compared with existing deep learning-based infrared small target detection algorithms. The algorithm is implemented in Pytorch with a batch size of 4. The optimizer uses Adams, where the momentum and weight decay coefficients are set to 0.9 and 0.0004, respectively. The initial learning rate is 0.001, and the decay strategy of CosineAnnealingLR is used. 150 epochs are trained on the IRST-5k and ISTR-AUG datasets, and 300 epochs are trained on the NUAA dataset. In terms of hardware, an RTX3090 GPU is used for training.

[0074] Loss Function

[0075] The network is trained by combining binary cross-entropy loss and cross-link loss, aiming to improve its performance in infrared small target detection. The segmentation-based infrared small target detection method can be regarded as a binary classification task, and the binary cross-entropy loss optimizes the classification accuracy of the model. The cross-link loss improves the positioning accuracy and ensures a high overlap between the predicted pixels and the true pixels. This combination not only enhances the detection accuracy, but also improves the robustness of the model in complex backgrounds.

[0076] Evaluation index

[0077] In terms of evaluation index, the classic semantic segmentation evaluation index, precision, recall, Fmeasure and cross-link (mIoU) are used. Precision and Recall represent the ratio of correctly classified pixels to all labeled targets and predicted targets, respectively, and the two are interdependent. In order to measure the relationship between them, Fmeasure is used, which means they are equally important. The formula is as follows, where TP, FP and FN represent the number of true positives, false positives and false negatives, respectively:

[0078]

[0079] In addition, ROC curve is used to represent the dynamic relationship between false positive rate (FPR) and true positive rate (TPR). The area under the curve (AUC) is used as a key indicator to evaluate ROC quantitatively, reflecting the model's ability to distinguish between positive and negative samples at all possible thresholds. The higher the AUC value, the better the classification performance of the model. A perfect classifier has an AUC value of 1, while a randomly guessed classifier has an AUC value close to 0.5. The formula is as follows:

[0080]

[0081] Comparative experiment

[0082] 1) Numerical evaluation: In order to accurately illustrate the effectiveness of OFSPNet, the present application adopts numerical method for quantitative evaluation. As can be seen from Table 1, the method proposed in the present application achieves the maximum value in both mIou and Fmeasure indicators. In addition, in order to more vividly show the comparison of AUC, Figure 5 the ROC curve of the method comparison is given. Experimental data show that OFSPNet has strong background suppression ability, can accurately detect targets, and can accurately segment targets.

[0083] Table 1. Comparison with the latest method on NUAA, SIRST-5K and SIRST-AUG data sets

[0084]

[0085] 2) Visual evaluation: AsFigures 6-8 As shown, typical infrared small target scenes are selected from three data sets respectively, and seven kinds of infrared small target detection methods are compared visually. In the figure, the red dashed box enlarges the detected small target, the green dashed circle represents the missed detection, and the yellow dashed circle represents the false detection. Compared with other methods, the method proposed in the application can not only accurately detect the target, but also detect more complete target pixels. In order to observe the clutter in the detection result, Figure 9 is the 3D display of 6 scenes. As can be seen from the peak and the area under the peak, the prediction confidence of the method of the application for the target is higher, and is closer to the ground-truth (GT).

[0086] Ablation experiment

[0087] In order to verify the rationality of OFSPNet, by controlling variables, each module is ablated, and it is verified that each module has a certain improvement on the performance of the network. Concat means that the outputs of the branches are simply concatenated at the channel level, and ODFM is the feature fusion module of the application.

[0088] Table 2 module ablation

[0089]

[0090] Parameter setting:

[0091] For the setting of the loss function, the binary cross entropy loss and the cross-link loss are combined to improve the accuracy of detection. The binary cross entropy loss function optimizes the classification accuracy of the model, and the IOU loss focuses on the accuracy of segmentation. The attention degree of the model to the pixel-level classification and the overall segmentation performance can be flexibly adjusted by weighting, so as to better cope with the class imbalance problem. Among them, α, β are balance coefficients, and the formula is as follows:

[0092] l i = αBce (y, ) + βIou (y, )

[0093] Table 3 parameter setting

[0094] α∶β mIOU Precision Recall Fmeasure 1∶1 70.44 82.59 83.73 83.15 2∶1 70.55 84.38 81.83 83089 3∶2 73.62 85.87 83.77 84.80 2∶3 66.06 77.30 81.96 79.56

[0095] As can be seen from Table 3, when α: β = 3:2, the performance is the best.

[0096] The details of the application not described herein are considered known to those skilled in the art. Although the foregoing specific embodiments of the application have been described in some detail for the purposes of clarity of understanding, it will be apparent that certain modifications can be practiced within the spirit and scope of the application, as defined by the appended claims.

Claims

1. Infrared small target detection network with same paradigm feature extraction and ODE guided feature fusion, characterized by: The network adopts a three-branch network architecture of CNN, Transformer, and Mamba. The structures of the Mamba and Transformer are optimized, retaining the MLP module and residual module in the Transformer structure, de-specifying the specific attention module, and introducing the gating module as a structural block under the same paradigm. At the same time, a feature fusion module guided by ordinary differential equations is designed to integrate visual features from different encoding strategies to enhance the visual representation ability of the network model. In the structural blocks under the same paradigm, each structural block consists of two residual sub-blocks; the first residual sub-block contains an unspecified module and a gating mechanism; the second residual sub-block consists of a two-layer MLP with nonlinear activation. The formula of the structural block is as follows: （1） （2） in, Indicates the The input of the layer, represents the output of the first residual block, Indicates SiLu, Represents an unspecified module, which is designated as an attention module, an SSM module, and a multi-scale convolution module. The formula is as follows: （3） （4） （5） in, Represents standard attention, and the calculation formula is , 、 、 Represents query, key, and value matrices; Indicates one of the four scanning directions; Represents the feature sequence output by the extended scan; Then pass The blocks process the four resulting feature sequences individually; Represents a state-space model processing an input feature sequence.

2. The infrared small target detection network with same paradigm feature extraction and ODE guided feature fusion according to claim 1 is characterized in that: The feature fusion module guided by the ordinary differential equation is designed to integrate visual features from different encoding strategies, specifically: Functions for ordinary differential equations For variables The equations of its derivatives are as follows: （6） in, A time-varying function with discrete time-varying duration is defined, , represents the initial value of the ODE, time The solution is expressed as: （7） Formula (7) is rewritten as: （8） in, , by definition , and , Formula (8) is mapped to Resblock in ResNet, and Formula (8) is expressed in Formula (9) as: = + （9） The fourth-order Runge-Kutta method is used instead of the first-order Euler method to reduce the local truncation error of formula (8), and the formula is as follows: （10） in, , , , It represents the slope calculated at different intermediate points; Described Over time rate of change; Indicates the step size.

3. The infrared small target detection network with same paradigm feature extraction and ODE guided feature fusion according to claim 2 is characterized in that: The target detection network is trained using binary cross entropy loss and cross-linking loss to improve the performance of the network in infrared small target detection. The formula is as follows: （11） in, represents the binary cross entropy loss, Indicates loss of cross-linking.

Images