A small infrared target segmentation method based on multi-feature fusion

Abstract

The present invention discloses a method for infrared small target segmentation based on multi-feature fusion. The specific steps of the present invention are as follows: 1. Constructing a data set; 2. Building an ID-UNet model for infrared small target segmentation; the ID-UNet model includes a densely connected UNet structure, an ISFTE module, and a residual attention mechanism module; 3. Introducing full-scale deep supervision; the ID-UNet model is directly supervised by the real annotation mask at each decoder level, so that an independent side output is generated at each decoder level, thereby achieving the goal of comprehensive deep supervision. The present invention improves the performance of infrared small target segmentation tasks by densely connecting multi-scale features and effectively fusing small target features. This strategy helps the model to comprehensively learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets; through this all-round deep supervision mechanism, ID-UNet can better understand and capture the hierarchical structure information of infrared small targets.

Description

Technical Field

[0001] The present invention provides an infrared small target segmentation method based on multi-feature fusion. Background Art

[0002] The UNet architecture has been widely used in image segmentation and other fields. Due to the limitations of the underlying UNet network and the characteristics of infrared small target images, the segmentation process of small targets in infrared images faces challenges such as uncertain contour information, complex and dynamic backgrounds, and monotonous color distribution. Furthermore, the characteristic information of small infrared targets is gradually lost during the transmission of information between upper and lower layers.

[0003] In the field of segmentation, accurately segmenting objects of different sizes simultaneously is an extremely challenging task. The authors of UNet++ believe that shallow UNets are more suitable for segmenting relatively small targets, while deeper UNets are capable of segmenting larger targets. To address this problem, UNet++ enables the model to handle targets of different sizes by sharing encoders from UNets of different depths, and guides the training of UNets of different depths through deep supervision. Finally, different UNets are cleverly integrated together through dense skip connections, enabling them to adaptively adjust to segment targets of different sizes. This method effectively improves the performance of UNet. However, this method still has certain limitations. Deep UNets have difficulty handling small targets well because as the depth of the UNet increases and downsampling operations are performed, the field of view of the features will expand, resulting in the loss of some detailed features. Summary of the Invention

[0004] The present invention addresses the shortcomings of existing technologies and, from the perspective of enhancing information flow, provides a multi-feature fusion-based infrared small target segmentation method by densely connecting multi-scale features and effectively fusing small target features. A UNet-based network structure, ID-UNet (Densely Connected UNet Architecture for Infrared Small Target Segmentation), is proposed. Experiments demonstrate that the proposed model effectively preserves small target information, thereby improving the performance of infrared small target segmentation tasks.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] Step 1: Build a dataset;

[0007] Step 2: Build the ID-UNet model for infrared small target segmentation; the ID-UNet model includes a densely connected UNet structure, an ISFTE module, and a residual attention mechanism module.

[0008] Step 3: Introducing full-scale deep supervision. The ID-UNet model is directly supervised by ground-truth annotated masks at each decoder level, generating independent side outputs at each decoder level, thus achieving the goal of comprehensive deep supervision. This strategy helps the model comprehensively learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small objects. Through this comprehensive deep supervision mechanism, ID-UNet can better understand and capture the hierarchical structure of small infrared objects.

[0009] Furthermore, the dataset described in step 1 includes three latest large-scale infrared dim target datasets, namely IRSTD-1k, NUDT-SIRST and NUAA-SIRST.

[0010] Furthermore, step 2 is implemented as follows:

[0011] The densely connected UNet structure aims to prevent the loss of small objects within the UNet model, thereby enhancing information flow between layers. The ISTFE module is used for optimization and feature fusion. This module assigns equal weights to infrared small target features across different layers to facilitate multi-scale feature fusion and extraction across densely connected unit layers. The residual attention mechanism module enhances the model's feature representation capabilities, aiming to eliminate redundancy and extract feature information from small objects and background.

[0012] The densely connected UNet structure described:

[0013] like Figure 1 As shown in Figure 1, this architecture is based on the UNet network and uses dense connections to enhance information exchange between upper and lower layers. The encoder consists of encoding layers E1 to E5, and from shallow to deep, each encoding layer is connected to all shallower encoding layers. The decoder consists of decoding layers D4 to D1, and from deep to shallow, each decoding layer is connected to all deeper decoding layers and encoding layers of the same depth.

[0014] The ISFTE module includes a feature alignment module and a feature extraction module.

[0015] Feature Alignment Module: The feature alignment module is used to resize feature maps extracted at different scales to the same size and number of channels, ensuring information balance. The feature fusion module connects multi-scale features and uses a series of consecutive residual blocks to merge these features, resizing feature maps extracted from different scales to the same size and number of channels to ensure information balance. Max pooling or bilinear interpolation is used to upscale the features to the required width and height. Convolutional blocks are then deployed to achieve feature alignment. The resized features reduce the channel size, enabling better information interaction in the subsequent feature fusion process. The increased channel and spatial attention effectively helps extract shallow and deep features of small infrared targets.

[0016] like Figure 2 As shown in (a), the feature alignment module of ISTFE is presented. This module solves the inherent inefficiency of feature size variation and unifies them to achieve efficient performance. The max pooling layer is used for downsampling and bilinear upsampling is implemented for upsampling. In order to introduce the structure more comprehensively and directly, the i-layer of ISTFE (i = 1, 2, ..., 5) is proposed, as shown in Figure 2 As shown, E i represents the concatenated encoder ith layer, where i assumes the ith downsampling layer along the encoder. Subsequently, at j = 0, each node receives features from the previous densely connected layer.

[0017] E i =G[F(P MAX (E1)),…,F(P MAX (E i-j )),…,F(P MAX (E i-1 ))], (1)

[0018] Among them, P MAX (·) denotes a pooling layer, F(·) is a residual attention module (RCSM), which uniformly reduces the input dimension to a fixed channel C value of 16, as described in the paper, where […] denotes a cascade layer and G(·) denotes a feature enhancement module.

[0019] Feature extraction module: After merging multi-scale features, the merged multi-scale features are extracted by stacking densely connected multi-scale sub-networks. Figure 2As shown in (b), this module aligns feature maps to improve the segmentation accuracy and robustness of small objects. It uses a splicing technique to fuse feature maps, and integrates the resulting spliced ​​feature maps through a residual network to ensure comprehensive feature consistency. By splicing feature maps and fusing and interacting them through a residual network, feature information at different scales is integrated, thereby improving the segmentation accuracy and robustness of small objects. The architecture consists of multiple stacked residual blocks, a spatial attention module, and a channel attention module. The introduced residual connections alleviate the vanishing gradient problem during information transmission, ultimately enhancing the network's training and generalization capabilities to improve the segmentation accuracy and robustness of small objects. Through multiple stacked residual attention modules, the system effectively extracts deep features of infrared targets. By adjusting the weights of channel and spatial features, the model's perception and anti-interference capabilities for infrared small targets are enhanced, which in turn enhances the network's training and generalization capabilities, thereby improving the segmentation accuracy and robustness of small objects. Finally, channel and spatial attention adjust the weights of channel and spatial features to enhance the model's perception and anti-interference capabilities for infrared small targets.

[0020] The residual attention mechanism module described above: In order to cope with the significant challenges that may arise from detecting and segmenting small infrared targets under complex backgrounds and noise. The deep residual attention module can enhance the feature representation and attention mechanism of the model. The residual module captures the edge details of small infrared targets and integrates them into the feature representation, significantly improving the segmentation accuracy. At the same time, the attention mechanism improves the model accuracy of target positioning and segmentation by focusing on key infrared feature areas. It integrates the attention mechanism module, combining spatial attention and channel attention into the residual module. This attention mechanism improves the representativeness of the model, focusing on key features through spatial channel attention, and suppressing unimportant features. This method pays more attention to small target features, using the skip connection of the residual network to retain the original features and reduce the loss of small targets. If the output and input channels are not the same, a 1×1 convolution block is used to convert the input to the output channel. The structural process of the residual attention mechanism module is summarized as follows:

[0021] CA(X)=σ(MLP(P avg (X))+MLP(P max (X))), (2)

[0022]

[0023] in represents element-by-element multiplication, σ represents the sigmoid function, P avg (X) and P max(X) represents average pooling and max pooling with a stride of 2, respectively. CA(X) represents channel attention and SA(X) represents the spatial attention network. The shared network consists of a multilayer perceptron with a hidden layer. This study summarizes the spatial attention process in a similar way to the channel attention process:

[0024] SA(X′)=σ(Conv([mean(X′),max(X′)])), (4)

[0025]

[0026] Where Conv(X) represents a convolution operation with a filter size of 7×7. The final " is added to the original input to form a residual structure.

[0027] The step 3 is specifically implemented as follows:

[0028] Comprehensive deep supervision consists of the total loss and manual labels for each decoder, where a higher weight is assigned to the losses generated by shallower layers. This weighting strategy aims to prioritize model guidance and correction at the shallower decoder levels, recognizing their key role in shaping the hierarchical features for effective segmentation. By combining these weighted losses, the comprehensive deep supervision mechanism ensures that the model receives stronger signals from shallower layers, promoting the learning of more local and detailed features to achieve accurate segmentation of small infrared objects. The deep supervision loss is expressed as the following equation:

[0029] Loss=(λ1Loss1+…+λ5Loss5) / (λ1+…+λ5), (6)

[0030] Which represents different weight values, Loss i represents the loss generated by the i-th decoding layer. In the experimental setting, each λ i Set to λ i+1 Twice the value.

[0031] The beneficial effects of the present invention are as follows:

[0032] The ID-UNet of the present invention adopts a simpler approach. It does not introduce a new shallow UNet, but rather uses dense connections between shallow and deep layers to more fully fuse shallow features and deep features, thereby reducing the loss of small target features when encoding deep layers, and reducing the loss of background features in the decoding layer. Compared with the surrounding complex background features, the infrared small target features account for a relatively small proportion. If a direct connection method is used, the infrared small target features will gradually weaken during the downsampling process and be submerged in the background features. The use of dense connections between different layers can enable the small target features to be fused with low loss into the high-level features of the deep layer. Through full-scale deep supervision, each layer of the model has the ability to segment the image, which enhances the robustness of the model.

[0033] The overall structure of the ID-UNet network of the present invention is as follows Figure 1 As shown in the figure. This model is built on the basis of the UNet network. It strengthens the information exchange between the upper and lower layers by introducing dense connections, and performs feature alignment and fusion through the ISFTE module. Among them, E1 to E5 represent the encoding layers from shallow to deep. Each encoding layer will connect to all the encoding layers shallower than itself, while D4 to D1 represent the decoding layers from deep to shallow. Each decoding layer will connect to all the encoding layers deeper than itself and the encoding layers of the same depth. The infrared image flows through the ID-UNet structure and the output is the infrared small target segmentation map. This model effectively retains the small target information, thereby improving the performance of the infrared small target segmentation task.

[0034] By deeply supervising full-scale features, this paper enables each layer of the ID-UNet model to detect small infrared targets, effectively improving the segmentation performance of small targets and thus the overall performance of the infrared small target segmentation task. Finally, by incrementally optimizing the characteristic changes of each UNet layer's parameters, the model controls the increase in parameter volume within a reasonable range, preventing the loss of small targets due to excessive bottom-level weights. This reduces the total number of parameters, maintains the performance of small infrared target segmentation, and improves the robustness of the model. Compared with the UNet and UNet++ structures, the number of parameters has been reduced by over 80%. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the overall structure of the ID-UNet network of the present invention.

[0036] Figure 2 Schematic diagram of the feature extraction module ISTFE of the present invention.

[0037] Figure 3 are the lightweight ID-UNet network parameters of the present invention.

[0038] Figure 4 This is the ID-UNet full-scale supervision structure of the present invention.

[0039] Figure 5 This is a display of the data set used in the experiments of this invention.

[0040] Figure 6 ROC curves on the NUDT-SIRST dataset of the present invention are compared with other methods.

[0041] Figure 7 This is the visual comparison result of ID-UNet of the present invention and other methods.

[0042] Figure 8The 3D visualization results of different methods in this invention on 6 test images.

[0043] Figure 9 This is the detection result of ID-UNet on the NUDT-SIRST dataset.

[0044] Figure 10 Heat maps of different layers of ID-UNet of the present invention.

[0045] Figure 11 This is the heat map of the left and right dense connection ablation in the ID-UNet of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to specific embodiments of the present invention.

[0047] This study employs a U-shaped network structure to enhance the segmentation process and accurately capture geometric information. This architecture allows for accurate semantic descriptions through well-extracted high-level features, while low-level features capture the target's geometric information. This allows for efficient information exchange between low-level and high-level features. This study proposes a densely connected network to address the challenge of accurately segmenting small infrared targets of varying sizes in diverse scenarios. This densely connected network fuses features from different scales, facilitating information flow between layers. Compared to the traditional UNet architecture, this densely connected network allows deep units to access all previous feature maps, eliminating the need to retain contextual information. The proposed improvements significantly enhance the performance of the small infrared target segmentation network. The densely connected structure helps capture global contextual information, while the infrared small target feature extraction module (ISTFE) fuses features of different scales from multiple layers. The attention fusion module enhances the feature representation of small infrared targets by ensuring the effective fusion of shallow and deep features. Since different network layers contain different semantic information, the attention fusion module adaptively amplifies deep and shallow features, providing more spatial and channel-wise positional information for the target.

[0048] The present invention designs a significantly lightweight parameter change through the synergistic combination of dense connections and ISTFE modules. As shown in Table 1, in the traditional UNet model, each layer requires an exponentially growing number of channels to store the information of the previous layer, and the number of channels of each layer is twice that of the previous layer. However, through the combination of dense connections, it is no longer necessary to store the information of the previous layer, but to import it directly from the dense connections into the subsequent layers. This results in a reduction in the number of channels per layer. This chapter adopts a novel parameter increment mode, which increases the number of channels in each layer by only a factor of the initial layer channels. This leads to a significant reduction in the overall model parameters. Calculations show that among units with similar structures, ID-UNet reduces 81% of the parameters compared to UNet and 83% of the parameters compared to the traditional UNet model. Figure 3 The number of channels in each layer of the UNet network and the ID-UNet network is more intuitively displayed, among which the ventilation number of ID-UNet is much smaller than that of the UNet network.

[0049] Table 1 Parameter comparison between various UNet variants

[0050]

[0051] like Figure 4 As shown in the figure, to further integrate the multi-feature information of small infrared targets and accurately segment infrared targets of different sizes, ID-UNet introduces full-scale deep supervision. Compared to the deep supervision in UNet++, ID-UNet is directly supervised by the ground-truth annotated masks at each decoder level. This means that independent side outputs are generated at each decoder level, thus achieving the goal of comprehensive deep supervision. This strategy helps the model comprehensively learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets. Through this comprehensive deep supervision mechanism, ID-UNet can better understand and capture the hierarchical structure of small infrared targets.

[0052] Experimental design and result analysis

[0053] This section demonstrates the effectiveness of the proposed method on single infrared images from real scenes. First, we provide an overview of the dataset, evaluation metrics, and experimental parameter settings. Subsequently, we compare ID-UNet with state-of-the-art infrared small object segmentation methods from the past few years. Finally, we conduct an ablation study to more thoroughly examine the network's performance.

[0054] Dataset:

[0055] 1) IRSTD-1k Dataset: IRSTD-1k consists of 1001 infrared images, each 512×512 pixels. This dataset covers a diverse range of small objects, including drones, animals, ships, and vehicles. The backgrounds include oceans, rivers, fields, mountains, cities, and clouds, accompanied by a large amount of complex noise. As a challenging benchmark, IRSTD-1k provides a comprehensive test of model performance in complex scenes.

[0056] 2) NUDT-SIRST Dataset: NUDT-SIRST consists of 1,327 infrared images with an image size of 256×256 pixels. The dataset covers diverse background scenes, including urban areas, fields, and oceans. Each image is constructed by combining different targets with the real background using a Gaussian 5×5 filter. Approximately 37% of the images contain at least two targets, and 27% of the targets occupy no more than 0.01% of the total image area. This dataset is designed to test the model's adaptability to multiple and small targets, providing a wider range of scenarios.

[0057] 3) NUAA-SIRST Dataset: The NUAA-SIRST dataset consists of 427 infrared images of varying sizes, averaging 320×320 pixels. The dataset covers diverse background scenes, including fields, sky, and ocean. Approximately 90% of the images contain only a single object, and approximately 55% of these objects occupy only 0.02% of the image area. This dataset emphasizes the challenges of single and small objects, placing higher demands on the model's detail capture and segmentation accuracy.

[0058] The experiments used the same split ratios as the original datasets to maintain consistency. Specifically, the split ratio for IRSTD-1K was 80:20, for NUDT-SIRST it was 50:50, and for NUAA-SIRST it was 50:50. To further eliminate the impact of size differences on model performance, the three datasets were resized: IRSTD-1K images were resized to 512×512, for NUDT-SIRST images to 256×256, and for NUAA-SIRST images to 512×512.

[0059] Evaluation indicators

[0060] CNN-based segmentation work mainly uses pixel-level evaluation indicators such as Intersection over Union (IoU), nIoU, pixel-level recall, pixel-level accuracy, etc. These indicators mainly focus on target shape evaluation. This chapter uses IoU to evaluate shape description capabilities.

[0061] 1) IoU (Intersection over Union) is a pixel-level evaluation metric commonly used to evaluate the performance of computer vision tasks such as object detection and semantic segmentation. It indicates the degree of overlap between the predicted result and the true label. It is calculated by dividing the area of ​​the intersection of the predicted result and the true label by the area of ​​the union of the two. The specific calculation formula is as follows:

[0062]

[0063] 2) nIoU (Normalized Intersection over Union) is a normalized IoU, an evaluation metric designed for small infrared target segmentation to better measure the segmentation performance of small targets and prevent the segmentation results of large targets from affecting the overall evaluation metric. Its specific formula is as follows:

[0064]

[0065] 3) F1 metric is a segmentation-based evaluation metric used to measure the relationship between precision and recall. Precision, recall, and F1 metric are defined as follows:

[0066]

[0067] Where N represents the number of infrared images in the test set, TP represents the intersection of the model prediction result and the true label of the data, T+P-TP represents the union of the model prediction result and the true label of the data, P represents the model prediction result predict, and T represents the true label ground truth of the data.

[0068] Experimental parameter settings

[0069] The experiments were conducted on a Linux computer equipped with a 2.50GHz CPU, 11GB RAM, and a GeForce GTX 2080ti GPU. The ID-UNet training process was implemented using Python and PyTorch. The SoftIoU-Loss loss function was used and the Adgrad

[72] The method was optimized. When initializing the weights and biases of the model, Xavier was used.

[73] The method uses 1500 epochs, 4 batch-size, 0.05 initial lr, and 1e-5 minimum lr.

[0070] Comparative experiment

[0071] Table 2 shows comparative experimental results between ID-UNet and other infrared small target segmentation algorithms. All other infrared small target segmentation methods follow the fixed threshold method described in their original papers, and all parameters remain unchanged. IRSTD-1K, NUDT-SIRST, and NUAA-SIRST are challenging datasets whose images contain varying signal-to-clutter ratios, cluttered backgrounds, target shapes and sizes, and scenes with targets of varying sizes, types, and poses. Experimental results show that on these three datasets, the ID-UNet model not only achieves the best infrared small target segmentation performance compared to other algorithms, but also has a lower parameter count.

[0072] Table 2. Comparison of quantitative evaluation indicators on the infrared small target dataset. The bold part is the best performance.

[0073]

[0074] The newly designed backbone network is tailored for infrared small target segmentation and is capable of learning discriminative features that are robust to scene changes. Dense connections improve information transmission efficiency and deep network stability. The channel and spatial attention modules enhance the network's discrimination and generalization capabilities. The multi-feature fusion module extracts and fuses feature information of different scales and depths to obtain a more comprehensive and consistent feature representation. At the same time, the fully connected UNet module also plays a vital role in improving the segmentation accuracy and robustness of small targets. In addition, compared with the most advanced algorithms, ID-UNet can more accurately depict the target shape. The innovative backbone design effectively eliminates noise by fusing deep information and adapts to various clutter backgrounds, target shapes and target size challenges, thereby achieving excellent performance. As Figure 6 As shown in Figure 2, the ID-UNet model was applied and compared with the most advanced models using false positive rate and true positive rate. By plotting ROC curves at different thresholds, it was analyzed and concluded that the ID-UNet model surpassed the existing best model.

[0075] The visualization results of the NUDT-SIRST dataset are as follows Figure 7 As shown in the figure. The target area is magnified within the blue square for enhanced visualization. Correctly detected target areas, false positives, and missed detection areas are represented by red, yellow, and green dashed circles, respectively. The proposed ID-UNet demonstrates extremely high accuracy in target localization and shape segmentation with a very low false positive rate. The method produces results with more accurate target localization and shape segmentation, with extremely low false positive and missed detection rates compared to the state-of-the-art infrared small target segmentation algorithms. This is attributed to the novel backbone architecture, which can effectively handle infrared small targets of different sizes, as well as different clutter backgrounds and noise interference, ultimately improving performance.

[0076] Figure 8 Depicted is a comparative analysis of the outputs of various models for infrared images containing small targets. The red dashed circles indicate correctly detected target regions. These outputs are generated by processing the input through a series of convolutional layers, then applying a sigmoid activation function, ultimately producing an output matrix. Experimental results demonstrate that the ID-UNet model demonstrates strong performance on a large number of infrared small target images. Compared to other state-of-the-art infrared small target segmentation algorithms, the ID-UNet model generates images that are closer to true 3D representations and accurately detects more targets. When faced with a variety of infrared small targets in diverse and complex environments, the ID-UNet model consistently demonstrates robustness, accurately distinguishing between background and target.

[0077] Figure 9 The figure shows the detection results of the ID-UNet model for small target infrared images. Targets detected by the model are indicated by red circles, and they are aligned with annotated targets for reference. As shown in the figure, the ID-UNet model demonstrates high detection efficiency for small targets of various sizes, shapes, and orientations. For example, the model successfully identifies drones, ships, and other small targets of varying shapes, while providing accurate localization and boundary estimation. These results further demonstrate the ID-UNet model's superior performance and robustness in small target detection tasks, providing reliable support for real-world application scenarios. Although the model may miss or misdetect targets in some cases, we believe that with appropriate tuning and optimization of the model, detection efficiency and overall performance can be improved.

[0078] Figure 10 Heatmaps of different layers of the ID-UNet model for infrared images of small objects are shown. Different color schemes indicate the distribution of the model's attention in different regions. The thermal images show that the ID-UNet model performs exceptionally well across most layers, exhibiting high thermal concentration around small objects, indicating that the model is focused on them. The thermal concentration steadily increases from the encoding layer to the decoding layer, further emphasizing the robust performance and perceptual capabilities of the ID-UNet model for small object detection tasks. As the model progresses from the encoding layer to the decoding layer, it gradually extracts more semantic information and features, which are seamlessly integrated to ultimately achieve accurate localization and segmentation of small objects.

[0079] Figure 11Heat maps of different layers of the ID-UNet model for infrared images of small objects are shown. Different color schemes indicate the distribution of the model's attention in different regions. The thermal images reveal that the ID-UNet model exhibits superior performance across most layers, with high heat concentration around small objects, indicating that the model is focused on them. Heat concentration steadily increases from the encoding layer to the decoding layer, further highlighting the robust performance and perceptual capabilities of the ID-UNet model in the small object detection task. As the model progresses from the encoding layer to the decoding layer, it gradually extracts more semantic information and features, which are seamlessly integrated to ultimately achieve accurate localization and segmentation of small objects.

[0080] Ablation experiments

[0081] This paper compares the ID-UNet model with several variants to evaluate the potential advantages of network modules and design choices. Ablation experiments examine the role of dense connections, the role of attention modules, the number of residual blocks, variations in the UNet architecture, and the impact of image size.

[0082] (a) Impact of the combination of dense connections and attention. Ablation experiments are conducted to evaluate the impact of dense connections and attention modules in the ID-UNet architecture, considering their demonstrated advantages. The experiments compare ID-UNet with three variants:

[0083] 1) No densely connected architecture: The densely connected structure is removed and direct cross-layer connections are eliminated.

[0084] 2) No attention module: bypass the spatial and channel attention modules and directly connect the residual block output.

[0085] 3) No dense connection and attention modules: The spatial and channel attention modules are removed from the dense connection architecture and the remaining channel / spatial modules.

[0086] As shown in Table 3, comparative analysis shows that the performance of the ID-UNet model degrades when certain components are excluded. Specifically, excluding the attention module results in a 3.46% decrease in IoU and a 4.27% decrease in nIoU. The omission of the densely connected architecture results in a 4.81% decrease in IoU and a 3.93% decrease in nIoU. When both the attention module and dense connections are omitted, the IoU and nIoU decrease by 7.47% and 3.43%, respectively. Similar trends are observed using the NUDT-SIRST dataset. These results emphasize the key role that dense connections and the attention module play in the performance of the ID-UNet model.

[0087] Table 3 Ablation experiments of dense connections and attention in ID-UNet network

[0088]

[0089] (b) The role of dense connections. A systematic evaluation was conducted to determine the contribution of the dense connection module in the ID-UNet model, specifically in the context of infrared small object segmentation. Table 4 illustrates the comparative impact of different configurations on the IoU metric. Experimental data shows that omitting the dense connections on the right side results in a 3.34% reduction in IoU, while missing the dense connections on the left side results in a 2.47% reduction in IoU. Removing the dense connections on both sides further exacerbates this reduction, resulting in a 4.55% reduction in IoU. These results emphasize the critical role of dense connections in ensuring robust information propagation and amplifying model performance.

[0090] A detailed comparison with a variant lacking dense connections shows that the implementation of right dense connections, while beneficial for model accuracy, also leads to a significant increase in computational requirements, as evidenced by a decrease in frame rate (frames per second, FPS). This emphasizes the trade-off between dense connections and computational efficiency.

[0091] Superior performance of dense connections through Figure 10 This is further validated by the heatmap visualization shown in Figure 2. A variant of the ID-UNet model that forgoes full integration of dense connections exhibits deficiencies in detecting small infrared objects. In contrast, the full ID-UNet model demonstrates significant proficiency in localizing such objects, reinforcing the value of the densely connected module in enhancing detection capabilities.

[0092] Table 4 Ablation experiment of left and right dense connections on ID-UNet model

[0093]

[0094] (c) Impact of spatial and channel attention. A comparative analysis was conducted to evaluate the differential impact of spatial and channel attention mechanisms within the residual block on model performance, with the results shown in Table 5. Experimental results show that excluding channel attention leads to a 1.69% decrease in IoU. Similarly, removing spatial attention leads to a 1.85% decrease in IoU. Notably, the simultaneous absence of both attention mechanisms leads to an even more significant 2.24% decrease in IoU. These results highlight the synergistic impact of spatial and channel attention in improving segmentation accuracy for the infrared small object detection task.

[0095] Table 5 Ablation experiments of spatial attention and channel attention in ID-UNet

[0096]

[0097] (d) Impact of the number of residual blocks on ISTFE feature fusion. The performance of the ID-UNet model is evaluated using different numbers of residual blocks in the ISTFE feature fusion module, as shown in Table 6. Empirical data shows that as the number of residual blocks increases from 1 to 3, the Intersection over Union (IoU) metric gradually improves, with enhancements of 1.71% and 0.86%, respectively. Conversely, an additional increment from m = 3 to m = 4 results in a 1.87% decrease in IoU. Based on these experimental results, it is concluded that the segmentation performance of small infrared targets is optimal when m is set to 3.

[0098] Table 6 Ablation experiments on the model with different numbers of residual blocks in the ISTFE module

[0099]

[0100] (e) The impact of spatial and channel attention. To identify the most effective attention mechanism for the infrared small object segmentation task, different attention modules were integrated into the framework. The results of these alternatives are detailed in Table 7. After comprehensive experimental analysis, it was observed that CBAM significantly outperformed other attention mechanisms on performance metrics relevant to this specific domain. Compared to the BAM attention module, CBAM achieved improvements of 3.58%, 3.43%, and 2.05% in IoU, nIoU, and F1, respectively.

[0101] Table 7 Comparative experiments of various attention modules on ID-UNet

[0102]

[0103] (f) Impact of U-Net Improvements. A detailed comparative analysis of the UNet architecture, its respective variants, and the proposed ID-UNet is presented in Table 8. When configured with similar computational units, ID-UNet demonstrates superior architectural effectiveness. It achieves a 2.91% improvement over the traditional UNet in terms of IoU and a 2.03% improvement over the UNet++ network on the same performance metric. Furthermore, an inspection of the model parameterization shows that despite having comparable structural units, the ID-UNet model has a significantly reduced number of parameters, 81% fewer than UNet and 83% fewer than UNet++. This demonstrates that ID-UNet not only outperforms in segmentation performance but also exhibits a significantly finer parameter curve, confirming the efficiency of the ID-UNet framework while maintaining accuracy and complexity.

[0104] Table 8 Comparative experiments of different UNet structures in the infrared small target segmentation task.

[0105]

[0106] (g) Impact of Deep Supervision. We evaluated the impact of incorporating a deep supervision module on the performance of ID-UNet, with detailed results shown in Table 9. Significant improvements in the IoU and nIoU metrics are achieved relative to the baseline ID-UNet lacking deep supervision. Specifically, a 0.8% improvement in IoU and a 1.63% improvement in nIoU are recorded on the IRSTD-1K dataset. In the context of the NUDT-SIRST dataset, the implementation of deep supervision leads to a marginal increase in IoU of 0.23% and a marginal increase in F1 score of 0.13%. In contrast, the IoU and nIoU on the NUAA-SIRST dataset decrease by 0.04% and 0.14%, respectively. Overall, these results demonstrate the beneficial role of deep supervision in enhancing model performance, with significant efficacy on both the IRSTD-1K and NUDT-SIRST datasets.

[0107] Table 9 Ablation experiments of deep full supervision in ID-UNet model.

[0108]

[0109] (h) Effect of image size. The effect of using datasets of different resolutions on model performance was investigated, and the respective results are summarized in Table 10. To account for the differences in dataset resolution, a series of experiments were conducted. The model ID-UNet shows reduced efficiency when applied to the NUDT-SIRST dataset with a resolution of 512x512, compared to its performance at a resolution of 256x256. Similarly, the results on the IRSTD-1K dataset at a resolution of 256x256 are inferior to those achieved at a resolution of 512x512. These findings indicate that changes in the original dataset resolution can significantly affect the efficiency of the model. This effect is particularly evident on the IRSTD-1K dataset, where the significant difference in detection results between the two resolutions can be largely attributed to the characteristics of the dataset, namely that most small infrared targets contain only a limited number of pixels. Therefore, the reduction in resolution may lead to the loss of critical information related to these small infrared targets.

[0110] Table 10 Comparative experiments of ID-UNet on infrared datasets at different resolutions

[0111]

[0112] Leveraging ablation studies, we conduct an in-depth analysis of the discrete components that comprise the ID-UNet model to characterize their individual contributions to its overall performance. The empirical evidence gathered from these studies sheds light on the impact of specific structural modifications on the efficacy of infrared small object segmentation. This, in turn, provides a directional framework for subsequent model refinement. The insights drawn from these ablation studies not only enhance our understanding of the performance advantages but also illuminate the architectural nuances that underpin the ID-UNet model.

[0113] This paper proposes the ID-UNet model, an enhanced architecture based on the UNet. It employs multi-scale dense connections to prevent the loss of small object features and segment small objects with multi-scale features. Empirical validation on three widely recognized public datasets, IRSTD-1K, NUAA-SIRST, and NUDT-SIRST, confirms the superior performance of the ID-UNet model. It outperforms competing methods and demonstrates excellent efficacy in challenging infrared small object segmentation of various types and sizes. Compared with the UNet family of frameworks, ID-UNet further demonstrates significant advantages, highlighting its potential for widespread application in this field. Therefore, it can provide valuable guidance for further research and development in this field. Furthermore, the proposed multi-scale feature fusion method has broad application prospects. However, the ID-UNet model has several limitations that need to be addressed. It cannot distinguish between the smallest objects and noise, and the combination of deep and shallow scale features focuses only on the pixel level rather than the object level, resulting in low detection accuracy and incomplete segmentation. Furthermore, despite implementing a linear scaling technique, memory usage remains high. We plan to address these issues in future work. Nevertheless, the current iteration of our model is not without limitations. It is worth noting that it is difficult to distinguish between the smallest-sized objects and noise artifacts. This is a key issue that needs to be explored and addressed in our future research work.

[0114] We are confident that the proposed ID-UNet model can significantly improve the segmentation accuracy of infrared small targets, providing key insights into the application of deep learning in infrared imaging. This paper develops ID-UNet, a densely connected network based on the UNet architecture. The goal is to enhance inter-layer information flow while minimizing the loss of small target information during information transfer. This chapter designs a UNet network structure with linearly increasing unit layer depth, leveraging the properties of densely connected architectures. The goal is to build a lightweight yet powerful network capable of accurately segmenting infrared small targets. We propose ISTFE, a feature extraction module for densely connected unit layers that amplifies multi-level and multi-scale features through feature alignment, feature fusion, and feature enhancement. ISTFE's primary goal is to efficiently extract features for small targets. Empirical results on two large-scale public infrared small target datasets, NUDT-SIRST and IRSTD-1K, demonstrate the effectiveness of our approach, which outperforms existing methods. Our approach demonstrates greater robustness to background clutter and variations in target size and shape, ultimately achieving state-of-the-art segmentation performance.

Claims

1. A small infrared target segmentation method based on multi-feature fusion, characterized in that: By densely connecting multi-scale features and effectively fusing small target features, the performance of infrared small target segmentation tasks is improved. The specific steps are as follows: Step 1: Build a dataset; Step 2: Build the ID-UNet model for infrared small target segmentation; the ID-UNet model includes a densely connected UNet structure, an ISFTE module, and a residual attention mechanism module; The densely connected UNet structure described: Based on the UNet network, dense connections are introduced to strengthen information exchange between upper and lower layers. The encoder includes encoding layers E1 to E5, and E1 to E5 are from shallow to deep. Each encoding layer is connected to all encoding layers shallower than itself. The decoder includes decoding layers D4 to D1, and D4 to D1 are from deep to shallow. Each decoding layer is connected to all decoding layers deeper than itself and to encoding layers of the same depth. The ISFTE module: Includes feature alignment module and feature extraction module; Feature alignment module: The feature alignment module is used to adjust the feature maps extracted at different scales to the same size and number of channels to ensure information balance. Specifically, the maximum pooling layer is used for downsampling and bilinear upsampling is implemented for upsampling. The i layer of the STFE module is proposed, i = 1, 2, ..., 5, E i represents the concatenated encoding ith layer, where i assumes the ith downsampling layer along the encoder; subsequently, at j = 0, each node receives features from the previous densely connected layer; HAVE BEEN i =G[F(P MAX (E1)),…,F(P MAX (HAVE BEEN i-j )),…,F(P MAX (HAVE BEEN i-1 ))], (1) Among them, P MAX (·) represents a pooling layer, F(·) is a residual attention module (RCSM), which reduces the input dimension to a fixed channel C value of 16, where […] represents a cascade layer, and G(·) represents a feature enhancement module; Feature extraction module: After merging multi-scale features, it extracts the merged multi-scale features by stacking densely connected multi-scale sub-networks. This module consists of multiple stacked residual attention modules, spatial attention modules, and channel attention modules. The stacked residual attention modules extract deep features of infrared targets, and the spatial attention modules and channel attention modules adjust the weights of channel features and spatial features, respectively, to enhance the model's perception and anti-interference capabilities for small infrared targets. The residual attention mechanism module described above: If the output and input channels are not the same, a 1×1 convolution block is used to convert the input to the output channel. The residual attention mechanism module process is summarized as follows: CA(X)=σ(MLP(P avg (X))+MLP(P max (X))), (2) in represents element-by-element multiplication, σ represents the sigmoid function, P avg (X) and P max (X) represents average pooling and maximum pooling with a stride of 2, respectively; CA(X) represents channel attention, SA(X) represents spatial attention network; the shared network consists of a multilayer perceptron with hidden layers; The spatial attention module process is summarized as follows: SA(X′)=σ(Conv([mean(X′),max(X′)])), (4) Where Conv(X) represents a convolution operation with a filter size of 7×7; the final X″ is added to the original input X to form a residual structure; Step 3: Introduce full-scale deep supervision; the ID-UNet model is directly supervised by the ground-truth annotation mask at each decoder level, thus generating independent side outputs at each decoder level, thus achieving the goal of full deep supervision; This strategy helps the model comprehensively learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets. Through this comprehensive deep supervision mechanism, ID-UNet can better understand and capture the hierarchical structural information of infrared small targets.

2. The infrared small target segmentation method based on multi-feature fusion according to claim 1 is characterized in that: The datasets described in step 1 include three latest large-scale infrared dim target datasets, namely IRSTD-1k, NUDT-SIRST and NUAA-SIRST.

3. The infrared small target segmentation method based on multi-feature fusion according to claim 1 is characterized in that Step 2 is implemented as follows: The goal of the densely connected UNet structure is to prevent the loss of small objects in the UNet model, thereby enhancing the information flow between layers; The ISTFE module is used for optimization and feature fusion. This module assigns the same weight to infrared small target features at different layers to promote multi-scale feature fusion and extraction between densely connected unit layers. The residual attention mechanism module is used to enhance the feature expression ability of the model, eliminate redundancy and extract feature information of small targets and background.

4. The infrared small target segmentation method based on multi-feature fusion according to claim 2 is characterized in that: The step 3 is specifically implemented as follows: Comprehensive deep supervision consists of the total loss of each decoder and the manual labels. The deep supervision loss is expressed as the following equation: Loss=(λ1Loss1+…+λ5Loss5) / (λ1+…+λ5), (6) Which represents different weight values, Loss i represents the loss generated by the i-th decoding layer; In the experimental setup, each λ i Set to λ i+1 Twice the value.

Images