Infrared small target segmentation method based on multi-feature fusion

Abstract

The invention discloses an infrared small target segmentation method based on multi-feature fusion. The method comprises the following specific steps: 1, constructing a data set; 2, building an ID-UNet model of infrared small target segmentation; the ID-UNet model comprises a densely connected UNet structure, an ISFTE module and a residual attention mechanism module; 3, introducing full-scale depth supervision; the ID-UNet model is directly supervised by a real labeling mask at each decoder level, so that independent side output is generated at each decoder level, and the target of comprehensive depth supervision is achieved. According to the method, multi-scale features are densely connected and small target features are effectively fused, so that the performance of an infrared small target segmentation task is improved, the strategy is helpful for a model to comprehensively learn and utilize feature representation of different decoder levels, and the overall perception and segmentation precision of a small target are enhanced; through the omnibearing deep supervision mechanism, the ID-UNet can better understand and capture the hierarchical structure information of the infrared small target.

Description

Technical Field

[0001] The invention provides an infrared small target segmentation method based on multi-feature fusion. Background Art

[0002] The UNet architecture has been widely used in the field of image segmentation and other different fields. Due to the limitations of the basic UNet network and the characteristics of infrared small target images, the segmentation process of small targets in infrared images faces challenges such as uncertain contour information, complex dynamic background, and single color distribution. In addition, the characteristic information of infrared small targets is gradually lost during the transmission of upper and lower layer information.

[0003] In the field of segmentation, accurately segmenting objects of different sizes at the same time is a very challenging task. The authors of UNet++ believe that shallow UNet is more suitable for segmenting relatively small targets, while deeper UNet is able to segment targets of larger sizes. To solve this problem, UNet++ enables the model to handle targets of different sizes by sharing encoders of UNets of different depths, and guides the training of UNets of different depths through deep supervision. Finally, different UNets are cleverly integrated together through dense jump connections, so that they can be adaptively adjusted to segment targets of different sizes. This method effectively improves the performance of UNet. However, this method still has certain limitations. Deep UNets have difficulty handling small targets well because the field of view of the features will expand with the increase of UNet depth and downsampling operations, resulting in the loss of some detailed features. Summary of the invention

[0004] The purpose of the present invention is to provide an infrared small target segmentation method based on multi-feature fusion by densely connecting multi-scale features and effectively fusing small target features in view of the shortcomings of the prior art, from the perspective of enhancing information flow. A network structure based on UNet, ID-UNet (Densely Connected UNet Architecture for Infrared Small Target Segmentation). The experiments of the present invention prove that the model effectively retains the small target information, thereby improving the performance of the infrared small target segmentation task.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Build a data set;

[0007] Step 2: Build an ID-UNet model for infrared small target segmentation; the ID-UNet model includes a densely connected UNet structure, an ISFTE module, and a residual attention mechanism module.

[0008] Step 3: Introduce full-scale deep supervision. The ID-UNet model is supervised directly by the true annotation mask at each decoder level, so independent side outputs are generated at each decoder level, thereby achieving the goal of comprehensive deep supervision. This strategy helps the model to fully learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets. Through this all-round deep supervision mechanism, ID-UNet can better understand and capture the hierarchical information of infrared small targets.

[0009] Furthermore, the data set described in step 1 includes three latest large-scale infrared dim target data sets, namely IRSTD-1k, NUDT-SIRST and NUAA-SIRST.

[0010] Furthermore, step 2 is specifically implemented as follows:

[0011] The goal of the densely connected UNet structure is to prevent the loss of small targets in the UNet model, thereby enhancing the information flow between layers. The ISTFE module is used for optimization and feature fusion. This module assigns the same weight to the infrared small target features of different layers to promote multi-scale feature fusion and extraction between densely connected unit layers. The residual attention mechanism module enhances the feature expression ability of the model, aiming to eliminate redundancy and extract feature information of small targets and background.

[0012] The densely connected UNet structure described is:

[0013] like Figure 1 As shown in the figure, the structure is based on the UNet network and strengthens the information exchange between the upper and lower layers by introducing dense connections. The encoder includes encoding layers E1 to E5, and E1 to E5 are from shallow to deep, and each encoding layer will connect to all encoding layers shallower than itself. The decoder includes decoding layers D4 to D1, and D4 to D1 are from deep to shallow, and each decoding layer will connect to all decoding layers deeper than itself and encoding layers of the same depth.

[0014] The ISFTE module includes a feature alignment module and a feature extraction module.

[0015] Feature alignment module: The feature alignment module is used to adjust the feature maps extracted from different scales to the same size and number of channels to ensure information balance. The feature fusion module connects the multi-scale features and uses a series of continuous residual blocks to merge these features, adjusting the feature maps extracted from different scales to the same size and number of channels to ensure information balance. Maximum pooling or bilinear interpolation is used to upgrade the features to the required width and height. Convolution blocks are then deployed to achieve feature alignment, and the resized features will reduce the channel size for better information interaction in the subsequent feature fusion process. The increased channel and spatial attention effectively help extract shallow and deep features of small infrared targets.

[0016] like Figure 2 As shown in (a), the feature alignment module of ISTFE is presented. This module solves the inherent inefficiency of feature size variation and unifies them to achieve efficient performance. The max pooling layer is used for the downsampling process, and bilinear upsampling is implemented for the upsampling process. In order to introduce the structure more comprehensively and directly, the i-layer (i=1, 2, ..., 5) of ISTFE is proposed, as shown in Figure 2 As shown, E i represents the ith layer of the concatenated encoder, where i assumes the ith downsampling layer along the encoder. Subsequently, at j=0, each node receives features from the previous densely connected layer.

[0017] E i =G[F(P MAX (E 1 )),…,F(P MAX (E i-j )),…,F(P MAX (E i-1 ))], (1)

[0018] Where P MAX (·) denotes a pooling layer, F(·) is a residual attention module RCSM, which uniformly reduces the input dimension to a fixed channel C value of 16, as described in this paper, where […] denotes a cascade layer and G(·) denotes a feature enhancement module.

[0019] Feature extraction module: After merging multi-scale features, the merged multi-scale features are extracted by stacking densely connected multi-scale sub-networks. Figure 2As shown in (b), the module aligns feature maps to improve the segmentation accuracy and robustness of small targets. The splicing technique is used to fuse feature maps, and the spliced ​​feature maps are integrated through the residual network to ensure that the feature information is fully consistent. By splicing feature maps and fusing and interacting through the residual network, feature information of different scales is fused, thereby improving the segmentation accuracy and robustness of small targets. The architecture consists of multiple stacked residual blocks, spatial attention modules, and channel attention modules. The introduced residual connection alleviates the gradient vanishing problem during information transmission, and ultimately enhances the training and generalization capabilities of the network to enhance the segmentation accuracy and robustness of small targets. Through multiple stacked residual attention modules, the system effectively extracts deep features of infrared targets, and enhances the model's perception and anti-interference capabilities for infrared small targets by adjusting the weights of channel and spatial features, which enhances the training and generalization capabilities of the network to enhance the segmentation accuracy and robustness of small targets. Finally, channel and spatial attention adjust the weights of channel and spatial features to enhance the model's perception and anti-interference capabilities for infrared small targets.

[0020] The residual attention mechanism module described: In order to cope with the significant challenges that may arise from detecting and segmenting small infrared targets under complex backgrounds and noise. The deep residual attention module can enhance the feature representation and attention mechanism of the model. The residual module captures the edge details of small infrared targets and integrates them into the feature representation, significantly improving the segmentation accuracy. At the same time, the attention mechanism improves the model accuracy of target localization and segmentation by focusing on key infrared feature areas. It integrates the attention mechanism module, combining spatial attention and channel attention into the residual module. This attention mechanism improves the representativeness of the model, focusing on key features through spatial channel attention, and suppressing unimportant features. This method pays more attention to small target features, using the skip connection of the residual network to retain the original features and reduce the loss of small targets. If the output and input channels are not the same, a 1×1 convolution block is used to convert the input to the output channel. The structural process of the residual attention mechanism module is summarized as follows:

[0021] CA(X)=σ(MLP(P avg (X))+MLP(P max (X))), (2)

[0022]

[0023] in represents element-by-element multiplication, σ represents the sigmoid function, P avg (X) and P max(X) denotes average pooling and max pooling with a stride of 2, respectively. CA(X) denotes channel attention and SA(X) denotes spatial attention network. The shared network consists of a multilayer perceptron with hidden layers. This study summarizes the spatial attention process in a similar way to the channel attention process:

[0024] SA(X′)=σ(Conv([mean(X′),max(X′)])), (4)

[0025]

[0026] Where Conv(X) represents a convolution operation with a filter size of 7×7. The final " plus the original input forms a residual structure.

[0027] The step 3 is specifically implemented as follows:

[0028] Comprehensive deep supervision consists of the total loss and manual labels for each decoder, with higher weights assigned to losses generated by shallower layers. This weighting strategy aims to prioritize model guidance and correction at shallower decoder levels, recognizing their key role in shaping hierarchical features for effective segmentation. By combining these weighted losses, the comprehensive deep supervision mechanism ensures that the model receives stronger signals from shallower layers, promoting the learning of more local and detailed features to achieve accurate segmentation of small infrared objects. The deep supervision loss is expressed as the following equation:

[0029] Loss=(λ 1 Loss 1 +…+λ 5 Loss 5 ) / (λ 1 +…+λ 5 ), (6)

[0030] Which represents different weight values, Loss i represents the loss generated by the i-th decoding layer. In the experimental setting, each λ i Set to λ i+1 Twice the value.

[0031] The beneficial effects of the present invention are as follows:

[0032] The ID-UNet of the present invention adopts a simpler approach. It does not introduce a new shallow UNet, but rather makes shallow features and deep features more fully integrated through dense connections between shallow and deep layers, thereby reducing the loss of small target features when encoding deep layers, and reducing the loss of background features in the decoding layer. Compared with the surrounding complex background features, the infrared small target features account for a relatively small proportion. If a direct connection method is adopted, the infrared small target features will gradually weaken during the downsampling process and be submerged in the background features. The use of dense connections between different layers can make the small target features merged with the deep high-level features with low loss. Through full-scale deep supervision, each layer of the model has the ability to segment the image, which enhances the robustness of the model.

[0033] The overall structure of the ID-UNet network of the present invention is as follows Figure 1 As shown. This model is built on the basis of the UNet network. It strengthens the information exchange between the upper and lower layers by introducing dense connections, and performs feature alignment and fusion through the ISFTE module. Among them, E1~E5 represent the encoding layer from shallow to deep, and each encoding layer will connect to all the encoding layers shallower than itself, while D4~D1 represent the decoding layer from deep to shallow. Each decoding layer will connect to all the encoding layers deeper than itself and the encoding layers of the same depth. The infrared image flows through the ID-UNet structure and the output is an infrared small target segmentation map. The model effectively retains the small target information, thereby improving the performance of the infrared small target segmentation task.

[0034] By deeply supervising the full-scale features, each layer of the ID-UNet model has the ability to detect small infrared targets, effectively improving the segmentation performance of small targets, thereby improving the overall performance of the infrared small target segmentation task. Finally, the model incrementally optimizes the change in the feature quantity of each layer of the UNet parameters, controls the increase in the parameter quantity within a reasonable range, prevents the loss of small targets due to excessive bottom-level weights, reduces the total number of parameters, maintains the performance of small infrared target segmentation, and improves the robustness of the model. Compared with the UNet and UNet++ structures, the number of parameters is reduced by more than 80%. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall structure of the ID-UNet network of the present invention. Figure 2 It is a schematic diagram of the feature extraction module ISTFE of the present invention.

[0036] Figure 3 These are the lightweight ID-UNet network parameters of the present invention.

[0037] Figure 4 This is the full-scale supervision structure of ID-UNet of the present invention.

[0038] Figure 5 This is a display of the data set used in the experiments of this invention.

[0039] Figure 6 ROC curves on the NUDT-SIRST dataset of the present invention are compared with other methods.

[0040] Figure 7 This is the visual comparison result of ID-UNet of the present invention and other methods.

[0041] Figure 8 The 3D visualization results of different methods in this invention on 6 test images.

[0042] Fig. 9 This is the detection result of ID-UNet of the present invention on the NUDT-SIRST dataset.

[0043] Fig.10 Heat map of different layers of ID-UNet of the present invention.

[0044] Fig.11 It is the heat map of the left and right dense connection ablation in the ID-UNet of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with specific embodiments of the present invention.

[0046] In this study, a U-shaped network structure is used to enhance the segmentation process to accurately capture geometric information. This architecture allows accurate semantic description through fully extracted high-level features, while low-level features are able to capture the geometric information of the target. It enables efficient information exchange when low-level and high-level features interact. This study proposes to implement a densely connected network to address the challenge of accurately segmenting infrared small targets of different sizes in different scenarios. The densely connected network fuses features of different scales and promotes the flow of information between layers. Compared with the traditional UNet structure, the densely connected network allows deep unit layers to access all previous feature maps, eliminating the need to additionally retain contextual information. The proposed improvements significantly enhance the performance of the small infrared target segmentation network. The densely connected structure helps capture global contextual information, while the infrared small target feature extraction module ISTFE (Infrared Small Target Feature Extraction) fuses features of different scales in multiple layers. The attention fusion module enhances the feature representation of small infrared targets by ensuring the effective fusion of shallow and deep features. Since different network layers contain different semantic information, the attention fusion module adaptively amplifies deep and shallow features to provide more spatial and channel location information for the target.

[0047] The present invention designs a significantly lightweight parameter change through the synergistic combination of dense connections and ISTFE modules. As shown in Table 1, in the traditional UNet model, each layer requires an exponentially growing number of channels to store the information of the previous layer, and the number of channels of each layer is twice that of the previous layer. However, through the combination of dense connections, it is no longer necessary to store the information of the previous layer, but to import it directly from the dense connections to the subsequent layers. This leads to a reduction in the number of channels per layer. This chapter adopts a novel parameter increment mode, increasing the number of channels in each layer by only a factor of the initial layer channels. This leads to a significant reduction in the overall model parameters. Calculations show that in units with similar structures, ID-UNet reduces 81% of parameters compared to UNet and 83% of parameters compared to the traditional UNet model. Figure 3 The number of channels in each layer of the UNet network and the ID-UNet network is more intuitively displayed, among which the ventilation number of ID-UNet is much smaller than that of the UNet network.

[0048] Table 1 Parameter comparison between various UNet variants

[0049]

[0050] like Figure 4 As shown in the figure, in order to further integrate the multi-feature information of small infrared targets and accurately segment small infrared targets of different sizes, ID-UNet introduces full-scale deep supervision. Compared with the deep supervision in UNet++, ID-UNet is directly supervised by the real annotation mask at each decoder level, which means that independent side outputs are generated at each decoder level, thereby achieving the goal of comprehensive deep supervision. This strategy helps the model to fully learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets. Through this all-round deep supervision mechanism, ID-UNet can better understand and capture the hierarchical structure information of small infrared targets.

[0051] Experimental design and results analysis

[0052] This section verifies the effectiveness of the proposed method on infrared single-frame images of real scenes. First, an overview of the dataset, evaluation metrics, and experimental parameter settings is given. Subsequently, ID-UNet is compared with the state-of-the-art infrared small target segmentation methods in the past few years. Finally, an ablation study is performed to more thoroughly review the performance of the network.

[0053] Dataset:

[0054] 1) IRSTD-1k dataset: IRSTD-1k consists of 1001 infrared images, each of which is 512×512 pixels in size. The dataset covers different types of small targets such as drones, animals, ships, and vehicles, and its backgrounds include oceans, rivers, fields, mountains, cities, and clouds, accompanied by a large amount of complex noise. As a challenging test benchmark, IRSTD-1k provides a comprehensive test of the model's performance in complex scenes.

[0055] 2) NUDT-SIRST dataset: NUDT-SIRST consists of 1,327 infrared images with an image size of 256×256 pixels. The dataset covers different background scenes such as cities, fields, and oceans. Each image is synthesized by Gaussian 5×5 with different targets and real backgrounds. About 37% of the images contain no less than 2 targets, and 27% of the targets occupy no more than 0.01% of the area of ​​the entire image. The dataset is designed to test the adaptability of the model to multiple targets and small targets, and provide more diverse scenes.

[0056] 3) NUAA-SIRST dataset: The NUAA-SIRST dataset consists of 427 infrared images with different image sizes and an average size of 320×320 pixels. The dataset covers different background scenes such as fields, sky, and ocean. About 90% of the images contain only one target, and about 55% of the targets only occupy 0.02% of the image area. The dataset emphasizes the challenges of single targets and small targets, and puts higher requirements on the model's detail capture and segmentation accuracy.

[0057] The split ratios originally used in the datasets were used in the experiments to maintain consistency. Specifically, the split ratios of IRSTD-1K are 80:20, the split ratios of NUDT-SIRST are 50:50, and the split ratios of NUAA-SIRST are 50:50. In order to further eliminate the impact of size differences on model performance, the three datasets were resized uniformly, with images of IRSTD-1K being 512×512, images of NUDT-SIRST being 256×256, and images of NUAA-SIRST being 512×512.

[0058] Evaluation indicators

[0059] CNN-based segmentation work mainly uses pixel-level evaluation indicators, such as IoU, nIoU, pixel-level recall, pixel-level accuracy, etc. These indicators mainly focus on target shape evaluation. This chapter uses IoU to evaluate shape description capabilities.

[0060] 1) IoU (Intersection over Union) is a pixel-level evaluation metric and one of the commonly used indicators for evaluating the performance of computer vision tasks such as object detection and semantic segmentation. The intersection over union ratio indicates the degree of overlap between the predicted result and the true label. It is calculated by dividing the area of ​​the intersection of the predicted result area and the true label area by the area of ​​the two combined. The specific calculation formula is as follows:

[0061]

[0062] 2) nIoU (Normalized Intersection over Union) is a normalized IoU, an evaluation metric designed for small infrared target segmentation to better measure the segmentation performance of small targets and prevent the segmentation results of large targets from affecting the overall evaluation metric. The specific formula is as follows:

[0063]

[0064] 3) F1 metric is a segmentation-based evaluation indicator used to measure the relationship between accuracy and recall. Accuracy, recall and F1 metric are defined as follows:

[0065]

[0066] Where N represents the number of infrared images in the test set, TP represents the intersection of the model prediction results and the true labels of the data, T+P-TP represents the union of the model prediction results and the true labels of the data, P represents the model prediction result predict, and T represents the true label ground truth of the data.

[0067] Experimental parameter settings

[0068] The experiments were conducted on a Linux computer equipped with a 2.50GHz CPU, 11GB RAM, and a GeForce GTX 2080ti GPU. The ID-UNet training process was implemented in Python and PyTorch, using the SoftIoU-Loss loss function and Adgrad

[72] The method was optimized. When initializing the weights and biases of the model, Xavier was used

[73] The method uses 1500 epochs, 4 batch-size, 0.05 initial lr, and 1e-5 minimum lr.

[0069] Comparative experiment

[0070] Table 2 shows the comparative experimental results of ID-UNet and other infrared small target segmentation algorithms. All other infrared small target segmentation methods follow the fixed threshold method described in their original papers, and the parameters are kept in the original configuration. IRSTD-1K, NUDT-SIRST and NUAA-SIRST are challenging datasets whose images contain different signal-to-clutter ratios, cluttered backgrounds, target shapes and sizes, and scenes with targets of different sizes, types and poses. The experimental results show that compared with other algorithms on these three datasets, the ID-UNet model not only shows the best infrared small target segmentation performance, but also has a lower number of parameters.

[0071] Table 2 Comparison of quantitative evaluation indicators on infrared small target dataset. The bold part is the best performance.

[0072]

[0073] The newly designed backbone network is tailored for infrared small target segmentation and is capable of learning discriminative features that are robust to scene changes. Dense connections improve the efficiency of information transmission and the stability of deep networks. The channel and spatial attention modules enhance the network's discrimination and generalization capabilities. The multi-feature fusion module extracts and fuses feature information of different scales and depths to obtain a more comprehensive and consistent feature representation. At the same time, the fully connected UNet module also plays a vital role in improving the segmentation accuracy and robustness of small targets. In addition, compared with the state-of-the-art algorithms, ID-UNet can more accurately depict the target shape. The innovative backbone design effectively eliminates noise by fusing deep information and adapts to various clutter backgrounds, target shapes, and target size challenges, thereby achieving excellent performance. Figure 6 As shown in Figure 1, the ID-UNet model was applied and compared with the most advanced models using false positive rate and true positive rate. By plotting ROC curves for different thresholds, it was analyzed and concluded that the ID-UNet model surpassed the existing best model.

[0074] The visualization results of the NUDT-SIRST dataset are as follows: Figure 7 As shown. The target area is enlarged within the blue square for enhanced visualization. The correctly detected target area, false alarm and missed detection areas are represented by red, yellow and green dashed circles, respectively. The proposed ID-UNet shows extremely high accuracy in target localization and shape segmentation with extremely low false alarm rate. The results produced by the method have more accurate target localization and shape segmentation, with extremely low false alarm and missed detection rates compared to the state-of-the-art infrared small target segmentation algorithms. This is attributed to the novel backbone architecture, which can effectively handle infrared small targets of different sizes, as well as different clutter backgrounds and noise interference, ultimately improving the performance.

[0075] Figure 8 A comparative analysis of various model outputs for infrared images containing small targets is depicted, with the red dashed circles indicating correctly detected target regions. These outputs are generated by processing the input through a series of convolutional layers and then applying a sigmoid activation function, which ultimately produces an output matrix. Experimental results show that model ID-UNet exhibits strong performance on a large number of infrared small target images. Compared to other advanced infrared small target segmentation algorithms, model ID-UNet generates images that are closer to true 3D representations and accurately detects more targets. When faced with a variety of infrared small targets in different complex environments, model ID-UNet consistently exhibits robustness and is able to accurately distinguish between backgrounds and targets.

[0076] Fig. 9 The detection results of the ID-UNet model on infrared images of small targets are shown. The targets detected by the model are represented by red circles, and they are aligned with the annotated targets for reference. As shown in the figure, the ID-UNet model shows high detection efficiency for small targets of various sizes, shapes, and orientations. For example, the model successfully identifies drones, ships, and other small targets of different shapes while providing accurate localization and boundary estimation. These results further confirm the excellent performance and robustness of the ID-UNet model in small target detection tasks, providing reliable support for practical application scenarios. Although the model may miss or misdetect targets in some cases, we believe that the detection efficiency and overall performance can be improved after proper adjustment and optimization of the model.

[0077] Fig.10 Heatmaps of different layers of the ID-UNet model for an infrared image of a small object are shown. Different color schemes indicate the distribution of the model’s attention in different regions. The thermal images show that the ID-UNet model performs exceptionally well on most layers, exhibiting high thermal concentration around small objects, indicating that the model focuses on them. The thermal concentration increases steadily from the encoding to the decoding layers, further emphasizing the robust performance and perceptual capabilities of the ID-UNet model in the small object detection task. As the model progresses from the encoding to the decoding layers, it gradually extracts more semantic information and features, which are seamlessly integrated to ultimately achieve accurate localization and segmentation of small objects.

[0078] Fig.11Heatmaps of different layers of the ID-UNet model for an infrared image of a small object are shown. Different color schemes indicate the distribution of the model's attention in different regions. The thermal images show that the ID-UNet model performs exceptionally well on most layers, showing high heat concentration around small objects, indicating that the model focuses on them. The heat concentration increases steadily from the encoding layer to the decoding layer, further emphasizing the robust performance and perceptual ability of the ID-UNet model in the small object detection task. As the model progresses from the encoding layer to the decoding layer, it gradually extracts more semantic information and features, which are seamlessly integrated to ultimately achieve accurate localization and segmentation of small objects.

[0079] Ablation experiment

[0080] This paper compares the ID-UNet model with several variants to evaluate the potential advantages of network modules and design choices. Ablation experiments involve the role of dense connections, the role of attention modules, the number of residual blocks, changes in the UNet framework, and the impact of image size.

[0081] (a) Impact of the combination of dense connections and attention. Ablation experiments are performed to evaluate the impact of dense connections and attention modules in the ID-UNet architecture, considering their demonstrated advantages. The experiments compare ID-UNet with three variants:

[0082] 1) No densely connected architecture: The densely connected structure is removed and direct cross-layer connections are eliminated.

[0083] 2) No attention module: bypass the spatial and channel attention modules and directly connect the residual block output.

[0084] 3) No densely connected and attention modules: The spatial and channel attention modules are removed from the densely connected architecture and the remaining channel / spatial modules.

[0085] As shown in Table 3, the comparative analysis shows that the performance of the ID-UNet model degrades when certain components are excluded. Specifically, excluding the attention module results in a 3.46% decrease in IoU and a 4.27% decrease in nIoU. The absence of the densely connected architecture results in a 4.81% decrease in IoU and a 3.93% decrease in nIoU. When both the attention module and dense connections are omitted, the IoU and nIoU decrease by 7.47% and 3.43%, respectively. Similar trends are observed using the NUDT-SIRST dataset. These results highlight the key role played by dense connections and attention modules in the performance of the ID-UNet model.

[0086] Table 3 Ablation experiments of dense connections and attention in ID-UNet network

[0087]

[0088] (b) The role of dense connections. A systematic evaluation was performed to determine the contribution of the dense connection module in the ID-UNet model, especially in the context of infrared small object segmentation. Table 4 illustrates the comparative impact on the IoU metric under different configurations. Experimental data show that omitting the dense connections on the right side leads to a 3.34% reduction in IoU, while the absence of dense connections on the left side leads to a 2.47% reduction in IoU. The removal of dense connections on both sides further exacerbates this reduction, resulting in a 4.55% reduction in IoU. These results highlight the key role of dense connections in ensuring robust information propagation and amplifying model performance.

[0089] A detailed comparison with a variant lacking dense connections shows that the implementation of right dense connections, while beneficial for model accuracy, also results in a significant increase in computational requirements, as evidenced by a decrease in frame rate (frames per second, FPS). This highlights the trade-off between dense connections and computational efficiency.

[0090] Superior performance with dense connections Fig.10 This is further validated by the heatmap visualization shown in . A variant of the ID-UNet model that abandons full integration of dense connections exhibits deficiencies in detecting small infrared objects. In contrast, the full ID-UNet model demonstrates significant proficiency in localizing such objects, reinforcing the value of the densely connected module in enhancing detection capabilities.

[0091] Table 4 Ablation experiment of left and right dense connections on ID-UNet model

[0092]

[0093] (c) Effects of spatial and channel attention. A comparative analysis is performed to evaluate the differential effects of spatial and channel attention mechanisms within the residual block on the model performance, and the results are shown in Table 5. The experimental results show that excluding channel attention leads to a 1.69% reduction in IoU. Similarly, removing spatial attention leads to a 1.85% reduction in IoU. Notably, the simultaneous lack of both attention mechanisms leads to a more significant 2.24% drop in IoU. These results highlight the synergistic impact of spatial and channel attention in improving the segmentation accuracy of the infrared small object detection task.

[0094] Table 5 Ablation experiments of spatial attention and channel attention in ID-UNet

[0095]

[0096] (d) The effect of the number of residual blocks on ISTFE feature fusion. The performance of the ID-UNet model is evaluated using different numbers of residual blocks in the ISTFE feature fusion module, as shown in Table 6. Empirical data show that as the number of residual blocks increases from 1 to 3, the intersection over union (IoU) metric gradually improves, increasing by 1.71% and 0.86%, respectively. In contrast, an additional increment from m = 3 to m = 4 results in a 1.87% decrease in IoU. Based on the experimental results, it is concluded that the segmentation performance of small infrared targets is best when m is set to 3.

[0097] Table 6 Ablation experiments on the model with different numbers of residual blocks in the ISTFE module

[0098]

[0099] (e) The impact of spatial attention and channel attention. In order to determine the most effective attention mechanism for the infrared small object segmentation task, different attention modules are integrated into the framework. The results of these alternatives are detailed in Table 7. After comprehensive experimental analysis, it is observed that CBAM significantly outperforms other attention mechanisms in terms of performance metrics relevant to this specific domain. Compared with the BAM attention module, CBAM improves IoU, nIoU and F1 by 3.58%, 3.43% and 2.05% respectively.

[0100] Table 7 Comparative experiments of various attention modules on ID-UNet

[0101]

[0102] (f) Impact of U-Net improvements. A detailed comparative analysis of the UNet architecture, its respective variants, and the proposed ID-UNet is performed, and the comparison results are shown in Table 8. When similar computational units are configured, ID-UNet demonstrates superior architectural effectiveness. It improves by 2.91% over the traditional UNet in terms of IoU and by 2.03% over the UNet++ network on the same performance metrics. In addition, an inspection of the model parameterization shows that despite having comparable structural units, the model ID-UNet has a significantly reduced number of parameters, with 81% less than UNet and 83% less than UNet++. This shows that ID-UNet not only performs well in segmentation performance, but also has a significantly finer parameter curve, confirming the efficiency of the ID-UNet framework while maintaining accuracy and complexity.

[0103] Table 8 Comparative experiments of different UNet structures in the infrared small target segmentation task.

[0104]

[0105] (g) Impact of Deep Supervision. The impact of incorporating a deep supervision module on the performance of ID-UNet is evaluated, and the detailed results are shown in Table 9. Significant improvements in the IoU and nIoU metrics are achieved relative to the baseline ID-UNet that lacks deep supervision. Specifically, an improvement of 0.8% in IoU and 1.63% in nIoU are recorded on the IRSTD-1K dataset. In the context of the NUDT-SIRST dataset, the implementation of deep supervision leads to a marginal increase of 0.23% in IoU and 0.13% in F1 score. In contrast, the IoU and nIoU of the NUAA-SIRST dataset decrease by 0.04% and 0.14%, respectively. Overall, these results demonstrate the beneficial role of deep supervision in enhancing model performance, with significant efficacy on both the IRSTD-1K and NUDT-SIRST datasets.

[0106] Table 9 Ablation experiments of deep full supervision in ID-UNet model.

[0107]

[0108] (h) Effect of image size. The effect of using datasets with different resolutions on the model performance is investigated and the respective results are summarized in Table 10. A series of experiments are conducted to cope with the difference in dataset resolution. The model ID-UNet is less efficient when applied to the NUDT-SIRST dataset with a resolution of 512x512, compared to the performance at a resolution of 256x256. Similarly, the results of the IRSTD-1K dataset at a resolution of 256x256 are inferior to those achieved at a resolution of 512x512. These findings suggest that changes in the original dataset resolution can significantly affect the efficiency of the model. This effect is particularly evident on the IRSTD-1K dataset, where the significant difference in detection results between the two resolutions can be largely attributed to the characteristics of the dataset, namely that most small infrared targets contain only a limited number of pixels. Therefore, the reduction in resolution may lead to the loss of critical information related to these small infrared targets.

[0109] Table 10 Comparative experiments of ID-UNet on infrared datasets at different resolutions

[0110]

[0111] Using ablation studies, the discrete components that make up the ID-UNet model are deeply analyzed to characterize their individual contributions to its overall performance. The empirical evidence gathered from these studies sheds light on the impact of specific structural modifications on the efficacy of infrared small object segmentation. This, therefore, provides a directional framework for subsequent model refinement. The insights drawn from these ablation studies not only improve the understanding of the performance advantages, but also shed light on the architectural nuances that underpin the ID-UNet model.

[0112] This paper proposes the ID-UNet model, which is an enhanced architecture based on UNet. It adopts multi-scale dense connections to prevent the loss of small target features and segment the multi-scale features of small targets. Empirical validation on three widely recognized public datasets IRSTD-1K, NUAA-SIRST, and NUDT-SIRST confirms the superior performance of the ID-UNet model. It surpasses competing methods and demonstrates excellent efficacy in infrared small target segmentation challenges of different types and sizes. Compared with the UNet series framework, ID-UNet further shows significant advantages, highlighting its potential for wide application in this field. Therefore, it can provide valuable guidance for further research and development in this field. In addition, the proposed multi-scale feature fusion method has broad application prospects. However, the model ID-UNet has some limitations that need to be addressed. It cannot distinguish between targets and noise of the smallest size, and the combination of deep and shallow scale features only focuses on the pixel level rather than the target level, resulting in low detection accuracy and incomplete segmentation effect. In addition, despite the implementation of linear scaling techniques, the memory usage is still high. We plan to address these issues in future work. Nevertheless, the current iteration of our model is not without limitations. It is worth noting that it is difficult to distinguish between the smallest-sized objects and noise artifacts. This key issue needs to be explored and addressed in our future research work.

[0113] We are confident that the proposed ID-UNet model can significantly improve the segmentation accuracy of infrared small targets, providing key insights for the application of deep learning in infrared imaging. In this paper, we develop ID-UNet, a densely connected network based on the UNet structure. The goal is to enhance the information flow between layers while reducing the information loss of small targets in information transfer. This chapter designs a UNet network structure with linearly increasing unit layer depth, making full use of the characteristics of densely connected architecture. The goal of this chapter is to build a lightweight and powerful network that can accurately segment infrared small targets, and propose ISTFE, a feature extraction module for densely connected unit layers, which amplifies multi-level and multi-scale features through feature alignment, feature fusion, and feature enhancement. The main goal of ISTFE is to efficiently extract features of small targets. The effectiveness of our method is confirmed by empirical results on two large public infrared small target datasets NUDT-SIRST and IRSTD-1K, which outperform other existing methods. Our method shows greater robustness in dealing with background clutter, target size and shape changes, and finally achieves the highest segmentation performance.

Claims

1. A small infrared target segmentation method based on multi-feature fusion, characterized in that: By densely connecting multi-scale features and effectively fusing small target features, the performance of infrared small target segmentation tasks is improved. The specific steps are as follows: Step 1: Build a data set; Step 2: Build an ID-UNet model for infrared small target segmentation; the ID-UNet model includes a densely connected UNet structure, an ISFTE module, and a residual attention mechanism module; Step 3: Introduce full-scale deep supervision; The ID-UNet model is directly supervised by the true annotation mask at each decoder level, so independent side outputs are generated at each decoder level, achieving the goal of comprehensive deep supervision; This strategy helps the model comprehensively learn and utilize feature representations at different decoder levels, enhancing the overall perception and segmentation accuracy of small targets; through this comprehensive deep supervision mechanism, ID-UNet can better understand and capture the hierarchical information of infrared small targets.

2. The infrared small target segmentation method based on multi-feature fusion according to claim 1 is characterized in that: The datasets described in step 1 include three latest large-scale infrared dim target datasets, namely IRSTD-1k, NUDT-SIRST and NUAA-SIRST.

3. The infrared small target segmentation method based on multi-feature fusion according to claim 1 is characterized in that Step 2 is implemented as follows: The goal of the densely connected UNet structure is to prevent the loss of small objects in the UNet model, thereby enhancing the information flow between layers; The ISTFE module is used for optimization and feature fusion. This module assigns the same weight to the infrared small target features of different layers to promote multi-scale feature fusion and extraction between densely connected unit layers. The residual attention mechanism module is used to enhance the feature expression ability of the model, eliminate redundancy and extract feature information of small targets and background.

4. The infrared small target segmentation method based on multi-feature fusion according to claim 2 is characterized in that: The densely connected UNet structure described is: Based on the UNet network, dense connections are introduced to strengthen the information exchange between the upper and lower layers. The encoder includes encoding layers E1~E5, and E1~E5 are from shallow to deep, and each encoding layer will connect to all encoding layers shallower than itself; the decoder includes decoding layers D4~D1, and D4~D1 are from deep to shallow, and each decoding layer will connect to all decoding layers deeper than itself and encoding layers of the same depth.

5. The infrared small target segmentation method based on multi-feature fusion according to claim 2 is characterized in that: The ISFTE module: It includes a feature alignment module and a feature extraction module; Feature alignment module: The feature alignment module is used to adjust the feature maps extracted at different scales to the same size and number of channels to ensure information balance. Specifically, the maximum pooling layer is used for downsampling and bilinear upsampling is implemented for upsampling. The i layer of the STFE module is proposed, i = 1, 2, ..., 5, E i represents the ith layer of the concatenated encoder, where the i-th assumption is along the ith downsampling layer of the encoder; subsequently, at j=0, each node receives features from the previous densely connected layer; HAVE BEEN i =G[F(P MAX (E1)),…,F(P MAX (HAVE BEEN i-j )),…,F(P MAX (HAVE BEEN i-1 ))], (1) Where P MAX (·) represents the pooling layer, F(·) is the residual attention module RCSM, which uniformly reduces the input dimension to a fixed channel C value of 16, where […] represents the cascade layer, and G(·) represents the feature enhancement module; Feature extraction module: After merging multi-scale features, the merged multi-scale features are extracted by stacking densely connected multi-scale sub-networks; this module consists of multiple stacked residual attention modules, spatial attention modules and channel attention modules; the deep features of infrared targets are extracted through multiple stacked residual attention modules, and the weights of channel features and spatial features are adjusted respectively through spatial attention modules and channel attention modules to enhance the model's perception and anti-interference capabilities for small infrared targets.

6. The infrared small target segmentation method based on multi-feature fusion according to claim 2 is characterized in that: The residual attention mechanism module described above: If the output and input channels are not the same, a 1×1 convolution block is used to convert the input to the output channel. The residual attention mechanism module process is summarized as follows: CA(X)=σ(MLP(P avg (X))+MLP(P max (X))), (2) in represents element-by-element multiplication, σ represents the sigmoid function, P avg (X) and P max (X) represents average pooling and maximum pooling with a step size of 2, respectively; CA(X) represents channel attention, SA(X) represents spatial attention network; the shared network consists of a multi-layer perceptron with hidden layers; The spatial attention module process is summarized as follows: SA(X′)=σ(Conv([mean(X′),max(X′)])), (4) Where Conv(X) represents a convolution operation with a filter size of 7×7; the final X″ plus the original input X forms a residual structure.

7. The infrared small target segmentation method based on multi-feature fusion according to claim 2 is characterized in that: The step 3 is specifically implemented as follows: Comprehensive deep supervision consists of the total loss of each decoder and the manual labels. The deep supervision loss is expressed as the following equation: Loss=(λ1Loss1+…+λ5Loss5) / (λ1+…+λ5), (6) Which represents different weight values, Loss i represents the loss incurred by the i-th decoding layer; In the experimental setup, each λ i Set to λ i+1 Twice the value.

Images