ConvNeXt-Unet infrared weak and small target detection method based on slice-assisted super segmentation

Abstract

The invention relates to the technical field of infrared target detection, in particular to a ConvNeXt-Unet infrared weak and small target detection method based on slice-assisted super segmentation. The method comprises the following steps: S1, constructing a CN-Unet network based on a coding module, a local contrast learning module and a decoding module; and S2, infrared weak and small target detection is carried out through the CN-Unet network obtained in the step S1. According to the ConvNeXt-Unet infrared weak and small target detection method based on slice-assisted super-segmentation provided by the invention, a ConvNeXt module is used as an encoder, a self-attention mechanism similar to a Swinin-Transform can be realized, a larger receptive field is obtained in feature extraction, the calculation speed is higher, the use is more convenient, and meanwhile, through a multi-scale local contrast learning module MSLCLM, the detection accuracy is improved. Therefore, the local contrast priori knowledge of the target is embedded into the learning training process of the deep neural network, and the network model is guided to more accurately position the potential target area.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared target detection, and in particular to a ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation. Background Art

[0002] As an important component of the infrared search and track (IRST) system, infrared small target detection and recognition technology plays an important role in target warning, ground monitoring, and flight guidance. However, due to the long imaging distance, complex imaging environment, and weak radiation intensity, accurate detection of infrared small targets is often a very challenging problem. This is because infrared small targets also have the following unique characteristics:

[0003] 1) Small size: Due to the long imaging distance, infrared targets are generally small, ranging from one pixel to dozens of pixels in the image.

[0004] 2) Dark: The signal-to-noise ratio (SCR) of infrared targets is usually low and they can be easily submerged in severe noise and clutter backgrounds.

[0005] 3) Shapeless: The shape features of small infrared targets are limited.

[0006] 4) Variability: The size and shape of infrared targets vary greatly in different scenarios.

[0007] Therefore, research on how to quickly and robustly detect and identify small infrared targets has long been a major challenge in infrared detection technology, attracting widespread attention from researchers worldwide. Based on the number of images required, infrared small target detection methods can be primarily categorized into two main categories: single-frame-based (SFB) and multi-frame-based (MFB). Multi-frame algorithms rely on large amounts of image data, have long response times, and have relatively low application potential. They are primarily suitable for continuous target tracking in sequential images or videos. Single-frame algorithms, however, are more widely used in various important technical fields, such as high-speed moving target detection and military early warning, due to their low complexity, strong timeliness, and ease of hardware implementation. Furthermore, based on different technical implementation methods, single-frame infrared small target detection algorithms can be further divided into two categories: model-based and deep learning-based. Model-based methods primarily employ optimization methods such as image filtering, the human visual system (HVS), and low-order approximations, relying on complex manual feature design, empirical observation, and fine-tuning of model parameters.

[0008] However, due to the lack of a reliable high-level understanding of the overall scene, these methods often exhibit poor robustness, making it difficult to effectively and reliably detect targets in complex backgrounds. In recent years, with the tremendous success of deep learning technology in numerous fields such as computer vision and artificial intelligence, research on infrared small target detection based on deep learning has also made rapid progress, with performance now generally outperforming various typical traditional detection methods. Deep learning methods, based on convolutional neural networks (CNNs) and Transformer technologies, often more efficiently extract intrinsic features of small infrared targets from images at different scales through a data-driven, end-to-end learning paradigm, without the need for human intervention. Researchers have also found that modeling infrared small target detection as a semantic segmentation problem rather than a typical object detection problem can better address the performance loss caused by small target size while also obtaining more accurate target distribution information.

[0009] Therefore, most current IRSTD algorithms utilize deep learning models based on semantic segmentation, the most typical being the U-Net network based on an encoder-decoder structure. This U-shaped segmentation network structure maximizes the preservation of shallow feature information of infrared dim targets, but it also places new demands on obtaining more high-quality infrared dim target segmentation sample data. Furthermore, while existing Transformer-based segmentation models can achieve a target receptive field exceeding that of CNNs through their self-attention mechanism, their more complex structure requires a larger number of training samples and a significant amount of computation, making it difficult to achieve a good balance between detection accuracy and computational speed. While purely CNN-based network models have fewer parameters, the receptive field of the convolution kernels is limited, making it prone to loss of infrared dim target features as the number of network layers increases during model training. Therefore, while there are numerous deep learning-based IRSTD methods, a comprehensive model that balances detection performance and efficiency is currently lacking, and many aspects still require continuous improvement and optimization.

[0010] To this end, a ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation was designed to provide a technical solution to the above technical problems. Summary of the Invention

[0011] Based on this, it is necessary to provide a ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation to solve the technical problems raised in the above background technology.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0013] The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation has the following steps:

[0014] S1: Construct a CN-Unet network based on the encoding module, local contrast learning module and decoding module;

[0015] S2: Detect infrared dim targets using the CN-Unet network obtained in step S1.

[0016] As a preferred embodiment of the ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation provided by the present invention, in step S1, the encoding module is composed of 5 stages, and Stage1, Stage3, Stage4, and Stage5 are all composed of conventional ResNet modules, and Stage2 is composed of the first 4x downsampling Stem module and the first Stage of the ConvNeXt network;

[0017] The features obtained by Stage are expressed as follows:

[0018]

[0019] Among them, C i is the number of channels.

[0020] As a preferred embodiment of the ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation provided by the present invention, in step S1, the local contrast learning module is obtained by stacking three local contrast units, and the steps are as follows:

[0021] After the convolution kernel is (2i+1)×(2i+1), i=1,2,3, the feature map F is obtained. i ;

[0022] After RELU activation function, we get

[0023] After a 3×3 dilated convolution with a dilation rate of 2i+1, the feature map is obtained.

[0024] After the RELU function, the feature maps F at different scales are output LCM1 、F LCM2 、F LCM3 ;

[0025] Splice the output of all scales, fuse multi-scale information in the channel dimension, perform scale average pooling, and obtain Y out .

[0026] As a preferred embodiment of the ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation provided by the present invention, the feature maps F at different scales are output through the RELU function. LCM1 、F LCM2 、F LCM3 , the expression is as follows:

[0027] Y i =ReLU(W 2i *ReLU(W 1i *X+b 1i )+b 2i );

[0028] The output of all scales is concatenated and expressed as follows:

[0029]

[0030] In the channel dimension, multi-scale information is integrated and scale average pooling is performed to obtain Y out , the expression is as follows:

[0031]

[0032] As a preferred embodiment of the ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation provided by the present invention, in step S1, the decoding module is used to re-fuse the target features enhanced at different scales, and fuse the semantic features at this time with the semantic features of the previous level layer by layer. The fusion steps are as follows:

[0033] E5 and M4 are fused using SBAM to obtain D4.

[0034] Fuse D4 with M3 to obtain D3, and so on. Finally, send the fused D1 into the SAHS module to obtain the final output result O, which is expressed as follows:

[0035] D4=E5+M4

[0036] D i =D i+1 +M i (i=1,2,3)

[0037] O=SAHS(D1);

[0038] Among them, D i is the result of the feature fusion of the corresponding decoder module and the local contrast module, M i is the output of the local contrast module, O is the output of the final model, and SAHS is the slice-assisted super-segmentation module.

[0039] As a preferred embodiment of the ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation provided by the present invention, the slice-assisted super-segmentation module is constructed in the following steps:

[0040] A. Split the image into several slices and resize the slices to increase the relative size of the target;

[0041] B. Predict the segmentation targets on all slices and calculate the score;

[0042] C. Perform pixel-based segmentation maximum suppression on the connected areas of each slice target, and finally obtain the scaled and filtered target segmentation result.

[0043] As a preferred embodiment of the ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation provided by the present invention, in step A, the image is segmented into several overlapping slices P1, P2...P n , and then resize the slice to obtain the probability mask M1, M2…M after inference n , the expression is as follows:

[0044] M i =O(Resize(P i ));

[0045] Among them, P i is slicing, Resize is the enlargement operation, O is model reasoning, M i It is the probability mask output after model inference.

[0046] As a preferred embodiment of the ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation provided by the present invention, in step B,

[0047] Binarize the image, set the threshold to p0 = 0.5, set the pixels greater than the threshold to 1, and the pixels less than the threshold to 0, and set the resulting mask to B1, B2...B n ;

[0048] In Section B i On the mask, use the eight-connected region method to find the target areas on the mask and the corresponding target coordinates

[0049] Separation B i All target areas on the mask Get a mask containing only a single target

[0050] according to In M i Seek to Score, where j = 1, 2, ..., m, is expressed as follows:

[0051]

[0052] The sum of the probability values of all pixels in the target area is divided by the number of pixels in the target area, and finally the coordinates of the target area are restored to the coordinates of the original image I (i_last,j_last) , the expression is as follows:

[0053]

[0054] Among them, Σ j (M kh ) i The target area The sum of the probability values of all pixels, Σ j (t kh ) i The target area The sum of all pixel points, is the identified target area, Q start It is the coordinate of the slice at the beginning of the original image.

[0055] As a preferred embodiment of the ConvNeXt-Unet infrared dim small target detection method based on slice-assisted super-segmentation provided by the present invention, in step C, the scores are sorted and the target area I with the highest score is calculated. max With the remaining target areas I1, I2…I n The iou value;

[0056] Compare with the preset maximum threshold q0, and the target area where iou is greater than q0, that is, the area with I max The highly overlapping areas are eliminated, and the expression is as follows:

[0057]

[0058] The mask with the second highest score is subjected to maximum suppression, and so on, until there is no area that can be removed; the remaining target area is the final target T.

[0059] It can be seen without a doubt that the above-mentioned technical solution of this application can definitely solve the technical problem to be solved by this application.

[0060] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0061] 1. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation provided by the present invention uses the ConvNeXt module as an encoder to implement a self-attention mechanism similar to Swin-Transformer, obtain a larger receptive field in feature extraction, and has faster calculation speed and is more convenient to use. At the same time, through the multi-scale local contrast learning module MSLCLM, the local contrast prior knowledge of the target is embedded in the learning and training process of the deep neural network, guiding the network model to more accurately locate the potential target area, thereby further improving the target detection and segmentation accuracy of the model.

[0062] 2. The CN-Unet of the present invention can achieve good results with smaller parameters and FLOPs, so it is suitable for deployment on edge devices with limited computing resources.

[0063] The present invention adopts the slice-assisted super-segmentation technology SAHS and uses carefully designed data enhancement means to slice and rescale the target image before the model prediction output to increase the relative size of the infrared weak target, which can further assist in enhancing the detection and segmentation performance during model reasoning.

[0064] 3. By introducing the ConvNeXt convolution structure to replace the traditional residual structure in Unet, the present invention can implement the same self-attention mechanism as Swin-Transformer in the form of pure convolution calculation, with faster calculation speed and the input feature size can be adjusted at will, thus effectively breaking through the problem of limited receptive field of conventional convolution kernels and obtaining richer intrinsic information of weak infrared targets.

[0065] 4. The present invention expands the original purely data-driven deep learning method by embedding visual knowledge prior between the encoder and decoder of the U-shaped network structure, forming a hybrid IRSTD model driven by both knowledge and data, which can further enhance the local contrast characteristics of weak infrared targets and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 Schematic diagram of the overall architecture of CN-Unet of the present invention;

[0068] Figure 2 Schematic diagram of the ConvNeXt model architecture of the present invention;

[0069] Figure 3 This is a traditional local comparison diagram of the present invention;

[0070] Figure 4 Schematic diagram of the dilated convolution of the present invention;

[0071] Figure 5 It is a schematic diagram of the MSLCLM structure of the present invention;

[0072] Figure 6 is the SAHS reasoning diagram of the present invention;

[0073] Figure 7 Schematic diagram of the experimental comparison of Miou on the NUAA, NUDT and IRSTD-1K datasets of the present invention;

[0074] Figure 8 This is a diagram showing the prediction effects of various algorithms of the present invention;

[0075] Figure 9 3D distribution diagram display of the targets predicted by various algorithms of the present invention;

[0076] Figure 10 This is a heat map display of the Grad-CAM interpretability of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0078] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0079] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions therein may be combined with each other.

[0080] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0081] Example 1

[0082] Reference Figures 1-6 ,ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation.

[0083] 1. Overall network structure

[0084] The overall structure of CN-Unet is as follows Figure 1 As shown in the figure, (a) Conv-Block is the encoding stage used to extract feature information of small targets, (b) MSLCLM module is used to increase the contrast between the target and the background, and (c) is the slice-assisted super-segmentation module, which is mainly divided into three parts: encoding module (Encoding Stages), local contrast learning module (MSLCLM) and decoding module (Decoding Stages). The shape change process of the input image between each part is shown in Table 1 below.

[0085] Table 1: Data flow

[0086]

[0087] Given an infrared image of any size, the encoding module is used to extract target features at different scales. Figure 1 As shown in its sub-figure (a), the encoding module consists of 5 stages, where Stage 2 is composed of the first 4-fold downsampling Stem module and the first stage of the ConvNeXt network, and Stage 1, Stage 3, Stage 4, and Stage 5 are composed of conventional ResNet modules. The features obtained by each stage are set to C i is the number of channels, set the initial feature C1 = 16, C i =2C i-1 , (i=2,3,4,5), the infrared small target continuously expands the channel dimension of the target feature after 5 encoding stages, gradually extracts the global semantic features of the small target, and finally obtains E5, at this time the number of channels C5=256.

[0088] For the target features at different scales obtained in each stage, the local contrast features of the target are further enhanced through the multi-scale local contrast module (MSLCLM), and the local contrast feature maps at different scales are obtained. The basic structure of MSLCLM is as follows Figure 1 As shown in (b), it aims to further guide the optimization direction of the network model by embedding visual prior knowledge, which can better screen infrared dim small targets that meet the local contrast characteristics, highlight the local features of small targets, and ultimately achieve the effect of more fully capturing dim small targets.

[0089] For the decoding module, the common fusion method SBAM (Simplified Bilinear Interpolation Attention Module) is used. Its function is to re-fuse the target features obtained by enhancement at different scales, and fuse the semantic features at this time with the semantic features of the previous level layer by layer. E5 and M4 are fused with SBAM to obtain D4, and then D4 is fused with M3 to obtain D3, and so on. Finally, the fused D1 is sent to the SAHS module to obtain the final output result O. The specific formula is as follows:

[0090] D4=E5+M4

[0091] D i =D i+1 +M i (i=1,2,3)

[0092] O=SAHS(D1) (1)

[0093] Among them, D i is the result of the feature fusion of the corresponding decoder module and the local contrast module, M i is the output of the local contrast module, O is the output of the final model, and SAHS is the slice-assisted super-segmentation module.

[0094] Next, the network will go through the data enhancement stage SAHS module as follows Figure 1 As shown in (c), after further prediction through slice-assisted super-segmentation, the output is the final mask for infrared dim target detection and segmentation.

[0095] 2. ConxNeXt module

[0096] Considering the excellent performance of Transformer in computer vision applications, Facebook introduced the Swin-Transformer architecture into the convolutional neural network (CNN), resulting in the ConvNeXt network. Its performance is similar to that of Swin-Transformer, but with much better computational performance. Under the same GFLOPs, the accuracy of ConvNeXt-T is 0.7 percentage points higher than that of Swin-T. Figure 2 .

[0097] This network imitates the training strategy of Swin-Transformer based on the traditional ResNet. First, for the Stem stage of the network, Stem is directly replaced with a convolution kernel with a convolution kernel size of 4 and a stride of 4. Next, ConvNeXt changes the ratio of the number of times the 4 Stage stacked Blocks are changed to 1:1:3:1. After each Stage, downsampling is performed using a 2×2 convolution with a stride of 2. The Stem module in the ConvNeXt network and the Stage1 with 3 ConvNeXt modules stacked are introduced into the model of the present invention as the Stage2 of the model of the present invention, thereby replacing the feature extraction function of Swin-Transformer.

[0098] Compared to the residual network, the ConvNeXt module uses depthwise separable convolution, adopts a larger 7×7 convolution kernel, increases the number of channels, and uses an anti-bottleneck structure, that is, a structure with thin ends and thick middle. It imitates the MSA structure of the Transformer, moves the depthwise separable convolution upward, changes the BN to LN, changes the activation function from RELU to GELU, and uses fewer normalization functions and activation functions. The specific calculation formula for each block is shown in (2):

[0099] X′=X+w2·GELU(w1·LN(DWConv(X)) (2)

[0100] Among them, DWConv 7×7 is a 7×7 depthwise separable convolution, LN is LayerNorm, w1 and w2 are linear projection weights, and GELU is the activation function.

[0101] The main purpose of using this module is to achieve the same performance as the Swin-Transformer with pure convolutional computation. ConvNeXt has proven to be superior. Furthermore, compared to the Swin-Transformer, ConvNeXt's purely convolutional architecture allows for flexible input feature size, making it suitable for multi-scale image processing.

[0102] In ConvNeXt, the Stem module is also an uncovered convolution because the convolution stride and size are identical. Using this module in ConvNeXt improves accuracy from 79.4% to 79.5% and reduces GFLOPs from 4.5 to 4.4%. Therefore, introducing the Stem layer into the detection network of our invention eliminates sliding window intersections and processes information from only one patch at a time.

[0103] 3. Local contrast module

[0104] The Multi-scale Local Contrast Learning Module (MSLCLM) is obtained by stacking three Local Contrast Learning Units (LCL_Unit). Figure 5 As shown in Figure 3, the whole stage is divided into three steps. The first is to pass through local contrast units of different scales, then stack the obtained feature maps, and finally fuse multi-scale information.

[0105] Traditional local contrast calculation usually uses the neighborhood of the pixel point, that is, the information of the patch is calculated as follows Figure 3 As shown in the figure, by comparing the grayscale difference between the central target T and the eight surrounding pixels B1-B8, the local contrast information of the target T is obtained. However, during the network training process, such a calculation method is not suitable for end-to-end networks. Therefore, the present invention considers embedding the above-mentioned local contrast calculation process into the deep learning network structure, and approximating the local contrast calculation of the target T through void convolution, thereby solving the problems of difficult hyperparameter adjustment and low accuracy of PCM (Patch-Based Contrast Measure) in complex and changeable real-world scenarios.

[0106] The LCL_Unit designed by the present invention uses dilated convolution to perform multi-scale contrast on the target, increase the contrast between the target and the background, and highlight the location information of small targets. In addition, the LCL_Unit uses a combination of convolution and dilated convolution to learn local contrast feature information. It uses three scales of dilated convolution, such as Figure 3 As shown, these are dilated convolutions of size 3×3 with dilation rates d of 1, 3, and 5, respectively. LCL_Unit improves upon PCM with a lightweight structure that can learn to generate local feature maps during network training. For example, in LCL_Unit2, the first step is to perform a 3×3 convolution with a stride of 1. This operation approximates the smoothing process of each region block in PCM. The second step, by performing a dilated convolution with a dilation rate of 1 in 3, approximates the metric method used in PCM and calculates the metric between the target region and neighboring regions.

[0107] like Figure 5 As shown, MSLCLM is stacked with 3 LCL_Units. The convolution kernel size and expansion rate of each LCL_Unit are different. First, the feature map F is obtained by ordinary convolution with a convolution kernel of (2i+1)×(2i+1) and i=1,2,3. i , and then through the RELU activation function to get Then after a 3×3 dilated convolution with a dilation rate of 2i+1, the feature map is obtained. Then pass through the RELU function to output feature maps F at different scales LCM1 、F LCM2 、F LCM3 The specific formula is shown in (3). Then, the outputs of all scales are spliced together, as shown in formula (4). Finally, the multi-scale information is fused in the channel dimension and scale average pooling is performed to obtain Y out .

[0108] Y i =ReLU(W 2i *ReLU(W 1i *X+b 1i )+b 2i ) (3)

[0109]

[0110] Among them, W 1i The shape is (2i+1)×(2i+1), i=0,1,2 ordinary convolution, W 2i The shape is a 3×3 (expansion rate 2i+1) dilated convolution with i=0, 1, 2.

[0111] 4. SAHS Reasoning

[0112] By borrowing the idea of slicing-assisted reasoning from the SAHI method, a new technical method of slicing-assisted hyper-segmentation module (Slicing Aided Hyper Segementation SAHS) was redesigned. Figure 6 As shown in the figure, this technology first divides the image into several slices and resizes the slices to increase the relative size of the target. It then predicts the segmentation targets on all slices and calculates the score. It then performs a pixel-based maximum value suppression operation on the connected areas of the targets in each slice, and finally obtains the scaled and filtered target segmentation result. The whole process can be divided into the following three stages:

[0113] The first stage is model inference. First, the image is divided into several overlapping slices P1, P2…P n , then resize the slice, that is, enlarge the slice to a certain extent, and then send the enlarged slice into the model for inference. The probability mask M1, M2…M after inference is obtained n , the specific formula is shown in (6):

[0114] M i =O(Resize(P i )) (6)

[0115] Among them, P iis slicing, Resize is the enlargement operation, O is model reasoning, M i It is the probability mask output after model inference.

[0116] The second stage is to find the coordinates of the target. First, the image is binarized, the threshold is set to p0 = 0.5, the pixels greater than the threshold are set to 1, the pixels less than the threshold are set to 0, and the resulting mask is set to B1, B2...B n , in Section B i On the mask, use the eight-connected region method to find the target areas on the mask and the corresponding target coordinates Separation B i All target areas on the mask Get a mask containing only a single target according to In M i Seek to Score, where j = 1, 2, ..., m, the calculation of Score is as shown in formula (7), the sum of the probability values of all pixels in the target area is divided by the number of pixels in the target area, and finally the coordinates of the target area are restored to the coordinates of the original image I (i_last,j_last) , as shown in formula (8):

[0117]

[0118] Among them, Σ j (M kh ) i The target area The sum of the probability values of all pixels, Σ j (t kh ) i The target area The sum of all pixel points, is the identified target area, Q start It is the coordinate of the slice at the beginning of the original image.

[0119] The third stage is non-maximum suppression. Sorting the scores, calculating the target area with the highest score I max With the remaining target areas I1, I2…I n The iou value is compared with the preset maximum threshold q0, and the target area with iou greater than q0 is selected, that is, the target area with I max Highly overlapping regions are removed, see formula (9). Then, the mask with the second highest score is subjected to maximum suppression, and so on, until there are no more regions that can be removed. The remaining target region is the final target T.

[0120]

[0121] The main difference between the SAHS of the present invention and the previous SAHI is that: SAHI is mainly used in the field of target detection, dividing the input image into multiple small-scale slices, performing target detection on each slice, obtaining the Bounding Box and its confidence corresponding to the candidate target area of each slice, and then uniformly converting and merging the prediction results on all slices into the original input image, and using the maximum suppression algorithm to obtain the final prediction result, thereby achieving the effect of assisting enhanced reasoning; the SAHS proposed in the present invention uses the core idea of the SAHI algorithm to further develop an inference algorithm adapted to the segmentation task, uses the continuity of infrared target pixels at the pixel level, and locks the target coordinates through connected areas, thereby replacing the Bounding Box used in target detection.

[0122] In order to apply the maximum suppression algorithm in the segmentation task, the present invention designs the NMS Score value in SAHS. Using the Score score, calculations only need to be performed in the inference stage, without the need to retrain and predict the entire model. This makes SAHS also applicable to other target segmentation tasks. It is simple and convenient, and can assist in reasoning on the Mask output by the network model, and finally obtain better and more refined segmentation results.

[0123] Example 2

[0124] refer to Figure 7-10 , based on the above embodiment 1, an experimental method is disclosed.

[0125] 1. Dataset

[0126] In the experiments, three public datasets are used, namely NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, which contain 427, 1327, and 1000 images, respectively. The training and test sets of NUAA-SIRST and NUDT-SIRST are segmented using the method in [B.Li et al., "Dense Nested Attention Network for Infrared Small Target Detection," ITIP, vol. 32, pp. 1745-1758, Feb 2023. doi: 10.1109 / TIP.2022.3199107.], and IRSTD-1K is segmented using the method in [M.Zhang, R.Zhang, Y.Yang, H.Bai, J.Zhang, and J.Guo, "ISNet: Shape Matters for Infrared Small Target Detection," in Proceedings of 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 867-876.]. Therefore, all segmentation criteria are unified.

[0127] 2. Evaluation indicators

[0128] We compare our model with state-of-the-art (SOTA) methods using several standard metrics.

[0129] 1). Normalized Union Intersection (Niou) and Mean Intersection-Union (Miou): Niou is the normalized version of iou, and Miou is the mean intersection-union, whose formula is (10). It represents the intersection-union of each category in the image. However, in the image of small infrared targets, there is only one category of targets, so the values of Miou and Niou are the same.

[0130]

[0131] Where T[i] is the true target region of the i-th class, P[i] is the predicted target region of the i-th class, and TP[i] is the intersection of the true target region and the predicted target region of the i-th class, i.e., the part that is predicted correctly.

[0132] 2). Detection probability (p d ): It is an indicator to measure the accuracy of model detection. The calculation formula is shown in (11):

[0133]

[0134] Among them, p d is the correct prediction target N pred With all targets N all According to the method of [B.Li et al., "Dense Nested Attention Network for Infrared Small Target Detection," ITIP, vol. 32, pp. 1745-1758, Feb 2023. doi: 10.1109 / TIP.2022.3199107.], if the deviation of the target center point is less than 3, the target prediction is considered correct.

[0135] 3). False alarm rate (F a ):F a is the incorrectly predicted target pixel N false With all pixels P in the image all The ratio is calculated as (12):

[0136]

[0137] 4) F1-score: It is the harmonic mean of precision and recall, and is applicable to problems with class imbalance. The calculation formula is shown in (13):

[0138]

[0139] Among them, Precision is the precision rate and Recall is the recall rate.

[0140] 3. Implementation process

[0141] Implementation details: During training, each image is normalized. To avoid overfitting, training data is augmented by random flipping and rotation. The weights and biases of the model are initialized using the Kaiming initialization method

[35] . The model is trained using the Soft-Iou loss function

[36] and optimized using the Adam optimizer with an initial learning rate of 0.0005 and a multi-stage learning rate that is decayed to 0.1 at 200 and 300 epochs. The batch size and number of epochs are set to 8 and 400, respectively. According to

[20] , the fixed threshold of the segmentation prediction map is set to 0.5.

[0142] Benchmark Models: To evaluate the performance of our method, we compared the slice-assisted super-segmentation CN_Unet network with the most advanced infrared small target methods, including five traditional methods: Top-Hat, Max-Median, WSLCM, IPI, and MSLSTIPT, and seven deep learning-based methods: ACM, ALCNet, RDIAN, ISTDU-Net, DNA-Net, UIU-Net, and Unet. To ensure a fair comparison, all deep learning-based methods were retrained using the same training dataset as our model, and the fixed thresholds used in the original papers were retained.

[0143] Table 2: Equipment

[0144]

[0145] 3.1 Comparison with State-of-the-Art Methods

[0146] 3.1.1 Quantitative analysis

[0147] Table 3: Miou (%), Pd (%), Fa (10 -6 ), F1(%) comparison

[0148]

[0149]

[0150] Table 3 shows the performance of each method on three public datasets, and the best and second-best values in each column are marked in bold and underlined, respectively.

[0151] Table 4: Model parameters and calculation speed

[0152]

[0153] Among them, Table 4 shows the number of parameters and operation speed of each model.

[0154] As shown in Tables 3 and 4, on the NUAA-SIRST dataset, the Miou index of the present invention reached the highest, and the Pd, Fa, and F1 values reached the top 2. In particular, Miou improved by 4.93% compared to the baseline model U-Net. On the NUDT-SIRST dataset, the Pd and F1 values reached the highest, Fa reached the top 2, and the Miou value ranked third among all models. The first was DNANet. However, compared to DNANet, the number of parameters of the present invention was only 47.18% of DNANet, but the computing speed was 4.97 times faster than DNANet. On the IRSTD-1K dataset, the Miou, Pd, and F1 of the present invention all achieved the best results. Although the Fa effect was not the best, compared to the RDIAN model ranked first in Fa, the model of the present invention had a Miou 4.12% higher, a Pd 1.09% higher, and an F1 0.6% higher. Compared with the competing methods DNANet and UIU-Net, the computational cost of the present invention was significantly reduced and the real-time performance was higher. This shows that the CN-Unet of the present invention effectively balances the requirements of computational complexity and detection performance, and is more suitable for deployment on infrared detection equipment with limited resources.

[0155] Figure 7 The bar chart shows the performance of various deep learning-based Miou algorithms. It can be seen that the Miou model of the present invention achieves the highest performance on the NUAA-SIRST and IRSTD-1K datasets. It also leads the way on the NUDT-SIRST dataset.

[0156] 3.1.2 Qualitative analysis

[0157] In order to further intuitively evaluate the detection results between the proposed method and other comparison methods, several infrared dim target images of different scenes were selected from the above dataset, and the different detection and segmentation effects of the above methods were compared as follows: Figure 8 As shown. It can be seen that the ACM algorithm often produces a large number of false alarms and missed detections. In addition, even when the target is detected, its outline is often unclear. For scene (b), only the model of the present invention and the DNANet model can accurately separate the UAV from the branches. Other algorithms have missed detections and false alarms. This is because the method of the present invention slices the image, indirectly enlarging the relative size of the target, thereby reducing the missed detection rate of small targets. For scene (c), only the networks CN-Unet, ALCNet and DNANet of the present invention can identify small targets hidden in complex backgrounds. For scenes (d) and (e), compared with other models, the model of the present invention can not only fully identify the small targets in the picture, but also the outlines of the small targets identified by the network of the present invention are clearer.

[0158] Further, Figure 9 A 3D distribution plot of the prediction results of various algorithms is provided, allowing for a clearer and more intuitive understanding of how each model recognizes small objects. As can be seen from the plot, the proposed CN-Unet model corresponds to the target's grayscale distribution in all scenarios, and the distribution results are very close to those of the GT map, confirming that the proposed method indeed has good detection and segmentation results.

[0159] Finally, the Grad-CAM interpretability heat map is used to visualize the feature maps of CN-Unet and the benchmark method UIUNet at different scales. Figure 10 As shown, in the feature extraction process of picture a, the model of the present invention first identifies the area of the entire picture. In this process, it can be seen that the model of the present invention can identify more small targets and reduce the missed detection rate of small targets; in the feature extraction process of picture b, it can be seen that the benchmark method UIUNet mainly extracts the outline of the background objects in the shallow image, and its attention is mainly focused on the non-target area, which easily causes the target segmentation results to be missing or incomplete during the subsequent feature fusion; and the model of the present invention can always pay attention to the target area even during shallow feature extraction, so the target information retained in the subsequent feature fusion will be more complete, which also proves that the method of the present invention can indeed obtain better target detection and segmentation effects in complex backgrounds.

[0160] 4. Ablation Experiment

[0161] To demonstrate the effectiveness of the main modules in the proposed CN-Unet, we conducted ablation experiments on the benchmark dataset using the same experimental process as the quantitative analysis in step 3.1.1. The experimental results are shown in Table 5 below, with the best results also highlighted in bold.

[0162] Table 5: CN-Unet model ablation experiment

[0163]

[0164] Using the original Unet model as the base model, the local contrast module (MSLCLM), encoder module (ConvNeXt), and slice-assisted super-segmentation module (SAHS) were gradually added to verify their effectiveness on different infrared small target datasets. As shown in Table 5, the addition of these modules continuously improved the algorithm's average performance. When all modules were added, the algorithm achieved the optimal performance. This demonstrates the effectiveness of the CN-Unet model designed in this paper for infrared small target detection.

[0165] 4.1. ConvNeXt module ablation experiment

[0166] Considering that too many ConvNeXt modules will increase the number of model parameters, affect the training speed, and may cause the model to overfit (because more data training is required), an ablation analysis is also performed here on the number of ConvNeXt modules added to the entire model. In other words, under the same experimental conditions, we tried to use ConvNeXt modules in all four stages, from Stage 2 to Stage 5. As shown in Table 6, it can be seen that the effect is best when Stage 2 is a ConvNeXt module, achieving the best results in terms of Miou and Fa indicators. Its parameter count of 2.199921M is also the smallest, and its computing speed of 1.832714GFlops is also the fastest. Therefore, in order to achieve the best balance between algorithm model performance and efficiency, only one ConvNeXt module needs to be added when the CN-Unet model is actually used.

[0167] Table 6: ConvNeXt module ablation experiment

[0168]

[0169] 4.2.MSLCLM Ablation Experiment

[0170] In order to determine the number of LCL_Units in the MSLCLM module, under the basic backbone network U-net+ConvNeXt structure of the present invention, by controlling the number of LCL_Units, the MSLCLM module is subjected to an ablation experiment to analyze the changes in various indicators. As shown in Table 7, it can be seen that with the increase in the number of LCL_Unit units, the performance of various indicators of the CN-Unet model is also continuously improved. When 3 LCL_Unit units are selected, all indicators reach the best, so MSLCLM(1,3,5) is selected. MSLCLM(1,3,5) consists of 3 LCL_Units, of which the first LCL_Unit consists of a 1×1 ordinary convolution and a 3×3 dilated convolution with a dilation rate of d=1, the second LCL_Unit consists of a 3×3 ordinary convolution and a 3×3 dilated convolution with a dilation rate of d=3, and the third LCL_Unit consists of a 5×5 ordinary convolution and a 3×3 dilated convolution with a dilation rate of d=5.

[0171] Table 7: MSLCLM module ablation experiment

[0172]

[0173] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation is characterized by: Here are the steps: S1: Construct a CN-Unet network based on the encoding module, local contrast learning module and decoding module; S2: Detect infrared dim targets using the CN-Unet network obtained in step S1.

2. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 1 is characterized in that: In step S1, the encoding module is composed of 5 stages, and Stage1, Stage3, Stage4, and Stage5 are all composed of conventional ResNet modules. Stage2 is composed of the first 4x downsampling Stem module and the first Stage of the ConvNeXt network. The features obtained by Stage are expressed as follows: Among them, C i is the number of channels.

3. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 1 is characterized in that In step S1, the local contrast learning module is obtained by stacking three local contrast units. The steps are as follows: After the convolution kernel is (2i+1)×(2i+1), i=1,2,3, the feature map F is obtained. i ; After RELU activation function, we get After a 3×3 dilated convolution with a dilation rate of 2i+1, the feature map is obtained. After the RELU function, the feature maps F at different scales are output LCM1 、F LCM2 、F LCM3 ; Splice the output of all scales, fuse multi-scale information in the channel dimension, perform scale average pooling, and obtain Y out .

4. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 3 is characterized in that After the RELU function, the feature maps F at different scales are output LCM1 、F LCM2 、F LCM3 , the expression is as follows: Y i =ReLU(W 2i *ReLU(W 1i *X+b 1i )+b 2i ); The output of all scales is concatenated and expressed as follows: In the channel dimension, multi-scale information is integrated and scale average pooling is performed to obtain Y out , the expression is as follows:

5. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 1 is characterized in that: In step S1, the decoding module is used to re-fuse the target features obtained by enhancement at different scales, and fuse the semantic features at this time with the semantic features of the previous level layer by layer. The fusion steps are as follows: E5 and M4 are fused using SBAM to obtain D4. Fuse D4 with M3 to obtain D3, and so on. Finally, send the fused D1 into the SAHS module to obtain the final output result O, which is expressed as follows: D4=E5+M4 D i =D i+1 +M i (i=1,2,3) O=SAHS(D1); Among them, D i is the result of the feature fusion of the corresponding decoder module and the local contrast module, M i is the output of the local contrast module, O is the output of the final model, and SAHS is the slice-assisted super-segmentation module.

6. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 5 is characterized in that: The construction steps of the slice-assisted super-segmentation module are as follows: A. Split the image into several slices and resize the slices to increase the relative size of the target; B. Predict the segmentation targets on all slices and calculate the score; C. Perform pixel-based segmentation maximum suppression on the connected areas of each slice target, and finally obtain the scaled and filtered target segmentation result.

7. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 6 is characterized in that: In step A, the image is divided into several overlapping slices P1, P2...P n , and then resize the slice to obtain the probability mask M1, M2…M after inference n , the expression is as follows: M i =O(Resize(P i )); Among them, P i is slicing, Resize is the enlargement operation, O is model reasoning, M i It is the probability mask output after model inference.

8. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 6 is characterized in that: In step B, Binarize the image, set the threshold to p0 = 0.5, set the pixels greater than the threshold to 1, and the pixels less than the threshold to 0, and set the resulting mask to B1, B2...B n ; In Section B i On the mask, use the eight-connected region method to find each target area on the mask and the corresponding target coordinates Separation B i All target areas on the mask Get a mask containing only a single target according to In M i Seek to Score, where j = 1, 2, ..., m, is expressed as follows: The sum of the probability values of all pixels in the target area is divided by the number of pixels in the target area, and finally the coordinates of the target area are restored to the coordinates of the original image I (i_last,j_last) , the expression is as follows: Among them, Σ j (M kh ) i The target area The sum of the probability values of all pixels, Σ j (t kh ) i The target area The sum of all pixel points, is the identified target area, Q start It is the coordinate of the slice at the beginning of the original image.

9. The ConvNeXt-Unet infrared small target detection method based on slice-assisted super-segmentation according to claim 6, characterized in that: In step C, sort the scores and calculate the target area I with the highest score. max With the remaining target areas I1, I2…I n The iou value; Compare with the preset maximum threshold q0, and the target area where iou is greater than q0, that is, the area with I max The highly overlapping areas are eliminated, and the expression is as follows: The mask with the second highest score is subjected to maximum suppression, and so on, until there is no area that can be removed; the remaining target area is the final target T.

Images