Infrared small target segmentation method based on weak supervision

Abstract

The invention discloses an infrared small target segmentation method based on weak supervision. Only the infrared small target image and the infrared small target bounding box data are input, and the final output result is the segmentation mask of the small target in the infrared small target image. The method comprises the steps of model pre-training, model loading, teacher model pseudo mask generation, student model training and double-model EMA, wherein teacher model pseudo mask generation, student model training and double-model EMA need to be repeated for multiple times of iterative training. On the basis of using weak supervision loss PS-Loss, an adaptive learning rate adjustment mechanism is introduced, and the learning rate is dynamically adjusted according to the expression of the model on a verification set. In an infrared small target weak supervision training process, an EMA-based strategy, namely index moving average, is adopted. Two models are maintained, one model is used for training, the other model is used for obtaining weak changes of the infrared small target image, and instability and fluctuation of the models in training are successfully reduced.

Description

Technical Field

[0001] The invention provides an infrared small target segmentation method based on weak supervision. Background Art

[0002] The scarcity of natural image datasets has always been one of the reasons that hinder the further improvement of the performance of natural image segmentation algorithms. Similarly, this problem also plagues infrared small target segmentation algorithms. This challenge has prompted researchers to seek more adaptive methods to deal with the problem of data scarcity. In this context, weakly supervised algorithms have become one of the solutions. By using data with only rough labels or less annotation information, weakly supervised algorithms can train models in the absence of sufficient annotations. In the field of infrared small target segmentation, the introduction of weakly supervised algorithms has brought new possibilities for overcoming data scarcity. By redesigning loss functions and training strategies, these algorithms can effectively utilize a small amount of annotation information to achieve accurate segmentation of infrared small targets. Therefore, weakly supervised algorithms have become one of the most popular directions in the current research on infrared small target segmentation.

[0003] At present, there is no related work in the field of infrared small target segmentation that uses weak supervision to solve the problem of scarce data sets. In order to fill this research gap, this paper solves this problem from the perspective of weak supervision for the first time, by introducing a weakly supervised image segmentation method BoxInf (Box based weakly supervised segmentation method for Infrared small targets) designed for the characteristics of infrared small targets. Summary of the invention

[0004] The purpose of the present invention is to provide a method for infrared small target segmentation based on weak supervision in view of the deficiencies of the prior art. The input of the present invention is only the infrared small target image and the infrared small target bounding box data, and the final output result is the segmentation mask of the small target in the infrared small target image. The present invention includes model pre-training, model loading, teacher model generating pseudo mask, student model training and dual model EMA, wherein the teacher model generating pseudo mask, student model training and dual model EMA need to be repeated for multiple iterations.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Use the infrared small target image and the corresponding bounding box mask to pre-train the model in the PS-Loss loss function.

[0007] Step 1: Copy the parameters of the pre-trained model to the teacher model and the student model respectively. Fix the parameters of the teacher model for use in subsequent steps.

[0008] Step 2: Generate a pseudo mask for the infrared small target image through the teacher model.

[0009] Step 3: Compare the pseudo mask with the bounding box mask and remove the pseudo mask outside the bounding box, because the small infrared target has been preset to be within the bounding box.

[0010] Step 4: Use the bounding box mask of step 1 and the pseudo mask processed in step 3 as the guidance labels of the loss function to control the training of the student model and obtain the output of the student model. In the process of training the student model through the loss function, the bounding box mask uses the PS-Loss loss function, while the pseudo mask uses the general image segmentation loss function DiceIoU-Loss loss function.

[0011] Step 5: Input the student model and the teacher model into the EMA algorithm to update the teacher model, then return to step 3 and perform multiple repeated iterative training. The final student model is used as the infrared small target segmentation model, and the prediction mask output by the final student model is used as the final output for evaluation.

[0012] The beneficial effects of the present invention are as follows:

[0013] The present invention introduces an adaptive learning rate adjustment mechanism based on the use of weak supervision loss PS-Loss, and dynamically adjusts the learning rate according to the performance of the model on the validation set. This helps the model better adapt to different data distributions and scenarios and improves generalization ability. In the weak supervision training process of infrared small targets, an EMA-based strategy, i.e., exponential moving average, is adopted. By maintaining two models, one for training and the other for obtaining slight changes in infrared small target images, the instability and fluctuation of the model during training are successfully reduced. This strategy not only improves the robustness of training, but also accelerates the convergence process. In addition, data enhancement techniques are introduced in training, including random rotation, scaling, and horizontal flipping, etc., to increase the adaptability of the model to different scales and angles, and color enhancement techniques are additionally used for the student model. These methods help to improve the generalization performance of the model and make it more robust in practical applications. The infrared small target weak supervision training framework proposed by the present invention comprehensively considers multiple aspects such as loss function design, learning rate adjustment, EMA strategy and data enhancement to achieve better performance and robustness. Experiments have verified the effectiveness of the framework on different data sets and achieved satisfactory performance improvement.

[0014] In summary, the present invention uses the infrared small target bounding box mask as supervision, combines the out-of-box loss, in-box loss and the dual-model based EMA iterative algorithm, and provides a more economical and efficient training method for infrared small target segmentation. Without the need for large-scale annotation, the model can acquire knowledge from limited weak labels, thereby performing well in the infrared small target segmentation task. The experimental results show that the cumbersome and expensive problems of manual annotation methods have been successfully overcome, providing a more efficient and sustainable training method for infrared small target segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the overall framework of the weak supervision algorithm BoxInf for infrared small targets.

[0016] Figure 2 It is the scope of loss outside the frame and loss inside the frame.

[0017] Figure 3 is the mapping between the real mask and the bounding box mask.

[0018] Figure 4 This is the Project-Loss flow chart for the out-of-box loss function.

[0019] Figure 5 Principles for edge shape generation of small infrared targets.

[0020] Figure 6 Loss edge points and internal points within the box.

[0021] Figure 7 This is the Shape-Loss flow chart for the in-box loss.

[0022] Figure 8 False positive rate of segmented data with different thresholds on the infrared small target dataset.

[0023] Fig. 9 Convert the infrared small target dataset into the dataset required for weak supervision.

[0024] Fig.10 Compare the weak supervision algorithm with other algorithms. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the embodiments.

[0026] The weak supervision loss function PS-Loss of infrared small targets consists of the projection loss Project-Loss outside the frame and the shape loss Shape-Loss inside the frame. Figure 2As shown in the figure, according to the different scopes of the two losses, the goal of the out-of-box loss is to limit the small infrared target within the bounding box, while the goal of the in-box loss is to adjust the shape of the small target within the bounding box. Specifically, the out-of-box projection loss maps the maximum value of the predicted image to the x-axis and y-axis, and calculates the loss values ​​on the x-axis and y-axis by comparing the bounding box mask. In contrast, the in-box loss uses the prior knowledge of the infrared small target image as the label value, obtains the edge pixels and internal pixels of the infrared small target image as labels through the directional difference convolution kernel, and then calculates the shape adjustment loss. The final infrared small target weak supervision loss L ps Defined as:

[0027] L ps =L project +λL shape (1)

[0028] Where L project is the out-of-frame projection loss, with a weight value of 1; L shape It is the shape loss within the box, and the weight value defaults to λ, which is 0.1 in the experiment. The weak supervision framework is applied by modifying the loss function without modifying the original infrared small target detection model and the small target segmentation model outside the model. The design of this loss function enables the training framework to effectively guide the model to learn the accurate positioning and fine segmentation of infrared small targets, thereby improving the performance of the model.

[0029] The specific operation of the out-of-frame projection loss Project-Loss is as follows Figure 3 As shown in , first, the predicted mask output by the infrared small target segmentation model needs to be mapped to the x-axis and y-axis, and the bounding box mask of the infrared small target image also needs to be mapped in the same way. Then, DiceIoU-Loss is introduced for calculation. This loss function is used to measure the accuracy and position alignment of the small target predicted by the model within the box, as shown in Figure 4 As shown in the figure. By comparing the similarity between the predicted mask output by the model and the actual label on the x-axis and y-axis, DiceIoU-Loss effectively guides the model to ensure that the predicted result is as consistent as possible with the actual position and shape. The introduction of this loss function further enhances the performance of the method in small target positioning and segmentation accuracy. While fully considering the position and shape information, this method shows more outstanding performance in the infrared small target segmentation task.

[0030] The main purpose of the out-of-box projection loss Project-Loss is to map small targets into bounding box masks. By ensuring that the bounding box mask mapping matches the predicted mask mapping on the x-axis and y-axis, the model has the ability to distinguish small targets from the background. Where m represents the bounding box mask and b represents the predicted mask output by the student model. The projection process obtains the corresponding projection m on the x-axis and y-axis by calculating the maximum value of the bounding box mask on the y-axis and x-axis.x and m y , as shown in formula (2). Similarly, the corresponding projections b of the predicted mask on the x-axis and y-axis can be obtained x and b y , as shown in formula (3). Then, the DiceIoU-Loss loss between the bounding box mask projection and the predicted mask projection is calculated. Formula (4) is the loss function formula of the out-of-box loss Project-Loss. The formula is as follows:

[0031] Project x (m) = max y (m) = m x , (2)

[0032] Project y (b) = max x (b) = b x , (3)

[0033] L Project =L DiceIoU (m x ,b x )+L DiceIoU (m y ,b y ), (4)

[0034] The loss function is to minimize the difference between the bounding box mask and the predicted mask. y (m) represents the mapping of the bounding box mask on the y-axis, and max x (b) represents the mapping of the predicted mask on the x-axis.

[0035] The Shape-Loss in-frame shape loss uses a directional difference convolution kernel to extract the shape features of small infrared targets, and pre-sets labels based on the shape features to generate prior labels. The core of this idea is to use the directional difference convolution kernel method to capture the shape features of small infrared targets and distinguish between internal and external pixels in the bounding box. Figure 5 This process is shown in Figure 1. By using this convolution kernel, the labels of the shape features can be set in advance, thereby guiding the process of generating labels. This method helps to more accurately guide the model to learn the shape information of small infrared targets and improves the performance of the model in terms of in-frame loss. This design enables the model to better adapt to the shape changes of small infrared targets, further enhancing the accuracy and robustness of segmentation.

[0036] Specifically, for an infrared small target image I∈R w*h , use the x-axis and y-axis direction differential convolution kernels v' and h' to extract the shape feature map U of the infrared small target in two directions x and U y, and then use the square sum of the two variables to generate the shape U. By setting the threshold, the larger and smaller values ​​are set as edge pixels and internal pixels respectively, and the edge shape graph U∈R is generated by edge pixels and internal pixels. w*h , guiding the model to perform training optimization. The formula for obtaining the edge shape graph is as follows:

[0037]

[0038] For the predicted value, it needs to be mapped to an edge value or an internal value. The prior knowledge of this idea is that it is generally believed that a small target is a whole with no ring inside and is stable. The predicted values ​​of two pixels on the x-axis or y-axis are the same to classify them. If they are the same, the pixel is an internal pixel, and if they are different, the pixel is an external pixel. Figure 6 As shown in the figure, the gray area is the background, the white area is the target, the red area is the internal pixel after edge distinction, and the blue area is the external pixel. By obtaining the values ​​of the left and right ends of a pixel and calculating their difference, it is determined whether the pixel is an internal pixel of the infrared small target or a boundary pixel. After obtaining the prediction map, for each pixel, it is necessary to calculate its similarity with the two predicted values ​​on the left and right to determine whether it is an internal pixel on the x-axis. By calculating the consistency of its neighbor labels, it is judged as an internal pixel if it is higher than the set threshold λ, and it is judged as an external pixel if it is lower than the threshold α. Pixels with moderate thresholds are relatively blurred and should be discarded in the loss calculation.

[0039] The in-box shape loss Shape-Loss divides each pixel into edge pixels and internal pixels. For the infrared small target image I, the pixels in the loss function are set to be inside the box, and the pixels outside the box are removed by the bounding box mask. In order to generate the label value, a directional difference convolution kernel v' is used to generate the shape map V, and the thresholds λ and α are set, where λ>α. This makes the points in the shape map V greater than λ defined as edge pixels, and the points less than α are defined as internal pixels, thereby successfully generating the label value. For the predicted value Y', the corresponding edge shape map U is also required. x In the x-axis direction, the predicted values ​​y of the pixels on the left and right sides of the pixel e are obtained. i,j and k,l , and then calculate the probability P(U e =1). The specific formula is as follows:

[0040] P(U e =1) =y i,j *y k,l +(1-y k,l )*(1-y i,j ), (6)

[0041] P(U e =0) = 1-P(U e =1). (7)

[0042] The same calculation method is applied to the y-axis direction. That is, the shape map H is generated by using the difference convolution kernel h' in another direction, and then the same calculation and operation as the shape map V are performed. That is, formulas (6) and (7) are applicable to the edge shape map U corresponding to the shape map V. x , which also applies to the edge shape graph U corresponding to the shape graph H y .

[0043] After obtaining the shape diagram, Figure 7 As shown, the Shape-Loss in the frame of the current infrared small target image is calculated, and the loss function is calculated as follows:

[0044] L shape =(I v＞λ log(P(U e =1))+(1-I v＜α )log(1-P(U e =0)) / (sum(V)) (8)

[0045] Among them, v refers to any label value in the shape graph V corresponding to the infrared small target image I; sum(V) means summing the label values ​​of the shape graph V.

[0046] To confirm this idea, the experiment visualized the relationship between the false positive rate of pixels and the threshold size on the infrared small target dataset NUDT-SIRST. Figure 8 It shows that when the thresholds λ and α change, the false positive rate of edge pixels decreases as the threshold λ increases, while the false positive rate of internal pixels decreases as the threshold α decreases. If the λ threshold is 0.2, only 2% of the pixels in the box are misjudged as positive edges. If λ is further increased, this proportion can be further reduced, but too large a threshold will reduce the number of supervised edges. In the experiment, it was found that the threshold is not sensitive to the final performance. Given a high proportion of positive edges, during training, it is safe to assume that all pixels with v≥λ are positive, and then only their losses are calculated. We will discard other edges during the loss calculation process. Similarly, when v≤α, all pixels are marked as internal pixels. Finally, the predicted value is also mapped to internal pixels and external pixels to obtain the in-box loss function Shape-Loss.

[0047] In the weakly supervised segmentation algorithm for infrared small targets, the combination of the framework based on the EMA algorithm and the weakly supervised loss function is explored. The EMA algorithm is often used to smooth time series data. By calculating the moving average, it suppresses noise and better captures trends. Borrowing the idea of ​​EMA, it is applied to the training framework of infrared small target segmentation. Formula (9) shows the change formula of the teacher model update, where f teacher (θ) and f student (θ) represents the specific parameters of the teacher model and the student model, and β represents the learning iteration amplitude of the teacher and student models, which is in the range of (0,1). In the iterative update of the algorithm, β is gradually approached to 1 from a very small value, steadily improving the reliability of the pseudo mask, thereby improving the learning stability of the model for small infrared targets:

[0048] f teacher (θ)=β*f student (θ)+(1-β)*f teacher ( (9)

[0049] Considering that the box supervision method can generate high-quality masks without mask annotations, a self-training strategy can be chosen to use these high-quality masks as pseudo labels to train a fully supervised instance segmentation method. Specifically, the BoxInst algorithm, which has been proven to be effective in the field of weakly supervised image segmentation, is used to generate pseudo instance masks on a given dataset, which only contain box annotations. For each input image X, let B and M represent the bounding box mask and the predicted mask, respectively. The present invention proposes a simple and effective weakly supervised segmentation algorithm for infrared small targets based on EAM, and its pseudo code is shown in Algorithm 1. The predicted mask M generated by the teacher model is corrected by the real bounding box mask B, and the corrected predicted mask M is assigned to the student model as the bounding box mask. The area outside the bounding box is deleted to correct the assigned instance, and then the student model is trained using the dataset with pseudo labels as the instance mask, and then the teacher model is updated by EMA and the student model, and iterative training produces a higher level of mask. This training process makes up for the lack of original supervision data with the help of high-quality pseudo labels, and improves the performance and generalization ability of the instance segmentation model.

[0050] Assuming that the teacher model is not accurate enough at the beginning, it is necessary to let the correct coarse label guide the model training at the beginning. As the teacher model becomes more accurate over time, the loss ratio of pseudo masks can be increased. On the one hand, the teacher model needs to be updated through the EMA technology, and the gradient coefficient is slowly adjusted to make it more accurate. In addition, a weakly supervised loss function is introduced. Through self-training, high-quality masks are used as pseudo labels to train the fully supervised instance segmentation method. This method reduces the demand for labeled data to a certain extent, so that the model can better adapt to the infrared small target detection task. Based on the algorithm EMA, the above infrared small target training method is optimized to design the infrared small target weak supervision training method. In Algorithm 1, each time the infrared small target image mask is changed, the algorithm needs to complete the model training. The number of cycles is uncertain and extremely time-consuming and unstable. The infrared small target weak supervision training based on EMA needs to initialize two models at the same time. The algorithm needs to run two models at the same time, and slightly change the model to obtain the mask after each training. This is an improved model of cyclic training, which greatly reduces the training time. For the infrared small target weak supervision algorithm, this EMA-based training method has significant advantages. By running two models in parallel, the image mask of small infrared targets can be obtained and updated more flexibly without being constrained by the uncertain number of training times and time overhead in traditional methods. The design of this improved model makes the algorithm more stable and efficient, and more adaptable to the characteristics of different small infrared targets. At the same time, by obtaining the mask through fine-tuning, the model is updated more quickly, which helps to improve the convergence and performance of the algorithm. In practical applications, this weakly supervised training method provides feasibility and reliability for the rapid and accurate recognition of small infrared targets.

[0051] Model includes

[0052]

[0053] By combining the EMA algorithm with a weakly supervised loss function, an innovative training framework is proposed to solve the data labeling problem in the infrared small target segmentation task and improve the model performance and generalization ability. The effectiveness of this framework will be verified in subsequent experiments, and its potential application and impact in the field of infrared small target segmentation will be explored. This study provides new ideas and methods for the development of the field of infrared small target segmentation. The ablation experiment conducts an in-depth study on the proposed EMA framework and weakly supervised loss function to evaluate their independent contributions and interactions in the infrared small target segmentation task. In the verification experiment of the EMA framework, different components of the EMA algorithm are gradually introduced, and the model performance is compared. The results show that the introduction of the EMA algorithm significantly improves the robustness and training stability of the model, effectively suppresses the noise interference in the training process, and makes the model better adapt to the characteristics of infrared small targets. Secondly, in the verification experiment of the weakly supervised loss function, the influence of different forms of weakly supervised signals on the model performance is analyzed. By comparing the models trained with different pseudo-labels, the importance of weakly supervised loss function in improving model performance is concluded. Especially when the labeled data is limited, the weakly supervised loss function provides more effective information for the model, which helps to improve the generalization ability of the model. Finally, the EMA framework and weakly supervised loss function were jointly experimented to verify their synergistic effect. The experimental results show that the EMA framework and weakly supervised loss function complement each other and work together on the infrared small target segmentation task, further improving the performance of the model. The results of these ablation experiments provide strong support for the rationality and effectiveness of the proposed method, and provide a practical solution for the research and application of infrared small target segmentation.

[0054] Experimental design and results analysis

[0055] Dataset

[0056] For the existing dataset, it is necessary to convert it into a box-annotated dataset to simulate the box mask dataset. Specifically, in this process, the same infrared small target image is used to convert the infrared small target annotation mask into a matrix with the same length and width, so as to better simulate the box-annotated dataset. Fig. 9 The process and results of this conversion using manual annotations to simulate coarse labels are shown.

[0057] This image mask conversion is designed to simulate weakly supervised labels and provide a more challenging dataset for the study of small infrared targets. In this process, the simulation of weakly supervised labels is achieved by converting the infrared small target annotation mask into the corresponding matrix form. The main purpose is to provide the model with approximate information about the target location in the absence of detailed annotation information. Through such weakly supervised labels, it can be closer to the high cost of annotation in actual application scenarios, while challenging the robustness of the model when dealing with data that lacks precise annotations.

[0058] When faced with a large-scale infrared small target image dataset, manually labeling the precise location of each small target is a tedious and expensive task. By introducing such weakly supervised labels, we can better cope with the challenges of data acquisition and labeling, and provide a more practical and feasible solution for the training of infrared small target segmentation models. Such a data processing strategy aims to promote researchers' in-depth discussion of the infrared small target segmentation problem and provide a more operational data foundation for the development of related fields.

[0059] Experimental parameter settings

[0060] The experiments were conducted on a Linux computer equipped with a 2.50GHz CPU, 11GB RAM, and a GeForce GTX 2080ti GPU. The model was implemented in Python and PyTorch, and the training method was trained on ID-Unet using the IoU loss function and optimized by the Adgrad method. The parameters λ and α of the in-box loss Shape-Loss were 0.2 and 0.03 respectively, and the parameter β of the EMA teacher model algorithm was 1 / N, where N is the number of iterations. The experiment initialized the weights and biases of the model using the Xavier method, and all models were trained from scratch.

[0061] Comparative experiment

[0062] Due to the differences between natural images and infrared small target images, most weakly supervised methods in the field of image segmentation are not optimized for infrared small target features. Table 1 shows the comparative experimental results of this method and other image weakly supervised segmentation algorithms on infrared small target datasets. Full supervision refers to the experimental results using manually annotated real masks, while bounding box masks refer to the experimental results using bounding box masks directly without weak supervision. The experimental results show that this method is superior to other methods on all three datasets.

[0063] The proposed method was compared with the weakly supervised segmentation method of natural images on the NUDT-SIRST, NUAA-SIRST and IRSTD-1k datasets. The method in this chapter adopts the infrared weakly supervised loss function PS-Loss, which includes the out-of-box loss Project Loss and the in-box loss Shape-Loss, and introduces the dual-model iterative algorithm based on EMA, and finally achieves satisfactory experimental results. Compared with the current natural image segmentation algorithm BoxTeacher, the IoU, nIoU and F1 of the method in this chapter on the NUDT-SIRST dataset are improved by 2.69%, 4.43% and 1.67% respectively, the IoU, nIoU and F1 on the NUAA-SIRST dataset are improved by 0.99%, 0.62% and 0.65% respectively, and the IoU and F1 on the IRSTD-1k dataset are improved by 2.05% and 1.48% respectively, while the nIoU is reduced by 0.32%. Our method optimizes and improves the performance on three infrared small target datasets, showing higher image segmentation evaluation indicators IoU and F1 scores. By adopting a combination of out-of-frame loss and in-frame loss, the information loss of infrared small targets is successfully reduced, thereby improving the accuracy and robustness of the segmentation results. In general, the method in this chapter shows obvious advantages in processing infrared small target segmentation tasks, and provides an effective solution to the challenges in this field.

[0064] Table 1 Experimental comparison results with other weakly supervised image segmentation methods

[0065]

[0066]

[0067]

[0068] The introduction of out-of-box loss effectively limits the infrared small target to the predefined box, preventing the loss of small target information during target segmentation. This ensures that the model focuses on the area of ​​interest, thereby improving the accuracy of segmentation. Experimental results show that the use of out-of-box loss reduces the impact of the model on the area outside the box, effectively reduces the model's sensitivity to the background, and improves overall stability. The introduction of in-box loss enhances the modeling and learning of the target shape by adjusting the shape of the small target in the box. In infrared small target segmentation, shape information is crucial to the success of the task. Experimental results show that the in-box loss effectively improves the model's accurate understanding of the shape of the small target, making the segmentation results more in line with the actual situation and further improving the performance of the model. BoxInf makes full use of the advantages of out-of-box loss and in-box loss, and achieves accurate segmentation of infrared small targets by comprehensively considering the target position and shape information. In comparison with other methods, it has achieved significant advantages in accuracy and robustness, showing high segmentation performance. Due to the improvement of the loss function, the BoxInf method is universal for different detection models and is suitable for various infrared small target segmentation models. Experimental results show that our method shows good generalization in different scenes, background noise and small target changes, showing its potential in practical applications. In summary, the BoxInf method performs well in the infrared small target segmentation task and has obvious advantages in improving segmentation accuracy and robustness.

[0069] Compared with other methods, the experimental results of our method are shown in the table. The most advanced methods in the field of image segmentation are usually difficult to train and segment effectively in the field of infrared small targets. Our method experimentally compares these methods on three datasets, and the experimental results are shown in the table. The results show that our method works best in weakly supervised infrared small target images. The most advanced algorithms usually have difficulty in effectively segmenting and subdividing infrared small targets that change in complex scenes encountered in the NUDT-SIRST dataset. In contrast, BoxInf shows greater robustness to these changes. In addition, compared with the most advanced algorithms, BoxInf can more accurately depict the target shape. Our innovative backbone design effectively eliminates noise by fusing deep information and adapts to various clutter backgrounds, target shapes, and target size challenges, thereby achieving excellent performance. Fig.10 As shown in Figure 2, we applied the BoxInf model and compared it with the state-of-the-art models using false positive rate and true positive rate. By plotting ROC curves for different thresholds, we analyzed and concluded that the BoxInf model outperformed the existing best models.

[0070] Ablation experiment

[0071] The present invention uses ablation experiments to verify the effectiveness of the infrared weak supervision method and the influence of some modules, including the influence of two IoU loss functions on the method, the combined ablation of weak supervision loss functions, and the influence of the iterative algorithm based on EMA.

[0072] Table 2 Ablation experiments with different loss function combinations

[0073]

[0074] (a) The impact of IoU loss function. In the ablation experiment, we focused on the impact of IoU loss function on the performance of our method. In Table 2, separate experiments were conducted using two different loss functions to gain a deeper understanding of their performance under experimental conditions. The experimental results show that when the labels are inaccurate for the method, using the SoftIoU-Loss function can achieve better results and alleviate the negative impact of inaccurate labels on model performance. On the contrary, when the labels are accurate for the method, using DiceIoU-Loss performs better. In this case, DiceIoU-Loss can better capture the similarity of the target area, thereby improving the accuracy of segmentation. This series of experiments proves that choosing the appropriate loss function in different scenarios is crucial to the performance of infrared weakly supervised methods. The selected loss function directly affects the model's ability to adapt to label uncertainty and accuracy. Therefore, in practical applications, choosing the most appropriate loss function according to the characteristics of the specific scenario is an important factor in ensuring the superior performance of the model.

[0075] Table 3 Ablation experiment of BoxInf model module composition

[0076]

[0077] (b) Contribution of each module. Contribution of each loss term. The final loss function consists of two loss functions, and Table 3 shows the contribution of each loss term. Even if only the first projection term Project-Loss is used, satisfactory performance can be obtained. This is much higher than using only the bounding box mask. By further using the proposed edge loss Shape-Loss, the weakly supervised segmentation ability of small infrared targets can be further improved. Finally, the two loss functions are integrated into the iterative algorithm based on EMA to further improve the model performance. This series of experimental results show that the reasonable combination and adjustment of different loss terms have a significant impact on the improvement of overall performance. The weakly supervised method using all modules improves IoU, nIoU and F1 by 1.25%, 1.22% and 0.78% respectively compared with the method without EMA; IoU, nIoU and F1 are improved by 3.42%, 3.78% and 2.14% respectively compared with the weakly supervised method using only Project loss function; IoU, nIoU and F1 are improved by 23.67%, 18.45% and 6.74% respectively compared with the method using only bounding box mask for full supervision training. The experimental results prove the importance of each module in our method.

[0078] Table 4 Similarity between preprocessed data and true masks

[0079]

[0080]

[0081] (c) The impact of different data preprocessing methods. In the dataset preprocessing stage, the dataset is processed through a series of operations, and the processed dataset is directly used for the weak supervision task of infrared small target segmentation. In this task, the performance of the method is evaluated by indicators such as hit rate, IoU and nIoU. The higher the value of these indicators, the better the effect of subsequent weak supervision training. A variety of methods were used for experiments, including the GrabCut method, which uses the GrabCut algorithm to segment small targets in the box. In addition, the box method is introduced, that is, the pixel values ​​in the box annotation are set as target pixels. GrabCut is combined with the Box method. This method first uses the GrabCut algorithm to segment the small targets in the box, and then uses the box method to generate masks for the undetected small targets. Through the comparative experiments of these methods, it is aimed to find the best data processing and segmentation method in the weak supervision task to optimize the performance of infrared small target segmentation.

[0082] Table 5 Final training results of different preprocessing methods

[0083]

[0084] (d) Experimental results of data preprocessing with different methods. As shown in Table 5, the hit rate of using only GrabCut is far inferior to that of using bounding box mask and combination method, the algorithm cannot segment all small objects due to its limitations. The experimental results of the method are obviously not good. In Table 5, we experimentally compare the box using bounding box mask for full supervision training and the BoxGrabCut method that combines the advantages of the two methods. The experimental results show that BoxGrabCut using the combination method outperforms the box using bounding box mask for full supervision training on all three datasets. On the dataset NUDT-SIRST, IoU, nIoU and F1 are improved by 8.7%, 3.62% and 6.71% respectively; on the dataset NUAA-SIRST, IoU and F1 are improved by 0.55% and 0.38% respectively, while nIoU is reduced by 0.5%; on the dataset IRSTD-1K, IoU, nIoU and F1 are improved by 1.82%, 4.25% and 1.45% respectively. The experimental results prove the effectiveness of the combined method in image preprocessing, which can effectively improve the accuracy of subsequent weak supervision methods.

[0085] Table 6 Final effect of different β pair methods in EMA iteration algorithm

[0086]

[0087] (e) The impact of the weak supervision parameter λ on the experimental results. Table 6 shows the different effects of the λ parameter in the weak supervision method in this chapter. In the experiment, we set the maximum value of λ to 1, and experimented with the impact of different λ values ​​on the total number of training iterations N and the performance indicator IoU. Specifically, we selected experimental settings when λ was 1 / N, 2 / N, and 3 / N, and observed their impact on IoU. The experimental results show that when λ is 1 / N, IoU is only improved by 0.2% compared to when λ is 2 / N, and by 0.35% compared to when λ is 3 / N. This shows that the impact of the λ parameter on the method is not particularly significant. It is worth noting that the choice of λ may need to be fine-tuned in other experiments to better adapt to different infrared small target datasets. This flexibility makes our method more versatile and adaptable, and can achieve better performance in dealing with different data scenarios.

[0088] Table 7 Ablation experiments of loss function combinations

[0089]

[0090]

[0091] (f) Impact of annotation errors. In the process of coarse label annotation, some errors may be introduced due to various reasons. In order to test the robustness of our method to such errors, we conducted a series of simulation experiments, as shown in Table 7. Specifically, we simulated the errors that may occur in the annotation process by introducing an offset. The offset here is used to indicate the degree of error, where +1 means an extra circle of pixel values, +2 means an extra circle of pixel values, and +3 means an extra circle of pixel values. Through such simulations, we can evaluate the robustness of our method in the face of annotation errors and provide more comprehensive and reliable results for our research.

[0092] Experiments show that the infrared small target segmentation task is very sensitive to the accuracy of annotation. A small infrared target may only have a few pixel values. When the pixel values ​​are artificially expanded each time, the detection effect of the infrared small target weak supervision algorithm will be greatly reduced, proving that the accuracy of annotation will greatly affect the final segmentation efficiency. Experiments show that if the accuracy of manual annotation decreases due to the increase in data volume, it will also affect the final model performance, further highlighting the importance of weak supervision.

[0093] Table 8 Ablation experiment of weighted values ​​of PSLoss loss function

[0094]

[0095] (g) The influence of the weighted value of the loss function. The final loss function consists of two loss functions, which are divided into out-of-box loss and in-box loss, where the results of different weighted values ​​are tested. Table 8 shows the influence of different in-box loss weighted values ​​on the experimental results. The weighted value of the loss function plays a vital role in the training process, affecting the convergence speed and final performance of the model. By adjusting the weighted value of the in-box loss, the importance of the model in capturing the internal details and overall shape of the target can be balanced. The experimental results show that a suitable weighted value of the in-box loss can better promote the model to learn the internal details of the target, but too high a weighted value may cause the model to over-focus on the internal details of the target and ignore the overall shape, thereby affecting the accuracy of the segmentation. Therefore, in practical applications, it is necessary to reasonably select the weighted value of the loss function according to the characteristics of the specific task and dataset to achieve the best segmentation effect.

[0096] In summary, the present invention aims to solve the problems of small target size, sparse number and difficulty in labeling in infrared small target images. The comprehensive weakly supervised learning framework designed by the present invention simulates fine labeling by introducing the transformation of image masks, thereby reducing the complexity of labeling. Three different data sets were used for experimental verification to prove the effectiveness of BoxInf. The experimental results show that BoxInf has achieved a good performance improvement in the infrared small target segmentation task. By adopting a training strategy combining coarse labels and fine labels, it performs well in both target detection and segmentation accuracy. Especially in the face of complex scenes and various clutter backgrounds, it shows better robustness than traditional methods. By comparing the experimental results of different data sets, the generalization and adaptability of the algorithm are verified. The innovative application of weakly supervised methods to the infrared small target segmentation task provides a new idea for solving the infrared small target segmentation. By designing a weakly supervised method, the labeling cost is reduced to increase the infrared small target data set, solving the current development bottleneck of the infrared small target task, and making the infrared small target segmentation technology more practical. The method in this chapter not only shows significant advantages in labeling cost, but also provides an innovative solution for the infrared small target detection task on the basis of maintaining segmentation quality. The introduction of this new method will promote the continuous development of the field of infrared small target segmentation and provide more feasible and economical solutions for practical applications.

[0097] This paper proposes a simple, high-performance and end-to-end weakly supervised segmentation algorithm for infrared small target bounding boxes. This method is based on the ID-UNet model in Chapter 3 and proposes a high-performance weakly supervised method that can achieve mask-level instance segmentation of infrared small targets using only bounding box annotations used for training. Only for newly acquired infrared small target data, only the data needs to be annotated with annotation boxes, which greatly reduces the annotation time. Considering the huge cost of labeled instance segmentation, weakly supervised instance segmentation using image-level labels or bounding boxes has received widespread attention. However, we are also aware of the challenges of the algorithm in dealing with some special cases, such as small targets with strange shapes that cannot effectively and accurately obtain shape edges. Future work can further improve the performance of the algorithm by introducing more contextual information or adopting an attention mechanism. Overall, our research provides a new solution to the problem of scarce datasets in the infrared small target segmentation task and provides favorable help for the practical application of segmentation algorithms.

Claims

1. A weakly supervised infrared small target segmentation method, characterized in that: The input is only the infrared small target image and the infrared small target bounding box data, and the final output result is the segmentation mask of the small target in the infrared small target image. The method includes model pre-training, model loading, teacher model generation of pseudo mask, student model training and dual-model EMA, among which the teacher model generation of pseudo mask, student model training and dual-model EMA need to be repeated for multiple iterations; specifically, the following steps are included: Step 1: Use the infrared small target image and the corresponding bounding box mask to pre-train the model in the PS-Loss loss function; Step 2: Copy the parameters of the pre-trained model to the teacher model and the student model respectively; fix the parameters of the teacher model for use in subsequent steps; Step 3: The infrared small target image generates a pseudo mask through the teacher model; Step 4: Compare the pseudo mask with the bounding box mask and remove the pseudo mask outside the bounding box. This is because the infrared small target has been preset to be within the bounding box. Step 5: Use the bounding box mask of step 1 and the pseudo mask processed in step 3 as the guidance labels of the loss function to control the training of the student model and obtain the output of the student model; in the process of training the student model through the loss function, the bounding box mask uses the PS-Loss loss function, and the pseudo mask uses the general image segmentation loss function DiceIoU-Loss loss function; Step 6: Input the student model and the teacher model into the EMA algorithm to update the teacher model, then return to step 4 and perform multiple repeated iterative training. The final student model is used as the infrared small target segmentation model, and the prediction mask output by the final student model is used as the final output for evaluation.

2. The infrared small target segmentation method based on weak supervision according to claim 1 is characterized in that: The loss function PS-Loss of weak supervision of small infrared targets is composed of the projection loss Project-Loss outside the frame and the shape loss Shape-Loss inside the frame. ps Defined as: THE ps =L project +λL shape (1) Where L project is the out-of-frame projection loss, with a weight value of 1; L shape It is the in-box shape loss, and the weight value defaults to λ.

3. The infrared small target segmentation method based on weak supervision according to claim 2 is characterized in that: The loss function formula of the out-of-box projection loss Project-Loss is as follows: Proj x (m)=max y (m)=m x , (2) Proj y (b)=max x (b)=b x , (3) L Project =L DiceIoU (m x , b x ) + L DiceIoU (m y , b y ), (4) The loss function is to minimize the difference between the bounding box mask and the predicted mask, and the corresponding projection m on the x-axis and y-axis is obtained by calculating the maximum value of the bounding box mask on the y-axis and x-axis x and m y , predict the corresponding projections b of the mask on the x-axis and y-axis x and b y , max y (m) represents the mapping of the bounding box mask on the y-axis, and max x (b) represents the mapping of the predicted mask on the x-axis; L DiceIoU Represents the DiceIoU-Loss function.

4. The infrared small target segmentation method based on weak supervision according to claim 3 is characterized in that: The in-frame shape loss Shape-Loss uses a directional difference convolution kernel to extract the shape features of small infrared targets and pre-sets labels based on the shape features to generate prior labels. For infrared small target image I∈R w*h , use the x-axis and y-axis direction differential convolution kernels v' and h' to extract the shape feature map U of the infrared small target in two directions x and U y , and then use the square sum of the two variables to generate the shape; through the threshold setting, the larger and smaller values ​​are set as edge pixels and internal pixels respectively, and the edge shape graph U∈R is generated through the edge pixels and internal pixels w*h , guide the model to perform training optimization; the formula for obtaining the edge shape graph is as follows: The in-frame shape loss Shape-Loss divides each pixel into edge pixels and internal pixels. For the infrared small target image I, the pixels in the loss function are set to be in the frame, while the pixels outside the frame are removed by the bounding box mask. In order to generate the label value, a directional difference convolution kernel v' is used to generate the shape map V, and the thresholds λ and α are set, where λ>α. This makes the points in the shape map V greater than λ defined as edge pixels, and the points less than α are defined as internal pixels, thereby successfully generating the label value. For the predicted value Y', the corresponding edge shape map U is also required. x ; In the x-axis direction, by obtaining the predicted value y of the pixel points on the left and right ends of the pixel point e i,j and k,l , and then calculate the probability P(U e =1); the formula is as follows: P(U e =1) = and i,j *and k,l + (1- and k,l )*(1- and i,j ), (6) P(U e =0) = 1-P(U e =1). (7) The same calculation method is applied to the y-axis direction; that is, the shape map H is generated using the difference convolution kernel h' in another direction, and then the same calculation and operation as the shape map V are performed; that is, formulas (6) and (7) are applicable to the edge shape map U corresponding to the shape map V. x , which also applies to the edge shape graph U corresponding to the shape graph H y ; After obtaining the shape map, the Shape-Loss within the frame of the current infrared small target map is calculated. The loss function is calculated as follows: L shape =(I v＞λ log(P(U e =1))+(1-I v＜α )log(1-P(U e =0)) / (sum(V)) (8) Among them, v refers to any label value in the shape graph V corresponding to the infrared small target image I; sum(V) means summing the label values ​​of the shape graph V.

5. The infrared small target segmentation method based on weak supervision according to claim 4 is characterized in that: This method combines the EMA algorithm-based framework with the weakly supervised loss function; Formula (9) shows the changing formula for updating the teacher model, where f teacher (θ) and f student (θ) represents the specific parameters of the teacher model and the student model, β represents the learning iteration amplitude of the teacher and student models, and its interval is (0,1]; in the iterative update, β is gradually approached to 1 from a very small value, steadily improving the reliability of the pseudo mask, thereby improving the learning stability of the model for infrared small targets: f teacher (θ)=β*f student (θ)+(1-β)*f teacher (i). (9) These high-quality masks are used as pseudo labels to train a fully supervised instance segmentation method. Specifically, the BoxInst algorithm, which has been proven to be effective in the field of weakly supervised image segmentation, is used to generate pseudo instance masks on a given dataset, which only contain box annotations. For each input image X, let B and M represent the bounding box mask and the predicted mask, respectively. The predicted mask M generated by the teacher model is corrected by the true bounding box mask B, and the corrected predicted mask M is assigned to the student model as the bounding box mask. The area outside the bounding box is deleted to correct the assigned instance, and then the pseudo-label is used as the dataset of the instance mask to train the student model. The teacher model is then updated through EMA and the student model, and the iterative training produces a higher level of mask.

Images