Infrared small target detection method based on label evolution and global-local system

Abstract

The invention relates to an infrared small target detection method based on label evolution and a global-local system. The method comprises the following steps: S1, constructing a local-global interactive self-attention network model; s2, constructing a multi-direction and multi-scale neural network model; s3, training the self-attention network model and the neural network model by using a weak supervision training method based on label evolution; s4, obtaining an enhanced data set; and S5, performing fine tuning on the self-attention network infrared small target detection model and the neural network infrared small target detection model based on the enhanced data set training to obtain an actual to-be-detected image, and inputting the actual to-be-detected image into the fine-tuned self-attention network infrared small target detection model and the fine-tuned neural network infrared small target detection model. And an integrated detection result is obtained. Compared with the prior art, the method has the advantages that higher precision is achieved in a weak supervision infrared small target detection task, and the like.

Description

Technical Field

[0001] The invention relates to the technical field of infrared small target detection, and in particular to an infrared small target detection method based on label evolution and a global-local system. Background Art

[0002] Infrared small target detection is a task to separate small targets in complex scenes. It has important application significance in many fields such as military reconnaissance, traffic monitoring and maritime rescue. With the advent of the big data era, a large number of unknown samples bring huge labeling costs to the field of infrared small target detection. Existing infrared small target detection methods are difficult to achieve the required performance when the labeled samples are limited. Therefore, infrared small target detection under single coarse point supervision has become a problem that has attracted much attention.

[0003] Small infrared targets are small, textureless, and unstable in shape, and are easily submerged in complex background clutter. In view of these characteristics, existing infrared small target detection methods usually adopt feature fusion strategies, such as multi-scale feature fusion and multi-level feature fusion convolutional neural networks, but these methods only focus on the local saliency of small infrared targets and ignore the modeling of the global background. At the same time, existing methods are also limited by the lack of public datasets with pixel-by-pixel supervision, and there are bottlenecks in performance improvement and generalization.

[0004] Reference [1] B. Li et al., "Dense Nested Attention Network for Infrared Small Target Detection," in IEEE Transactions on Image Processing, vol. 32, pp. 1745-1758, 2023 and reference [2] H. Sun, J. Bai, F. Yang and X. Bai, "Receptive-Field and Direction Induced Attention Network for Infrared Dim Small Target Detection With a Large-Scale Dataset IRDST," in IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-13, 2023 both utilize step-by-step feature extraction and fusion modules and local feature enhancement modules to achieve the effect of extracting enhanced multi-scale local features. However, both methods are insufficient in extracting long-distance global contextual information and cannot focus on the global semantic differences between small infrared targets and background clutter. Summary of the invention

[0005] The purpose of the present invention is to provide an infrared small target detection method based on label evolution and global-local system in order to achieve higher accuracy in weakly supervised infrared small target detection tasks.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for detecting infrared small targets based on label evolution and global-local model integration, the method comprising the following steps:

[0008] S1. Construct a local-global interactive self-attention network model;

[0009] S2. Construct a multi-directional and multi-scale neural network model;

[0010] S3. Use the weak supervision training method based on label evolution to train the self-attention network model and the neural network model to obtain the self-attention network infrared small target detection model and the neural network infrared small target detection model;

[0011] S4, using the original data set, extracting a noise image from it, and mixing the noise image with the original data set to obtain an enhanced data set;

[0012] S5. Fine-tune the self-attention network infrared small target detection model and the neural network infrared small target detection model based on the enhanced data set training to obtain the actual image to be detected, and input the actual image to be detected into the fine-tuned self-attention network infrared small target detection model and the neural network infrared small target detection model to obtain the detection result.

[0013] Furthermore, the output of the local-global interactive self-attention network model is: the feature D of the channel cross attention module input to the self-attention network model i The output obtained after the decoding layer of the self-attention network model, where feature D i The calculation process is:

[0014] O=F i (I 1 ,I 2 ,I 3 ,I 4 )

[0015] D i =E i +FM i (O)(i=1,2,3,4)

[0016] Among them, E i is the local deep feature of the i-th scale in a single frame infrared image, I i For Ei The features obtained after patch embedding, F represents a spatial-channel cross attention module, FM i Represents the feature mapping operation, FM i (O) represents the features obtained by feature mapping, E i and FM i (O) Through residual connection.

[0017] Furthermore, the output of the multi-directional and multi-scale neural network model is enhanced local features, and the calculation process of the enhanced local features is:

[0018] Feature map F after enhancing the target e After passing through a feature extraction module consisting of four receptive fields of different scales, low-level features and high-level features are obtained. After being spliced ​​along the channel dimension, the low-level features and high-level features are successively passed through the channel attention and spatial attention modules to obtain enhanced local features.

[0019] Furthermore, the feature map F after the enhancement target e The calculation process is:

[0020] d k =Conv k (x)

[0021]

[0022] F e =DT(x)·F l

[0023] Among them, Conv k represents a fixed convolutional layer, d k represents the grayscale difference of the boundary pixel, σ represents the sigmoid function, x represents the input, and F l Represents shallow features.

[0024] Furthermore, the specific steps of S3 are:

[0025] Get the current labels L of the two models n , n represents the nth label update, according to the current label L n Zhongzheng pixel area The centroid of the current label L is cropped n and the corresponding current prediction result P n Then select local neighborhood candidate pixels from the d×d size region i represents the i-th target in the label;

[0026] The candidate pixels With the current label L nThe 8 connected areas without intersection of the positive pixel area are excluded to obtain the candidate pixels for eliminating false alarms The positive pixel area represents the label L n The area composed of pixels with a value greater than 0.5;

[0027] Candidate pixels for eliminating false alarms and the current label L n Perform average weighted summation to obtain the new label for the n+1th round of training.

[0028] Furthermore, the candidate pixels for:

[0029]

[0030] Among them, T adapt represents the adaptive threshold, Indicates the current prediction result P n The i-th target in .

[0031] Furthermore, the adaptive threshold is:

[0032]

[0033] Among them, h, w are the length and width of the input image, and r is 1.5×10 -3 The constant, T b is the minimum value of the dynamic threshold, k is the parameter that controls the speed at which the dynamic threshold increases with the increase of n, Represents the positive pixel area of ​​the i-th object in the label.

[0034] Furthermore, the new label of the n+1th round of training is:

[0035]

[0036] in, is the new label for the n+1th round of training, Corresponding to the current prediction result P n Candidate pixels after eliminating false alarms.

[0037] Furthermore, the specific steps of S4 are:

[0038] Randomly select a noisy image A from the original data set A i and the image B to be enhanced i , where A i ≠B i , image A i The length and height of image A are H and W respectively. i There is no image patch that meets the conditionsi , then select the new image A i Until the required conditions are met;

[0039] The required conditions are: i Select the image patch with length and width of H / 8 and W / 8 respectively i , there exist β and γ such that the image block Patch i The variance and mean of <Var(Patch i )<β,0 <Mean(Patch i )<γ, then the required conditions are met, where Var represents variance and Mean represents mean;

[0040] For an image A that satisfies the required conditions i Patch i Scale to a length and height of H and W respectively, and patch the image i Multiply by the scale factor δ and compare with the image B to be enhanced i Add and repeat the above steps until all images in the original data set are traversed to obtain the enhanced data set.

[0041] Furthermore, based on the enhanced dataset training, the self-attention network infrared small target detection model and the neural network infrared small target detection model are fine-tuned as follows:

[0042] The enhanced dataset is used as the training set, and the new labels obtained from the last round of training are used as supervision to fine-tune the self-attention network infrared small target detection model and the neural network infrared small target detection model.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention trains a multi-directional and multi-scale neural network model and a local-global interactive self-attention network model through a weakly supervised learning method of label evolution, overcomes the shortcomings of the existing model that ignores the attention to effective global information modeling and relies on fully supervised annotation information, and is accompanied by the extraction of global semantic difference features between small infrared targets and background clutter and weaker pixel annotation requirements. At the same time, the present invention also applies data enhancement to further fine-tune the model, which can achieve higher accuracy in the weakly supervised infrared small target detection task. Under the condition of using 5.43% of the fully supervised annotated pixels, the target-level detection probability on the WideIRSTD-Weak data set can reach 69.70%, which is 1.12% higher than the detection probability of the baseline model obtained by fully supervised training, and 6.06% higher than the detection probability of the baseline model obtained by weakly supervised training under the same conditions; the pixel-level detection probability can reach 35.22%, and the performance reaches 94% of the baseline model, which is 5.95% higher than the detection probability of the baseline model obtained by weakly supervised training under the same conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0047] The present invention proposes an infrared small target detection method based on label evolution and global-local system. The flow chart of the method is as follows: Figure 1 The method comprises the following steps:

[0048] S1. Construct a local-global interactive self-attention network model;

[0049] S2. Construct a multi-directional and multi-scale neural network model;

[0050] S3. Use the weak supervision training method based on label evolution to train the self-attention network model and the neural network model to obtain the self-attention network infrared small target detection model and the neural network infrared small target detection model;

[0051] S4, using the original data set, extracting a noise image from it, and mixing the noise image with the original data set to obtain an enhanced data set;

[0052] S5. Fine-tune the self-attention network infrared small target detection model and the neural network infrared small target detection model based on the enhanced data set training to obtain the actual image to be detected, and input the actual image to be detected into the fine-tuned self-attention network infrared small target detection model and the neural network infrared small target detection model to obtain the detection result.

[0053] Based on the idea of ​​deep learning, the present invention uses a local-global interactive self-attention network model to extract the global semantic difference features between small infrared targets and background clutter, uses a multi-directional and multi-scale neural network model to extract enhanced multi-scale local features, and combines a weakly supervised training method based on label evolution to propose an infrared small target detection method based on label evolution and global-local system. This method solves the limitation of the existing infrared small target detection problem that is highly dependent on pixel-by-pixel supervision data. Under the condition of using 5.43% of the fully supervised annotated pixels, the target-level detection probability on the WideIRSTD-Weak dataset can reach 69.70%, which is 1.12% higher than the detection probability of the baseline model obtained by fully supervised training, and 6.06% higher than the detection probability of the baseline model obtained by weakly supervised training under the same conditions; the pixel-level detection probability can reach 35.22%, and the performance reaches 94% of the baseline model, which is 5.95% higher than the detection probability of the baseline model obtained by weakly supervised training under the same conditions.

[0054] The present invention uses a label evolution weakly supervised learning method to train a multi-directional and multi-scale neural network model and a local-global interactive self-attention network model, overcoming the shortcomings of existing models that ignore the focus on effective global information modeling and rely on fully supervised annotation information, accompanied by the global semantic difference feature extraction of infrared small targets and background clutter and weaker pixel annotation requirements. At the same time, the method also applies data enhancement to further fine-tune the model, which can achieve higher accuracy in weakly supervised infrared small target detection tasks.

[0055] In S1, the local-global interaction-based self-attention network model uses a spatial-channel cross-attention module with long-distance jump connections to encode the semantic differences between small infrared targets and background clutter; in S2, the multi-directional and multi-scale neural network model uses a multi-directional target enhancement module to enhance target information in shallow features, and further uses a multi-receptive field feature extraction module to learn multi-scale local features.

[0056] In S5, the two models obtained in S3 are fine-tuned using the data obtained in S4. After the fine-tuning is completed, for an infrared image to be detected, it is input into model 1 and model 2, and the model output results are integrated to obtain the predicted pixel area corresponding to the infrared small target.

[0057] In S1, the existing local-global interactive self-attention network model framework is used to remove its multi-scale deep supervision strategy, and only the output of the last layer of the decoding layer is used as the detection result of the model. The specific formula is as follows:

[0058] O=F i (I 1 ,I 2 ,I 3 ,I 4 )

[0059] D i =E i +FM i (O)(i=1,2,3,4)

[0060] Among them, E i is the local deep feature of the i-th scale in a single frame infrared image, which is obtained by passing the input image through i residual modules and 1 maximum pooling layer. i By E i After patch embedding, F represents a spatial-channel cross-attention module that can encode the semantic difference between infrared small targets and background clutter. i It represents the feature mapping operation, which is used to transform the output of F module O i Restore to and E i Same size. i and O i Through residual connection, the feature D i It is sent to the channel cross-attention module for local-global feature fusion, and then sent to the decoding layer to obtain the detection result of model 1.

[0061] In S2, the existing multi-directional and multi-scale neural network model is used to extract enhanced multi-scale local features. The specific formula is as follows:

[0062] d k =Conv k (x)

[0063]

[0064] F e =DT(x)·F l

[0065] The input image first passes through 8 fixed convolutional layers Conv k , get the grayscale difference d between the center pixel and the boundary pixels in 8 directions k , d kThe multi-directional attention weights are generated by multiplication and addition, where σ represents the sgmoid function. The feature map F after the enhanced target is obtained by dot multiplication with the shallow features e In the multi-receptive field feature extraction module, F e First, a feature extraction module consisting of four receptive fields of different scales is used to obtain low-level features containing more local information and high-level features containing more background information. After the obtained feature maps are spliced ​​along the channel dimension, they are further passed through the channel attention and spatial attention modules to extract enhanced local features of multi-scale fusion.

[0066] In S3, a weakly supervised training method based on label evolution is used to train models 1 and 2 in steps 1 and 2, and the evolved labels L1 and L2 are obtained. First, with single-point labels as supervision, after the initial training of T epochs, the first round of label update is performed, the current label is updated using the prediction results, and the updated label is generated and used to supervise network training until the next round of label update. After that, a round of label update is performed after each training of f epochs. For the nth label update, follow the following steps.

[0067] (a) Candidate pixel extraction. For each image’s current label L n , according to the positive pixel area (Note: Positive pixels refer to the label L n The centroid of the pixels with pixel values ​​greater than 0.5 in the image is used to crop the current label L n and the corresponding current prediction result P n The d×d size region is then used with an adaptive threshold T adapt Extract local neighborhood candidate pixels. The specific formula is as follows:

[0068]

[0069] Among them, h, w are the length and width of the input image, respectively, and r is a fixed value of 1.5×10 -3 The constant T b It is the minimum value of the dynamic threshold. k is used to control the speed at which the dynamic threshold increases with the increase of n.

[0070] (b) Elimination of false alarm pixels. The eight connected regions where the candidate pixels have no intersection with the positive pixels of the label are excluded. This step is to reduce the impact of false alarm detection results generated during the training process on the final prediction results.

[0071] (c) Update the label. We perform a weighted average summation of the local neighborhood candidate pixels and the current label to achieve label update. The specific formula is as follows:

[0072]

[0073] in, is the updated label, which is used as the new label for the n+1th round of training.

[0074] In S4, a noise source image A is randomly selected from the original dataset A. i and the image B to be enhanced i (A i ≠B i ), Picture A i The length and height are H and W respectively. i Select the image patch with length and width of H / 8 and W / 8 respectively i , there exist β and γ such that the image block Patch i The variance and mean of <Var(Patch i )<β,0 <Mean(Patch i )<γ. If image A i There is no image patch that meets the conditions i , then select the new picture A i Until the required conditions are met. The next step is to patch the image i Scale to a length and height of H and W respectively, multiply its pixel value by the proportional coefficient δ and the image B to be enhanced i Add. Operate one by one until all images in the data set are enhanced, and obtain the enhanced data set.

[0075] In S5, the enhanced image obtained in S4 is used as the input image to fine-tune Model 1 and Model 2 obtained by weak supervision training in S3. During the fine-tuning process, the label obtained by the last round of update in S3 is used as supervision to further obtain an optimized infrared small target detection model.

[0076] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for infrared small target detection based on label evolution and global-local model integration, characterized in that: The method comprises the following steps: S1. Construct a local-global interactive self-attention network model; S2. Construct a multi-directional and multi-scale neural network model; S3. Use the weak supervision training method based on label evolution to train the self-attention network model and the neural network model to obtain the self-attention network infrared small target detection model and the neural network infrared small target detection model; S4, using the original data set, extracting a noise image from it, and mixing the noise image with the original data set to obtain an enhanced data set; S5. Fine-tune the self-attention network infrared small target detection model and the neural network infrared small target detection model based on the enhanced data set training to obtain the actual image to be detected, and input the actual image to be detected into the fine-tuned self-attention network infrared small target detection model and the neural network infrared small target detection model to obtain the detection result.

2. The infrared small target detection method based on label evolution and global-local system according to claim 1 is characterized in that: The output of the local-global interactive self-attention network model is: the feature D of the channel cross attention module input to the self-attention network model i The output obtained after the decoding layer of the self-attention network model, where feature D i The calculation process is: O=F i (I1, I2, I3, I4) D i =E i +FM i (O)(i=1,2,3,4) Among them, E i is the local deep feature of the i-th scale in a single frame infrared image, I i For E i The features obtained after patch embedding, F represents a spatial-channel cross attention module, FM i Represents the feature mapping operation, FM i (O) represents the features obtained by feature mapping, E i and FM i (O) Through residual connection.

3. The infrared small target detection method based on label evolution and global-local system according to claim 1 is characterized in that: The output of the multi-directional and multi-scale neural network model is enhanced local features, and the calculation process of enhanced local features is: Feature map F after enhancing the target e After passing through a feature extraction module consisting of four receptive fields of different scales, low-level features and high-level features are obtained. After being spliced ​​along the channel dimension, the low-level features and high-level features are successively passed through the channel attention and spatial attention modules to obtain enhanced local features.

4. The infrared small target detection method based on label evolution and global-local system according to claim 3 is characterized in that: The feature map F after the enhancement target e The calculation process is: d k =C0nv k (x) F e =DT(x)·F l Among them, Conv k represents a fixed convolutional layer, d k represents the grayscale difference of the boundary pixel, σ represents the sigmoid function, x represents the input, and F l Represents shallow features.

5. The infrared small target detection method based on label evolution and global-local system according to claim 1 is characterized in that: The specific steps of S3 are: Get the current labels L of the two models n , n represents the nth label update, according to the current label L n Zhongzheng pixel area The centroid of the current label L is cropped n and the corresponding current prediction result P n Then select local neighborhood candidate pixels from the d×d size region i represents the i-th target in the label; The candidate pixels With the current label L n The 8 connected areas without intersection of the positive pixel area are excluded to obtain the candidate pixels for eliminating false alarms The positive pixel area represents the label L n The area composed of pixels with a value greater than 0.5; Candidate pixels for eliminating false alarms and the current label L n Perform average weighted summation to obtain the new label for the n+1th round of training.

6. The infrared small target detection method based on label evolution and global-local system according to claim 5, characterized in that: The candidate pixels for: Among them, T adapt represents the adaptive threshold, Indicates the current prediction result P n The i-th target in .

7. The infrared small target detection method based on label evolution and global-local system according to claim 6 is characterized in that: The adaptive threshold is: Among them, h, w are the length and width of the input image, and r is 1.5×10 -3 The constant, T b is the minimum value of the dynamic threshold, k is the parameter that controls the speed at which the dynamic threshold increases with the increase of n, Represents the positive pixel area of ​​the i-th object in the label.

8. The infrared small target detection method based on label evolution and global-local system according to claim 5, characterized in that: The new label for the n+1th round of training is: in, is the new label for the n+1th round of training, Corresponding to the current prediction result P n Candidate pixels after eliminating false alarms.

9. The infrared small target detection method based on label evolution and global-local system according to claim 1, characterized in that: The specific steps of S4 are: Randomly select a noisy image A from the original data set A i and the image B to be enhanced i , where A i ≠B i , image A i The length and height of image A are H and W respectively. i There is no image patch that meets the conditions i , then select the new image A i Until the required conditions are met; The required conditions are: i Select the image patch with length and width of H / 8 and W / 8 respectively i , there exist β and γ such that the image block Patch i The variance and mean of <Var(Patch i )<β,0 <Mean(Patch i )<γ, then the required conditions are met, where Var represents variance and Mean represents mean; For an image A that satisfies the required conditions i Patch i Scale to a length of H and a height of W respectively, and patch the image i Multiply by the scale factor δ and add to the image B to be enhanced i Add and repeat the above steps until all images in the original data set are traversed to obtain the enhanced data set.

10. The infrared small target detection method based on label evolution and global-local system according to claim 6, characterized in that: Based on the enhanced data set training, the self-attention network infrared small target detection model and the neural network infrared small target detection model are fine-tuned as follows: The enhanced dataset is used as the training set, and the new labels obtained from the last round of training are used as supervision to fine-tune the self-attention network infrared small target detection model and the neural network infrared small target detection model.

Images