Infrared small target recognition method based on color shape prompt learning

Abstract

The application discloses an infrared small target recognition method based on color shape prompt learning, comprising the following steps: S1, dividing an image data set of an infrared small target into a training set and a test set; S2, inputting images of the training set into a backbone network of an infrared small target detection network model to extract multi-scale features of the infrared small target images; S3, performing multi-scale feature enhancement on the multi-scale features of the infrared small target images; S4, performing cross-domain fusion through a double probability alignment DPA fusion framework to obtain multi-scale information features; S5, sending the multi-scale information features into a dynamic detection head DynamicHead for detection; the detection result is evaluated by a normalized Wasserstein distance NWD and an adaptive threshold focus loss ATFL; and S6, obtaining a final infrared small target detection result. The application has good reliability and robustness, and improves the detection performance of the infrared small target.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared small target detection, and in particular to an infrared small target recognition method based on color shape prompt learning. BACKGROUND

[0002] Infrared images refer to thermal energy radiation images captured by infrared sensors. Unlike visible light images, infrared images can provide thermal energy information of targets and are suitable for target detection in low light, complex background or severe weather conditions. Early common single-frame infrared small target detection methods are model-based methods. They can be summarized as filter-based methods, human visual system-based methods and low-rank-based methods. Among them, filter-based methods are only suitable for single and uniform scenes. Human visual system-based methods are only completely suitable for cases where the brightness of the object is relatively large and there is a more obvious difference with the surrounding background. Low-rank-based methods, including local low-rank and non-local low-rank, are suitable for almost all types of complex and rapidly changing backgrounds, but in practice, it needs GPU and other accelerators to meet real-time requirements. Model-driven methods are easily affected by clutter and noise, reducing the robustness of the detection model. In complex backgrounds, it often cannot find an acceptable template or learn the local contrast information of the object, or the object modeling is severely affected by the model hyperparameters, resulting in poor generalization performance.

[0003] In recent years, the surge of data-driven machine learning methods, especially deep learning methods, has made them rapidly become the most widely used method for detecting small infrared objects. Due to the great imbalance between the target and the background, the boundary box regression is extremely sensitive to the small infrared target, and the target information is easily lost at the high-level semantic layer. Existing small target detection methods only focus on the brightness and shape of the target, without considering color information, and traditional detection models can only extract target features from the grayscale information of infrared images, lacking effective use of different spectral channels (multispectral or pseudo-color infrared). In addition, traditional methods are prone to target boundary blurring, resulting in reduced detection accuracy. The present application is based on this and proposes an infrared small target recognition method based on color shape prompt learning. SUMMARY

[0004] To solve the above problems, the present application discloses an infrared small target recognition method based on color shape prompt learning, which can more effectively detect infrared small targets, has good reliability and robustness, and improves the detection performance of infrared small targets.

[0005] The infrared small target recognition method based on color shape prompt learning specifically includes the following steps:

[0006] Step S1, divide the image dataset of the infrared small target into a training set and a test set;

[0007] Step S2, the images of the training set are input into the backbone network of the infrared small target detection network model to extract multi-scale features containing infrared small target images H and W represent the height and width of the single-channel image respectively; the resolution of the multi-scale feature The images of the training set are single-channel images, corresponding to the layer resolution reduction multiple;

[0008] Step S3, the multi-scale features are enhanced by the pyramid network FPN, the path aggregation network PAN and the color shape prompt learning network ColorPeel; the multi-scale features obtained by the pyramid network FPN and the path aggregation network PAN belong to the spatial semantic feature domain, and the multi-scale features obtained by the color shape prompt learning network ColorPeel belong to the saliency color shape feature domain;

[0009] Step S4, the spatial semantic feature domain and the saliency color shape feature domain are fused by the dual probability alignment DPA fusion framework to obtain multi-scale information features;

[0010] Step S5, the fused multi-scale information features are sent to the dynamic detection head Dynamic Head for detection; the detection result is evaluated by the normalized Wasserstein distance NWD and the adaptive threshold focal loss ATFL; the normalized Wasserstein distance NWD (Normalized Gaussian Wasserstein Distance) and the adaptive threshold focal loss ATFL calculate the loss and guide the optimization process of the infrared small target detection network model;

[0011] Step S6, finally, the test set images are selected as the input of the trained infrared small target detection network model to obtain the final infrared small target detection result.

[0012] Further, the color shape prompt learning network ColorPeel in step S3 is used for color and shape enhancement of multi-scale features, which specifically includes:

[0013] a: given the RGB or color coordinates provided by the user, generate instance images and templates;

[0014] b: the color shape prompt learning network ColorPeel generates color embedding vectors c* and shape embedding vectors s*.​​​​​​​

[0015] Furthermore, the dynamic head detection in step S5 is used to detect and process the fused multi-scale information features, which specifically includes the following steps:

[0016] Step S51: Given feature tensor ,in represents all images in the training set, represents the number of feature pyramid layers, Indicates the size of the feature, = × ; 、 Represents the height and width of a single-channel image, Indicates the number of channels;

[0017] Step S52: Perceiving Attention through Scale , spatial perception and attention and task-aware attention Dynamically fuse the features to obtain , the formula is: ;

[0018] Step S53: Scale-aware attention Dynamic feature fusion is performed based on the multi-scale information features obtained in step S4. The formula is: ;in It is a 1×1 convolutional layer; is the hard-sigmoid function; spatial perception attention Use deformable convolution to fuse multi-scale information features of different levels in the same spatial position. The formula is: ;in represents the number of sparse sampling locations, represents the number of feature pyramid layers, Indicates in The position in the layer The eigenvalue at is the initial sampling position, is the spatial offset obtained by learning the input features, is in position The scalar is obtained by self-learning and used to adjust the feature weight of the sampling position; task-aware attention Dynamically switch ON and OFF channels to support different tasks. The formula is: ;in is a hyperfunction that controls the threshold, where 、 for weighting the feature channels, 、 for bias adjustment of the feature channels; by average pooling in dimension, and then using two fully connected layers and a normalization layer, and finally normalized by a sigmoid activation function.

[0019] Further, the adaptive threshold focal loss ATFL in step S5 sets a threshold value, and values below the threshold value are regarded as hard samples, where the adaptive threshold focal loss ATFL expression is: ; wherein represents the current average prediction probability value, represents the prediction value of the next epoch, is a hyperparameter.

[0020] Further, the formula of the normalized wasserstein distance NWD in step S5 is: ; wherein D is a constant related to the data set, , are Gaussian distributions modeled by the bounding box A and the bounding box B respectively, is and Wasserstein distance between

[0021] Advantages of the present application:

[0022] 1、The present application introduces the color shape prompt learning network ColorPeel into the infrared small target detection model. The color shape prompt learning network ColorPeel separates different colors or spectral information, which helps to enhance the contrast of the infrared small target and makes it easier to distinguish in a complex background. Traditional infrared small target detection methods may be disturbed by noise or background clutter, while the color shape prompt learning network ColorPeel can effectively reduce background interference and improve detection accuracy through specific color layer stripping technology. The color shape prompt learning network ColorPeel technology can more clearly preserve the edge features of the target, reduce the distortion of the target shape caused by filtering or background suppression, and improve the integrity of the target. In various complex backgrounds (such as clouds, ocean ripples, ground textures, etc.), the color shape prompt learning network ColorPeel improves the adaptability and generalization ability of the detection method through color feature enhancement and hierarchical stripping. Combined with deep learning methods, as a feature enhancement module, it improves the network's ability to recognize small targets, especially in cases where there is less data or a low signal-to-noise ratio.

[0023] 2、The application proposes an adaptive threshold focus loss ATFL, and adaptively adjusts the loss value according to the prediction probability value, reduces the background loss influence by increasing the target loss weight, makes the model more focused on extracting the infrared small target feature, improves the detection effect, and improves the detection performance of the infrared small target.

[0024] 3、The application proposes the Wasserstein distance between two Gaussian distributions, and the traditional IoU (Intersection over Union) is extremely sensitive in small target detection. Small boundary box offset will cause a large change in IoU, making the training unstable. NWD remodels the boundary box by using the Wasserstein distance, so that the regression error is more smooth, and the error amplification problem in small target detection is reduced; because the Wasserstein distance is more robust to target scale changes, NWD can better adapt to infrared targets of different sizes and improve the generalization ability of detection. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is the overall flowchart of the embodiment of the application;

[0026] Figure 2 is the implementation flowchart of the color feature prompt learning network ColorPeel of the embodiment of the application;

[0027] Figure 3 is the implementation flowchart of the dynamic detection head Dynamic Head of the embodiment of the application;

[0028] Figure 4 is the image detection result comparison chart of the mainstream infrared small target detection model and the method of the application on different infrared small target data sets in the comparative test of the application embodiment. DETAILED DESCRIPTION

[0029] The application will be further illustrated below in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and not to limit the scope of the application. It should be noted that the words "front", "back", "left", "right", "up" and "down" used in the following description refer to the directions in the drawings, and the words "in" and "out" refer to the directions towards or away from the geometric center of a particular component.

[0030] As shown in Figure 1 , Figure 2 and Figure 3 , the infrared small target recognition method based on color shape prompt learning specifically includes the following steps:

[0031] Step S1, divide the image data set of the infrared small target into a training set and a test set;

[0032] Step S2, the images of the training set are input into the backbone network of the infrared small target detection network model to extract multi-scale features containing infrared small target images , , and respectively represent the height and width of the single-channel image; the resolution of the multi-scale feature , is ; the image of the training set is a single-channel image, represents the corresponding layer resolution reduction multiple;

[0033] Step S3, the multi-scale features are enhanced by the pyramid network FPN, the path aggregation network PAN and the color shape prompt learning network ColorPeel; the multi-scale features enhanced by the pyramid network FPN and the path aggregation network PAN belong to the spatial semantic feature domain, and the multi-scale features enhanced by the color shape prompt learning network ColorPeel belong to the salient color shape feature domain; the color shape prompt learning network ColorPeel in step S3 is used for color and shape enhancement of the multi-scale features, which specifically includes:

[0034] a: given the RGB or color coordinates provided by the user, generate instance images and templates;

[0035] b: the color shape prompt learning network ColorPeel generates color embedding vectors c* and shape embedding vectors s*.

[0036] Step S4, the spatial semantic feature domain and the salient color shape feature domain are fused by the dual probability alignment DPA fusion framework to obtain multi-scale information features;

[0037] Step S5, the fused multi-scale information features are sent to the dynamic detection head Dynamic Head for detection; the detection result is evaluated by the normalized Wasserstein distance NWD and the adaptive threshold focal loss ATFL; the normalized Wasserstein distance NWD and the adaptive threshold focal loss ATFL calculate the loss and guide the optimization process of the infrared small target detection network model; the dynamic detection head Dynamic Head in step S5 is used for detecting and processing the fused multi-scale information features, which specifically includes the following steps:

[0038] Step S51: given the feature tensor , wherein represents all images in the training set, ​​denotes the number of feature pyramid layers, denotes the size of the feature, = × ; 、 denote the height and width of the single-channel image, respectively, denotes the number of channels;

[0039] Step S52: Dynamic fusion of features by scale-aware attention , spatial-aware attention and task-aware attention , formula: ;

[0040] Step S53: Dynamic feature fusion by scale-aware attention according to the multi-scale information features obtained in step S4, formula: ; wherein is a 1x1 convolution layer; is a hard-sigmoid function; spatial-aware attention uses deformable convolution to fuse multi-scale information features of different levels in the same spatial position, formula: ; wherein denotes the number of sparse sampling positions, denotes the number of feature pyramid layers, denotes the feature value at position in the layer, is the initial sampling position, is the spatial offset learned from the input feature, is a scalar learned from the position to adjust the feature weight of the sampling position; task-aware attention dynamically switches ON and OFF channels to support different tasks, formula: ; wherein is a hyperfunction of a control threshold, wherein 、 is used for weighting operation on the feature channel, 、 is used for bias adjustment of the feature channel; is reduced in the dimension by average pooling, then two fully connected layers and a normalization layer are used, and finally normalized by a sigmoid activation function. ​

[0041] In step S5, the adaptive threshold focus loss ATFL sets a threshold and regards values ​​below this threshold as hard samples, where the adaptive threshold focus loss ATFL expression is: ;in represents the current average predicted probability value, Indicates the predicted value of the next epoch, is a hyperparameter.

[0042] In step S5, the bounding box is modeled as a two-dimensional Gaussian distribution, and the Wasserstein distance between the two Gaussian distributions is calculated; two-dimensional Gaussian distribution:

[0043]

[0044] in , and denote the random variable vector, mean vector and covariance matrix of Gaussian distribution respectively;

[0045] when

[0046] Horizontal bounding box , using 𝒩 Modeled as a 2-D Gaussian distribution;

[0047]

[0048] in,( , ), and Represents the center coordinates, width and height respectively; two 2-D Gaussian distributions )and The 2-D Wasserstein distance between is defined as:

[0049]

[0050] in is the mean vector and The square of the Euclidean distance between the two distributions is used to measure the difference in the center locations of the two distributions. 、 They are probability distributions and The corresponding covariance matrix. The rank of a matrix is denoted.

[0051] Simplify to:

[0052]

[0053] where is the Frobenius norm, and the boundary box and the boundary box Gaussian distribution modeled by , The distance between

[0054] , the Wasserstein distance is normalized to the range of 0-1, and the formula of the normalized Wasserstein distance NWD is: ; wherein D is a constant related to the data set, , are Gaussian distributions modeled by the boundary box A and the boundary box B respectively, is the Wasserstein distance between and

[0055] Step S6, finally, the test set image is selected as the input of the infrared small target detection network model trained, and the final infrared small target detection result is obtained.

[0056] The experimental simulation content of the infrared small target recognition method based on color shape prompt learning of the present application is as follows: the experimental platform uses a 64-bit Ubuntu system, the system version is 20.04.4, the GPU model is GeForce RTX 2080Ti, Python is used as the programming language, the version is Python 3.9, and PyCharm is used as the software development platform; the model is implemented using the deep learning framework Pytorch 2.0, the input image is a 3-channel RGB image with a size of 512x512, in the training stage, the batch size is 6, the iteration round is 500, the Adam optimizer is used to optimize the parameters, the initial learning rate is set to 1e-3, and under the acceleration of a GeForce RTX 2080 Ti GPU, the entire network training time is about 12 hours.

[0057] Figure 4 ​For the comparison test of the embodiments of the present application, the image detection results of two mainstream infrared small target detection models and the method of the present application on different infrared small target data sets are compared, wherein method one and method two respectively adopt ACM and MDvsFA-cGAN, and method three is GT, which represents the infrared small target saliency label. Through comparison, it can be known that the present application can effectively detect the infrared small target and improve the accuracy of infrared small target detection.

[0058] The technical means disclosed in the scheme of the present application is not limited to the technical means disclosed in the above-mentioned embodiments, but also includes technical solutions composed of any combination of the above technical features.

Claims

1. An infrared small target recognition method based on color and shape cue learning, characterized by: The specific steps include: Step S1, dividing the image data set of small infrared targets into a training set and a test set; Step S2: The images of the training set Input into the backbone network of the infrared small target detection network model to extract multi-scale features of the infrared small target image , , and Represents the height and width of a single-channel image respectively; multi-scale features , The resolutions are ; The images of the training set are single-channel images, Indicates The factor by which the resolution of the corresponding layer is reduced; Step S3: Multi-scale features are enhanced through the pyramid network FPN, path aggregation network PAN, and color shape hint learning network ColorPeel; The multi-scale features enhanced by the pyramid network FPN and the path aggregation network PAN belong to the spatial semantic feature domain, and the multi-scale features enhanced by the color shape hint learning network ColorPeel belong to the salient color shape feature domain; the color shape hint learning network ColorPeel in step S3 is used to enhance the color and shape of the multi-scale features, specifically including: a: Given the RGB or color coordinates provided by the user, generate instance images and templates; b: Color shape hint learning network ColorPeel generates color embedding vector c* and shape embedding vector s*; Step S4: Using the dual probability alignment (DPA) fusion framework, the spatial semantic feature domain and the salient color and shape feature domain are heterogeneously fused to obtain multi-scale information features. Step S5: The fused multi-scale information features are fed into a dynamic detection head for detection. The detection results are evaluated using the normalized Wasserstein distance NWD and the adaptive threshold focal loss ATFL. The normalized Wasserstein distance NWD and the adaptive threshold focal loss ATFL are used to calculate the loss and guide the optimization process of the infrared small target detection network model. Step S6: Finally, the test set image is selected as the input of the trained infrared small target detection network model to obtain the final infrared small target detection result.

2. The infrared small target recognition method based on color and shape cue learning according to claim 1 is characterized in that: The dynamic head detection in step S5 is used to detect and process the fused multi-scale information features, and specifically includes the following steps: Step S51: Given feature tensor ,in represents all images in the training set, represents the number of feature pyramid layers, Indicates the size of the feature, = × ; 、 Represents the height and width of a single-channel image, Indicates the number of channels; Step S52: Perceiving Attention through Scale , spatial perception and attention and task-aware attention Dynamically fuse the features to obtain , the formula is: ; Step S53: Scale-aware attention Dynamic feature fusion is performed based on the multi-scale information features obtained in step S4. The formula is: ;in It is a 1×1 convolutional layer; is the hard-sigmoid function; spatial perception attention Use deformable convolution to fuse multi-scale information features of different levels in the same spatial position. The formula is: ;in represents the number of sparse sampling locations, represents the number of feature pyramid layers, Indicates in The position in the layer The eigenvalue at is the initial sampling position, is the spatial offset obtained by learning the input features, is in position The scalar is obtained by self-learning and used to adjust the feature weight of the sampling position; task-aware attention Dynamically switch ON and OFF channels to support different tasks. The formula is: ;in is a hyperfunction that controls the threshold, where 、 Used to perform weighted operations on feature channels, 、 Used to adjust the bias of the feature channel; By average pooling Dimensionality reduction is performed, and then two fully connected layers and a normalization layer are used, and finally normalization is performed through the sigmoid activation function.

3. The infrared small target recognition method based on color and shape cue learning according to claim 1 is characterized in that: In step S5, the adaptive threshold focus loss ATFL sets a threshold and regards values ​​below the threshold as hard samples, where the adaptive threshold focus loss ATFL expression is: ;in represents the current average predicted probability value, Indicates the predicted value of the next epoch, is a hyperparameter.

4. The infrared small target recognition method based on color and shape cue learning according to claim 1 is characterized in that: The formula for the normalized Wasserstein distance NWD in step S5 is: ; where D is a constant related to the data set, , are the Gaussian distributions modeled by bounding box A and bounding box B, respectively, yes and The Wasserstein distance between them.

Images