Infrared target detection method based on knowledge distillation and rapid multi-scale fusion

Abstract

The invention discloses an infrared target detection method based on knowledge distillation and fast multi-scale fusion, and the method comprises the steps: constructing an infrared target detection model based on a teacher-student network, dividing an obtained infrared target image into a training set and a test set, inputting the data of the training set into the model for training, and obtaining a trained model; and inputting the test set for testing, inputting a to-be-detected infrared target image into the model after the test is completed, and outputting an infrared target detection result to complete infrared target detection. According to the method, a knowledge distillation technology is utilized, a high-complexity teacher network model is used for assisting student network model training, a fast multi-scale fusion module and a fast self-adaptive sensitivity strategy are designed for a teacher network and a student network, computer storage space and computing resources are saved, small target positioning is more accurate, and the method is more practical. And features of different levels of the small target are reserved, a large amount of storage space and computing resources are effectively relieved, and accurate detection of the infrared small target is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared target detection, and in particular relates to an infrared target detection method based on knowledge distillation and fast multi-scale fusion. Background Art

[0002] Infrared target detection has a wide range of applications in medical imaging, fire alarms, vehicle-mounted assisted driving, agricultural production, etc. Infrared targets usually have the typical characteristics of "weak" and "small". "Weak" is mainly manifested in low contrast and low signal-to-noise ratio; "small" is due to the low number of pixels occupied by the target in the entire infrared image. In addition, the background is usually not smooth and uniform, including various environments and various natural meteorological conditions. Based on these factors, the target can easily be submerged in a complex scene.

[0003] The infrared small target detection algorithm mainly goes through four stages: background suppression, human vision, optimization theory, and deep learning. For the algorithm based on background suppression, since there are obvious differences between the target and the background, the background suppression algorithm emphasizes the use of image processing solutions to analyze the image and background. Representative methods include Tophat filtering, Max-Median filtering, etc. For the algorithm based on visual saliency, the algorithm is studied based on the principle of human visual processing. In recent years, many scholars have applied the mechanism based on human vision to the field of infrared small target detection. Representative methods include LCM algorithm model, MPCM algorithm model, etc. For the algorithm based on optimization theory, since the information in natural images generally has a low-rank property, that is, in a continuous data block, there is similarity between adjacent rows and columns. In this regard, the use of low-rank matrix recovery models for infrared small target detection has a wide range of applications. Its representative methods include IPI model, PSTNN model, etc. Infrared small target detection based on deep learning algorithms has emerged in recent years, mainly by building neural networks to achieve infrared small target detection, such as ACM network, DNANet network, AGPCNet network, MDFA-Net network. However, deep learning-based methods tend to extract features at different levels by designing some specific modules or retaining the information of small infrared targets through certain feature fusion strategies. Background suppression-based methods are easily affected by random clutter in complex backgrounds, often resulting in unsatisfactory target detection results; visual saliency-based methods are always affected by noise levels and complex backgrounds, resulting in low detection accuracy; optimization model-based methods utilize the low rank of the background and the sparsity of the target in the local infrared patch (IPI) to obtain better target detection results, but the setting of multiple regularization parameters requires rich experience, which limits the application of model-based regularization models in practice; neural network-based deep learning methods have achieved breakthrough performance in infrared dim small target detection, but only fine-tuning the existing networks designed for general targets lacks consideration of the unique characteristics of small infrared targets. These complex neural networks also consume a lot of storage space and computing resources. Therefore, it is necessary to invent an efficient and lightweight infrared target detection method without significantly affecting the excellent performance of deep learning methods. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an infrared target detection method based on knowledge distillation and fast multi-scale fusion to achieve accurate detection of small infrared targets.

[0005] The technical solution adopted by the present invention is: an infrared target detection method based on knowledge distillation and fast multi-scale fusion, and the specific steps are as follows:

[0006] S1. Build an infrared target detection model based on teacher-student network;

[0007] The infrared target detection model includes: a pre-processing fine-tuning module, a teacher network, a student network, and a post-processing fine-tuning module.

[0008] S2, based on step S1, dividing the acquired infrared target image into a training set and a test set, and then inputting the training set data into the infrared target detection model for training to obtain a trained model;

[0009] S3. Based on the trained model obtained in step S2, a test set is input for testing. After the test is completed, the infrared target image to be detected is input into the model, and the infrared target detection result is output to complete the infrared target detection.

[0010] Furthermore, the step S2 is specifically as follows:

[0011] S21, dividing the acquired infrared target image data of any size into a training set and a test set according to a ratio of 7:3;

[0012] The training set is used to train the parameters in the infrared target detection model, and the test set is used to evaluate the detection performance of the model.

[0013] S22, based on step S21, input the training set data into the pre-processing fine-tuning module for pre-processing;

[0014] Firstly, the batch normalization method is used to normalize each small batch of infrared target image data, and the length and width of the infrared target images are uniformly cropped to 480×480.

[0015] Then the infrared target image is data enhanced as follows:

[0016] Firstly, the infrared target image is randomly flipped and randomly cropped, then Gaussian noise is added to the infrared target image data, and then it is randomly erased. Finally, copy-paste adaptive enhancement is performed on the infrared target image.

[0017] The copy-paste adaptive enhancement is specifically as follows:

[0018] Two infrared target images are randomly acquired, and the target area in one of the infrared target images is adaptively enhanced according to the grayscale intensity and area ratio information of the target area in the other infrared target image. Then, the adaptively enhanced target area is pasted into the randomly flipped infrared target image to obtain the amplified infrared target image, that is, the data enhanced infrared target image.

[0019] S23, based on the image fine-tuned by the pre-processing in step S22, inputting into the teacher network to obtain a feature map and a prediction map;

[0020] During the training of the teacher network, the initial learning rate is set to 0.02, and gradually increases to reach the maximum value during the warm-up process. The learning rate adopts the form of exponential decay.

[0021] The teacher network includes: 3 fast downsampling modules, 1 fast multi-scale fusion module, and 3 fast upsampling modules.

[0022] Among them, the fast multi-scale fusion module includes: 3 fast convolution modules, 3 feature mapping modules, and 3 efficient channel ECA attention mechanism modules.

[0023] For the teacher network, the input image N×C×H×W is firstly downsampled by three fast downsampling modules and three residual modules, that is, feature extraction is performed by four modules of depth separable convolution, maximum pooling, activation function, and spatial and channel fusion attention mechanism to obtain the first layer. The first feature map U1, the second layer The second feature map U2, the third layer The third feature map U3 of . Then the expression of the fast downsampling module is as follows:

[0024] U i =SCC(Relu(Maxpool(DS(IM i )))) i=1,2,3

[0025] Where N represents the number of images input in a batch, C represents the number of channels of a single image, H represents the height of a single image, and W represents the width of a single image. i represents the input infrared image, DS represents depthwise separable convolution, the convolution kernel size is 3×3, and the convolution kernel step size is 1; Maxpool represents maximum pooling; SCC represents the spatial and channel fusion attention mechanism. The specific expression of the spatial and channel fusion attention mechanism is as follows:

[0026]

[0027] in, Represents the fusion weight. SC represents the spatial attention mechanism; CC represents the channel attention mechanism.

[0028] Then the feature maps U1~U3 are passed through the fast multi-scale fusion module.

[0029] Among them, the fast convolution module includes: 1 local convolution PConv and 2 standard convolution Conv as well as the normalization layer BN and the activation layer RELU.

[0030] When processing the input feature map, local convolution only processes the frontmost or last continuous c in the input feature map. p The channels are subjected to standard convolution operation, and the other channels remain unchanged, so that the number of channels of the input and output feature maps is equal, and then c is adjusted. p The value of is the number of channels

[0031] Two standard convolutional layers together form an inverted residual block, and the number of channels in the middle layer is expanded, and shortcut connections are placed to reuse input features. The expression of the inverted residual block is as follows:

[0032] Y i =RELU(BN(Conv2(Conv1(PConv(X i ))))) i=1,2,3

[0033] Among them, X i represents the infrared target image input at this stage, PConv represents local convolution, the convolution kernel size is 3×3, the convolution kernel step size is 1, and the number of convolution channels is Conv represents a 3×3 standard convolution, where Conv1 represents a downsampling convolution with a convolution kernel size of 3×3 and a convolution kernel step of 1; Conv2 represents an upsampling convolution, which uses a deconvolution with a convolution kernel size of 3×3 and a step of 1 to extract image information; BN represents a normalization layer; RELU represents an activation function.

[0034] Then perform the maximum pooling feature mapping on the image output by the fast convolution module to obtain the feature mappings Figure 1 , feature map Figure 2 and feature maps Figure 3 , and the sizes of the three maps are different.

[0035] Among them, through the feature mapping module, the lower layer image is connected to the upper layer image by linear interpolation method, and the upper layer image is fused with the lower layer image by downsampling method. Finally, fused image 1, fused image 2, and fused image 3 are obtained.

[0036] The fused images of different layers are input into the efficient channel attention mechanism module to obtain new fused downsampled images F1, fused downsampled images F2, and fused downsampled images F3. The expression of the efficient channel attention mechanism module is as follows:

[0037] F i＝ Sigmoid(Conv1D(GAP(X i )))⊙X i i=1,2,3

[0038] Among them, GAP represents global average pooling; Conv1D represents 1×1 single-point convolution; Sigmoid represents activation function; ⊙ represents channel-by-channel multiplication, which multiplies the normalized weight of the activation function with the corresponding channel of the original feature map to generate a weighted feature map. The fusion maps F1~F3 are the same size as the feature maps U1~U3.

[0039] Then the downsampled feature map U3 is input into the first fast upsampling module, and the weight vector generated by the feature map U3 and the weighted sum of the feature map are used as the output image, namely the upsampled map D1.

[0040] Among them, the fast upsampling module includes: transposed convolution module, single-point convolution module, squeeze-excitation attention mechanism module. The convolution kernel size of the transposed convolution module is 3×3, and the convolution kernel step size is 1; the convolution kernel size of the single-point convolution module is 1×1; the specific expression of the squeeze-excitation attention mechanism module is as follows:

[0041] Om i =Scale(Sigmoid(FC(RELU(FC(GAP(Mm i ))))))i=1,2,3

[0042] Among them, Mm i Represents the input image of this stage; GAP represents global average pooling; FC represents the fully connected layer. After two layers of full connection, the information of different channels in the input image will be assigned different weights. The activation function of the first fully connected layer is RELU; the activation function of the second fully connected layer is Sigmoid; Scale represents weight assignment, that is, the weighted sum of the weight vector generated by the input image and the feature map is used as the output image.

[0043] Then, the fusion image F3 is added to the upsampled image D1 as the input of the second fast upsampling module, and the upsampled image D2 is output. The upsampled image D2 is added to the fusion image F2 as the input of the third fast upsampling module, and the upsampled image D3 is output. Finally, the upsampled image D3 is added to the fusion image F1 to output the final prediction image P1.

[0044] Among them, the size of the up-sampled images D1~D3 is consistent with the feature images U1~U3. The expression of the fast up-sampling module is as follows:

[0045] P1=FUS(FUS(FUS(U3)+F3+F3)+F2)+F1

[0046] Among them, FUS represents the fast upsampling module.

[0047] S24, based on step S23, using the knowledge distillation network to let the student network learn the teacher network to improve the training process, and output the distillation image, that is, the student network prediction image;

[0048] The teacher network is pruned to obtain the student network, and the student network is optimized using the total loss function Loss composed of the hard loss function HardLoss and the soft loss function SoftLoss.

[0049] Among them, the student network prediction graph and the real result data together constitute the hard loss function, which is the weighted sum of the cross entropy loss CEL and the focus loss FL. The teacher network prediction graph and the student network prediction graph together constitute the soft loss function, which is the weighted sum of the TCKD divergence loss and the NCKD divergence loss function.

[0050] The cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The expression is as follows:

[0051]

[0052] Among them, t represents the true label, represents the probability value predicted by the model, and CEL represents the cross entropy loss.

[0053] The focal loss expression is as follows:

[0054] FL=--ε t (1-p t ) μ log(p t )

[0055] Among them, ε t represents a balance factor; μ represents a focusing parameter used to adjust the weight of difficult samples; p t Represents the probability that the model's prediction is correct.

[0056] TCKD divergence loss measures the difference between the output of the student network and the teacher network and the prediction consistency of the student network on different inputs by introducing a temperature parameter. The TCKD expression is as follows:

[0057]

[0058] Where T represents the temperature parameter, which is used to control the smoothness of the probability distribution of soft targets; ρ represents a balancing factor, which is used to adjust the weight between the two parts of the loss.

[0059] NCKD divergence loss introduces additional constraints based on the temperature parameter to adjust the output distribution of the teacher network and the student network to make the distribution smoother. The expression of NCKD is as follows:

[0060]

[0061] Among them, τ represents a hyperparameter used to control the weight of the conditional loss term; MSE represents the mean square error loss function. The hard loss function and the soft loss function together constitute the total loss function, and the total loss function expression is as follows:

[0062] Loss=α×SoftLoss+(1-α)×HardLoss

[0063] SoftLoss=β×TCKD+(1-β)×NCKD

[0064] HardLoss = γ × CEL + (1-γ) × FL

[0065] Among them, α, β, and γ represent the corresponding weights respectively.

[0066] S25, inputting the distilled image outputted in step S24 into a post-processing fine-tuning module to obtain an infrared target detection result;

[0067] The post-processing fine-tuning module uses a fast adaptive sensitivity strategy to adjust the prediction of infrared images. The fast adaptive sensitivity strategy uses a strong threshold to obtain strong targets and a weak threshold to obtain weak targets.

[0068] For strong targets, they need to be detected and segmented completely and precisely; for weak targets, they need to be detected in the form of points. The specific fast adaptive sensitivity strategy is as follows:

[0069] A1. Compare the strong threshold segmentation image and the weak threshold segmentation image, and keep the overlapping area and record it as o i (i=1,2...n), counted into the final prediction result set;

[0070] A2. Extract the non-overlapping areas of the weak threshold image separately and obtain the centroid of each non-overlapping area, denoted as C i (i=1,2...m), count the centroid into the final set of prediction results;

[0071] A3. The final prediction result graph is recorded as D = {d i =o i ∪c i |o i ∈O i ,c i ∈C i}, which is the union of the overlapping region map and the region centroids.

[0072] Among them, the final prediction result map of the fast adaptive sensitivity strategy includes both the strong target area with high confidence and the centroid position of the weak target with low confidence.

[0073] S26, repeat steps S21-S25, use the training set data to train the model 300 times, and obtain a trained model;

[0074] In the first 50 training sessions, the training set data was trained with the training set data after data augmentation by copy-paste adaptive enhancement. In the next 250 training sessions, the training set data was trained with copy-paste adaptive enhancement disabled and only the training set data was uniformly cropped to 480×480.

[0075] Beneficial effects of the present invention: The method of the present invention constructs an infrared target detection model based on a teacher-student network, divides the acquired infrared target image into a training set and a test set, then inputs the training set data into the model for training, obtains a trained model, inputs the test set for testing, and after the test is completed, inputs the infrared target image to be detected into the model, outputs the infrared target detection result, and completes the infrared target detection. The method of the present invention utilizes knowledge distillation technology, uses a highly complex teacher network model to assist the student network model training, designs a fast multi-scale fusion module and a fast adaptive sensitivity strategy for the teacher network and the student network, saves computer storage space and computing resources, makes the positioning of small targets more accurate, and retains the features of small targets at different levels, effectively alleviates a large amount of storage space and computing resources, and realizes accurate detection of infrared small targets. The method of the present invention constructs an end-to-end detection network, and the detection map can be obtained by inputting the image to be detected, without any additional manual parameter adjustment or calculation steps in the middle, which is suitable for the application scenario of automatic detection of infrared small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flow chart of an infrared target detection method based on knowledge distillation and fast multi-scale fusion of the present invention.

[0077] Figure 2 Schematic diagram of the teacher network structure described in an embodiment of the present invention.

[0078] Figure 3 Schematic diagram of the structure of a fast multi-scale fusion module in an embodiment of the present invention.

[0079] Figure 4 Schematic diagram of local convolution in an embodiment of the present invention.

[0080] Figure 5 Schematic diagram of knowledge distillation in an embodiment of the present invention.

[0081] Figure 6 Schematic diagram of a fast adaptive sensitivity strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0083] like Figure 1 As shown, a flow chart of an infrared target detection method based on knowledge distillation and fast multi-scale fusion of the present invention is shown, and the specific steps are as follows:

[0084] S1. Build an infrared target detection model based on teacher-student network;

[0085] The infrared target detection model includes: a pre-processing fine-tuning module, a teacher network, a student network, and a post-processing fine-tuning module.

[0086] S2, based on step S1, dividing the acquired infrared target image into a training set and a test set, and then inputting the training set data into the infrared target detection model for training to obtain a trained model;

[0087] S3. Based on the trained model obtained in step S2, a test set is input for testing. After the test is completed, the infrared target image to be detected is input into the model, and the infrared target detection result is output to complete the infrared target detection.

[0088] In this embodiment, step S2 is specifically as follows:

[0089] S21, dividing the acquired infrared target image data of any size into a training set and a test set according to a ratio of 7:3;

[0090] The training set is used to train the parameters in the infrared target detection model, and the test set is used to evaluate the detection performance of the model.

[0091] S22, based on step S21, input the training set data into the pre-processing fine-tuning module for pre-processing;

[0092] Firstly, the batch normalization method is used to normalize each small batch of infrared target image data to ensure reasonable data distribution, and the length and width of the infrared target image are uniformly cropped to 480×480.

[0093] Then the infrared target image is data enhanced as follows:

[0094] First, the infrared target image is randomly flipped and randomly cropped, and then Gaussian noise and random erasure are performed on the infrared target image data. This can effectively increase the diversity of training samples and prevent overfitting. Finally, the infrared target image is copy-pasted and adaptively enhanced.

[0095] The copy-paste adaptive enhancement is specifically as follows:

[0096] Two infrared target images are randomly acquired, and the target area in one of the infrared target images is adaptively enhanced according to the grayscale intensity and area ratio information of the target area in the other infrared target image. Then, the adaptively enhanced target area is pasted into the randomly flipped infrared target image to obtain the amplified infrared target image, that is, the data enhanced infrared target image.

[0097] S23, based on the image fine-tuned by the pre-processing in step S22, inputting into the teacher network to obtain a feature map and a prediction map;

[0098] In the process of training the teacher network, the initial learning rate is set to 0.02, and gradually increases to the maximum value during the warm-up process to avoid training instability caused by an initial learning rate that is too high. To prevent the model from falling into a local optimum in the later stages of training, the learning rate is exponentially decayed to improve the training efficiency of the model.

[0099] like Figure 2 As shown, the teacher network includes: 3 fast downsampling modules, 1 fast multi-scale fusion module, and 3 fast upsampling modules.

[0100] Among them, Figure 3 As shown in the figure, the fast multi-scale fusion module includes: 3 fast convolution modules, 3 feature mapping modules, and 3 efficient channel ECA attention mechanism modules.

[0101] For the teacher network, the input image N×C×H×W is firstly downsampled by three fast downsampling modules and three residual modules, that is, feature extraction is performed by four modules of depth separable convolution, maximum pooling, activation function, and spatial and channel fusion attention mechanism to obtain the first layer. The first feature map U1, the second layer The second feature map U2, the third layer The third feature map U3 of . Then the expression of the fast downsampling module is as follows:

[0102] U i =SCC(Relu(Maxpool(DS(IM i )))) i=1,2,3

[0103] Where N represents the number of images input in a batch, C represents the number of channels of a single image, H represents the height of a single image, and W represents the width of a single image. i represents the input infrared image, DS represents depthwise separable convolution, the convolution kernel size is 3×3, and the convolution kernel step size is 1; Maxpool represents maximum pooling; SCC represents the spatial and channel fusion attention mechanism. The specific expression of the spatial and channel fusion attention mechanism is as follows:

[0104]

[0105] in, represents the fusion weight, which is 0.5 in this embodiment. SC represents the spatial attention mechanism; CC represents the channel attention mechanism.

[0106] Then the feature maps U1~U3 are passed through the fast multi-scale fusion module.

[0107] Among them, the fast convolution module includes: 1 local convolution PConv and 2 standard convolution Conv as well as the normalization layer BN and the activation layer RELU.

[0108] The schematic diagram of local convolution is as follows Figure 4 As shown in the figure, local convolution reduces redundant calculations and memory accesses by applying convolution operations only to some channels of the input feature map while keeping other channels unchanged, thereby improving computational efficiency; in the figure, k represents the side length of the convolution kernel size, which is 3; w represents the input image width, h represents the input image height, and c represents the number of image channels.

[0109] When processing the input feature map, local convolution only processes the frontmost or last continuous c in the input feature map. p The standard convolution operation is performed on the channels, and the other channels remain unchanged. This process makes the number of channels of the input and output feature maps equal, and then adjust c p The value of is the number of channels The memory access can be reduced to that of a regular convolution The amount of computation becomes that of conventional convolution The model can be lightweight to a certain extent.

[0110] Two standard convolutional layers together form an inverted residual block, and the number of channels in the middle layer is expanded, and shortcut connections are placed to reuse input features. The expression of the inverted residual block is as follows:

[0111] Y i =RELU(BN(Conv2(Conv1(PConv(X i ))))) i=1,2,3

[0112] Among them, X i represents the infrared target image input at this stage, PConv represents local convolution, the convolution kernel size is 3×3, the convolution kernel step size is 1, and the number of convolution channels is Conv represents a 3×3 standard convolution, where Conv1 represents a downsampling convolution with a kernel size of 3×3 and a kernel step of 1; Conv2 represents an upsampling convolution, which uses a deconvolution with a kernel size of 3×3 and a step of 1 to extract image information; BN represents a normalization layer, which calculates the mean and variance on the feature dimension of each sample to ensure that the output value of each layer maintains a relatively stable distribution; RELU represents an activation function, which is used to prevent the gradient from disappearing.

[0113] Then perform the maximum pooling feature mapping on the image output by the fast convolution module to obtain the feature mappings Figure 1 , feature map Figure 2 and feature maps Figure 3 , and the sizes of the three maps are different.

[0114] In order to fully integrate the feature information of each layer, the feature mapping module uses linear interpolation to connect the lower layer image with the upper layer image, and uses downsampling to fuse the upper layer image with the lower layer image. Finally, fused image 1, fused image 2, and fused image 3 are obtained.

[0115] The efficient channel attention mechanism is a lightweight channel attention mechanism that dynamically adjusts the importance of each channel by introducing a one-dimensional convolution operation, thereby enhancing the expressiveness and robustness of the model.

[0116] The fused images of different layers are input into the efficient channel attention mechanism module to obtain new fused downsampled images F1, fused downsampled images F2, and fused downsampled images F3. The expression of the efficient channel attention mechanism module is as follows:

[0117] F i＝ Sigmoid(Conv1D(GAP(X i )))⊙X i i=1,2,3

[0118] Among them, GAP represents global average pooling; Conv1D represents a 1×1 single-point convolution, which is used to fuse the features of each channel; Sigmoid represents the activation function, which is used to prevent gradient explosion or disappearance; ⊙ represents channel-by-channel multiplication, which multiplies the normalized weight of the activation function with the corresponding channel of the original feature map to generate a weighted feature map. The size of the fusion map F1~F3 is the same as that of the feature map U1~U3.

[0119] Then the downsampled feature map U3 is input into the first fast upsampling module, and the weight vector generated by the feature map U3 and the weighted sum of the feature map are used as the output image, namely the upsampled map D1.

[0120] Among them, the fast upsampling module includes: transposed convolution module, single-point convolution module, squeeze-excitation attention mechanism module. The convolution kernel size of the transposed convolution module is 3×3, and the convolution kernel step size is 1; the convolution kernel size of the single-point convolution module is 1×1, which is used to fuse the information of each channel; the squeeze-excitation attention mechanism module can learn and use global information to dynamically adjust the weight of channel features, which can effectively improve the representation and generalization capabilities of the model. The specific expression of the squeeze-excitation attention mechanism module is as follows:

[0121] Om i =Scale(Sigmoid(FC(RELU(FC(GAP(Mm i ))))))i=1,2,3

[0122] Among them, Mm i Represents the input image of this stage; GAP represents global average pooling; FC represents the fully connected layer. After two layers of full connection, the information of different channels in the input image will be assigned different weights. The activation function of the first fully connected layer is RELU; the activation function of the second fully connected layer is Sigmoid; Scale represents weight assignment, that is, the weighted sum of the weight vector generated by the input image and the feature map is used as the output image.

[0123] Then, the fusion image F3 is added to the upsampled image D1 as the input of the second fast upsampling module, and the upsampled image D2 is output. The upsampled image D2 is added to the fusion image F2 as the input of the third fast upsampling module, and the upsampled image D3 is output. Finally, the upsampled image D3 is added to the fusion image F1 to output the final prediction image P1.

[0124] Among them, the size of the up-sampled images D1~D3 is consistent with the feature images U1~U3. The expression of the fast up-sampling module is as follows:

[0125] P1=FUS(FUS(FUS(U3)+F3+F3)+F2)+F1

[0126] Among them, FUS represents the fast upsampling module.

[0127] S24, based on step S23, using the knowledge distillation network to let the student network learn the teacher network to improve the training process, and output the distillation image, that is, the student network prediction image;

[0128] like Figure 5 As shown, the knowledge distillation schematic diagram of this embodiment performs network pruning on the teacher network to obtain the student network, reducing the number of parameters and the amount of computation. The student network is optimized using the total loss function Loss composed of the hard loss function HardLoss and the soft loss function SoftLoss.

[0129] Among them, the student network prediction graph and the real result data together constitute the hard loss function, which is the weighted sum of the cross entropy loss (CEL: Cross-EntropyLoss) and the focal loss (FL: FocalLoss). The teacher network prediction graph and the student network prediction graph together constitute the soft loss function, which is the weighted sum of the TCKD divergence loss and the NCKD divergence loss function.

[0130] The cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The expression is as follows:

[0131]

[0132] Among them, t represents the true label, represents the probability value predicted by the model, and CEL represents the cross entropy loss.

[0133] Focus loss is applicable to the problem of class imbalance. Focus loss introduces a dynamic weight factor to make the model pay more attention to difficult-to-classify samples during training and reduce the influence of easy-to-classify samples. The expression of focus loss is as follows:

[0134] FL=-ε t (1-p t ) μ log(p t )

[0135] Among them, ε t represents a balance factor, which is taken as 0.7 in this embodiment; μ represents a focusing parameter used to adjust the weight of difficult samples, which is taken as 0.2 in this embodiment; p t Represents the probability that the model's prediction is correct.

[0136] TCKD divergence loss measures the difference between the output of the student network and the teacher network and the prediction consistency of the student network on different inputs by introducing a temperature parameter. The TCKD expression is as follows:

[0137]

[0138] Wherein, T represents a temperature parameter, which is used to control the smoothness of the probability distribution of the soft target, and is taken as 3 in this embodiment; ρ represents a balance factor, which is used to adjust the proportion between the two parts of the loss, and is taken as 0.8 in this embodiment.

[0139] NCKD divergence loss introduces additional constraints based on the temperature parameter to adjust the output distribution of the teacher network and the student network, making the distribution smoother and facilitating the student network to learn parameters. The expression of NCKD is as follows:

[0140]

[0141] Wherein, τ represents a hyperparameter used to control the weight of the conditional loss term, which is taken as 0.5 in this embodiment; MSE represents the mean square error loss function.

[0142] The hard loss function and the soft loss function together constitute the total loss function, and the total loss function expression is as follows:

[0143] Loss=α×SoftLoss+(1-α)×HardLoss

[0144] SoftLoss=β×TCKD+(1-β)×NCKD

[0145] HardLoss = γ × CEL + (1-γ) × FL

[0146] Wherein, α, β, and γ represent corresponding weights respectively. In this embodiment, α=0.3, β=0.5, and γ=0.5.

[0147] S25, inputting the distilled image outputted in step S24 into a post-processing fine-tuning module to obtain an infrared target detection result;

[0148] like Figure 6 As shown in FIG. 1 , the post-processing fine-tuning module uses a fast adaptive sensitivity strategy to adjust the prediction of the infrared image. The fast adaptive sensitivity strategy obtains strong targets through a strong threshold and weak targets through a weak threshold.

[0149] Among them, for strong targets, completely fine detection and segmentation are required. For weak targets, they need to be detected in the form of points. In this embodiment, the strong threshold is taken as 0.5 and the weak threshold is taken as 0.15. The fast adaptive sensitivity strategy is as follows:

[0150] A1. Compare the strong threshold segmentation image and the weak threshold segmentation image, and keep the overlapping area and record it as O. i (i=1,2...n), counted into the final prediction result set.

[0151] A2. Extract the non-overlapping areas of the weak threshold image separately and obtain the centroid of each non-overlapping area, denoted as C i (i=1,2...m), and count the centroid into the final set of prediction results.

[0152] A3. The final prediction result graph is recorded as D = {d i =o i ∪c i |o i ∈O i ,c i ∈C i}, which is the union of the overlapping region map and the region centroids.

[0153] The final prediction result map of the fast adaptive sensitivity strategy includes both the strong target area with high confidence and the centroid position of the weak target with low confidence. Strengthening the detection of difficult targets can significantly improve the detection rate and effectively improve the detection accuracy and integrity of targets in infrared images.

[0154] S26, repeat steps S21-S25, use the training set data to train the model 300 times, and obtain a trained model;

[0155] In the first 50 training sessions, the training set data was trained with the training set data after data augmentation by copy-paste adaptive enhancement. In the next 250 training sessions, the training set data was trained with copy-paste adaptive enhancement disabled and only the training set data was uniformly cropped to 480×480.

[0156] In summary, the method of the present invention utilizes knowledge distillation technology, uses a highly complex teacher network model to assist student network model training, designs a fast multi-scale fusion module and a fast adaptive sensitivity strategy for the teacher network and the student network, saves computer storage space and computing resources, makes the positioning of small targets more accurate, and retains the features of small targets at different levels, effectively alleviating a large amount of storage space and computing resources, and achieving accurate detection of infrared small targets. The method of the present invention constructs an end-to-end detection network, and the detection map can be obtained by inputting the image to be detected, without any additional manual parameter adjustment or calculation steps in the middle, which is suitable for the application scenario of automatic detection of infrared small targets.

[0157] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An infrared target detection method based on knowledge distillation and fast multi-scale fusion, the specific steps are as follows: S1. Build an infrared target detection model based on teacher-student network; The infrared target detection model includes: Pre-processing fine-tuning module, teacher network, student network, post-processing fine-tuning module; S2, based on step S1, dividing the acquired infrared target image into a training set and a test set, and then inputting the training set data into the infrared target detection model for training to obtain a trained model; S3. Based on the trained model obtained in step S2, a test set is input for testing. After the test is completed, the infrared target image to be detected is input into the model, and the infrared target detection result is output to complete the infrared target detection.

2. The infrared target detection method based on knowledge distillation and fast multi-scale fusion according to claim 1, characterized in that: The step S2 is specifically as follows: S21, dividing the acquired infrared target image data of any size into a training set and a test set according to a ratio of 7:3; The training set is used to train the parameters in the infrared target detection model, and the test set is used to evaluate the detection performance of the model; S22, based on step S21, input the training set data into the pre-processing fine-tuning module for pre-processing; Firstly, the batch normalization method is used to normalize each small batch of infrared target image data, and the length and width of the infrared target image are uniformly cropped to 480×480; Then the infrared target image is data enhanced as follows: Firstly, the infrared target image is randomly flipped and randomly cropped, then Gaussian noise is applied to the infrared target image data, and then random erasure is performed, and finally copy-paste adaptive enhancement is performed on the infrared target image; The copy-paste adaptive enhancement is specifically as follows: Two infrared target images are randomly acquired, and the target area in one of the infrared target images is adaptively enhanced according to the grayscale intensity and area ratio information of the target area in the other infrared target image. Then, the adaptively enhanced target area is pasted into the randomly flipped infrared target image to obtain an amplified infrared target image, i.e., a data-enhanced infrared target image. S23, based on the image fine-tuned by the pre-processing in step S22, inputting into the teacher network to obtain a feature map and a prediction map; In the process of training the teacher network, the initial learning rate is set to 0.02, and gradually increases to reach the maximum value during the warm-up process. The learning rate adopts the form of exponential decay; The teacher network includes: 3 fast downsampling modules, 1 fast multi-scale fusion module, and 3 fast upsampling modules; Among them, the fast multi-scale fusion module includes: 3 fast convolution modules, 3 feature mapping modules, and 3 efficient channel ECA attention mechanism modules; For the teacher network, the input image N×C×H×W is firstly downsampled by three fast downsampling modules and three residual modules, that is, feature extraction is performed by four modules of depth separable convolution, maximum pooling, activation function, and spatial and channel fusion attention mechanism to obtain the first layer. The first feature map U1, the second layer The second feature map U2, the third layer The third feature map U3; then the expression of the fast downsampling module is as follows: IN i =SCC(Relu(Maxpool(DS(IM i ))))i=1,2,3 Among them, N represents the number of images input in a batch, C represents the number of channels of a single image, H represents the height of a single image, and W represents the width of a single image; IM i represents the input infrared image, DS represents depthwise separable convolution, the convolution kernel size is 3×3, and the convolution kernel step size is 1; Maxpool represents maximum pooling; SCC represents the spatial and channel fusion attention mechanism. The specific expression of the spatial and channel fusion attention mechanism is as follows: in, represents fusion weight; SC represents spatial attention mechanism; CC represents channel attention mechanism; Then the feature maps U1~U3 are passed through the fast multi-scale fusion module; Among them, the fast convolution module includes: 1 local convolution PConv and 2 standard convolution Conv as well as the normalization layer BN and the activation layer RELU; When processing the input feature map, local convolution only processes the frontmost or last continuous c in the input feature map. p The channels are subjected to standard convolution operation, and the other channels remain unchanged, so that the number of channels of the input and output feature maps is equal, and then c is adjusted p The value of is the number of channels Two standard convolutional layers together form an inverted residual block, and the number of channels in the middle layer is expanded, and shortcut connections are placed to reuse input features. The expression of the inverted residual block is as follows: AND i =RELU(BN(Conv2(Conv1(PConv(X i )))))i=1,2,3 Among them, X i represents the infrared target image input at this stage, PConv represents local convolution, the convolution kernel size is 3×3, the convolution kernel step size is 1, and the number of convolution channels is Conv represents a 3×3 standard convolution, where Conv1 represents a downsampling convolution with a convolution kernel size of 3×3 and a convolution kernel step size of 1; Conv2 represents an upsampling convolution, which uses a deconvolution with a convolution kernel size of 3×3 and a step size of 1 to extract image information; BN represents a normalization layer; RELU represents an activation function; Then, the image output by the fast convolution module is subjected to feature mapping by maximum pooling, and feature mapping 1, feature mapping 2 and feature mapping 3 are obtained respectively, and the sizes of the three mappings are different from each other; Among them, through the feature mapping module, the linear interpolation method is used to connect the lower layer image with the upper layer image, and the downsampling method is used to fuse the upper layer image with the lower layer image; finally, fused image 1, fused image 2, and fused image 3 are obtained; The fused images of different layers are input into the efficient channel attention mechanism module to obtain new fused down-sampled images F1, fused down-sampled images F2, and fused down-sampled images F3; the expression of the efficient channel attention mechanism module is as follows: F i＝ Sigmoid(Conv1D(GAP(X i )))⊙X i i=1,2,3 Among them, GAP represents global average pooling; Conv1D represents 1×1 single-point convolution; Sigmoid represents activation function; ⊙ represents channel-by-channel multiplication, which multiplies the normalized weight of the activation function by the corresponding channel of the original feature map to generate a weighted feature map; the fusion map F1~F3 is the same size as the feature map U1~U3; Then the downsampled feature map U3 is input into the first fast upsampling module, and the weight vector generated by the feature map U3 and the weighted sum of the feature map are used as the output image, that is, the upsampled map D1; Among them, the fast upsampling module includes: transposed convolution module, single-point convolution module, squeeze-excitation attention mechanism module; the convolution kernel size of the transposed convolution module is 3×3, and the convolution kernel step size is 1; the convolution kernel size of the single-point convolution module is 1×1; the specific expression of the squeeze-excitation attention mechanism module is as follows: Om i =Scale(Sigmoid(FC(RELU(FC(GAP(Mm i ))))))i=1,2,3 Among them, Mm i Represents the input image of this stage; GAP represents global average pooling; FC represents the fully connected layer. After two layers of full connection, the information of different channels in the input image will be assigned different weights; the activation function of the first fully connected layer is RELU; the activation function of the second fully connected layer is Sigmoid; Scale represents weight assignment, that is, the weighted sum of the weight vector generated by the input image and the feature map is taken as the output image; Then, the fusion image F3 is added to the upsampled image D1 as the input of the second fast upsampling module, and the upsampled image D2 is output. The upsampled image D2 is added to the fusion image F2 as the input of the third fast upsampling module, and the upsampled image D3 is output. Finally, the upsampled image D3 is added to the fusion image F1 to output the final prediction image P1. Among them, the size of the up-sampled images D1~D3 is consistent with the feature images U1~U3; the expression of the fast up-sampling module is as follows: P1=FUS(FUS(FUS(U3)+F3+F3)+F2)+F1 Among them, FUS represents the fast upsampling module; S24, based on step S23, using the knowledge distillation network to let the student network learn the teacher network to improve the training process, and output the distillation image, that is, the student network prediction image; The teacher network is pruned to obtain the student network, and the student network is optimized using the total loss function Loss composed of the hard loss function HardLoss and the soft loss function SoftLoss; Among them, the student network prediction graph and the real result data together constitute the hard loss function, which is the weighted sum of the cross entropy loss CEL and the focus loss FL; the teacher network prediction graph and the student network prediction graph together constitute the soft loss function, which is the weighted sum of the TCKD divergence loss and the NCKD divergence loss function; The cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the true label. The expression is as follows: Among them, t represents the true label, Represents the probability value predicted by the model, and CEL represents the cross entropy loss; The focal loss expression is as follows: FL=-e t (1-p t ) μ log(p t ) Among them, ε t represents a balance factor; μ represents a focusing parameter used to adjust the weight of difficult samples; p t Represents the probability that the model prediction is correct; TCKD divergence loss measures the difference between the output of the student network and the teacher network and the prediction consistency of the student network on different inputs by introducing a temperature parameter; the TCKD expression is as follows: Where T represents the temperature parameter, which is used to control the smoothness of the probability distribution of soft targets; ρ represents a balance factor, which is used to adjust the proportion between the two parts of the loss; Based on the temperature parameter, NCKD divergence loss introduces additional constraints to adjust the output distribution of the teacher network and the student network to make the distribution smoother; the expression of NCKD is as follows: Among them, τ represents a hyperparameter used to control the weight of the conditional loss term; MSE represents the mean square error loss function; The hard loss function and the soft loss function together constitute the total loss function, and the total loss function expression is as follows: Loss=α×SoftLoss+(1-α)×HardLoss SoftLoss=β×TCKD+(1-β)×NCKD HardLoss = γ × CEL + (1-γ) × FL Among them, α, β, and γ represent the corresponding weights respectively; S25, inputting the distilled image outputted in step S24 into a post-processing fine-tuning module to obtain an infrared target detection result; The post-processing fine-tuning module uses a fast adaptive sensitivity strategy to adjust the prediction of infrared images; the fast adaptive sensitivity strategy obtains strong targets through a strong threshold and weak targets through a weak threshold; Among them, for strong targets, completely fine detection and segmentation are required; for weak targets, they need to be detected in the form of points; the fast adaptive sensitivity strategy is as follows: A1. Compare the strong threshold segmentation image and the weak threshold segmentation image, and keep the overlapping area and record it as O. i (i=1,2...n), counted into the final prediction result set; A2. Extract the non-overlapping areas of the weak threshold image separately and obtain the centroid of each non-overlapping area, denoted as C i (i=1,2...m), count the centroid into the final set of prediction results; A3. The final prediction result graph is recorded as D = {d i =o i ∪c i |o i ∈O i ,c i ∈C i }, the union of the overlapping region map and the centroid of the region; Among them, the final prediction result map of the fast adaptive sensitivity strategy includes both the strong target area with high confidence and the centroid position of the weak target with low confidence; S26, repeat steps S21-S25, use the training set data to train the model 300 times, and obtain a trained model; In the first 50 training sessions, the training set data was trained with the training set data after data augmentation by copy-paste adaptive enhancement. In the next 250 training sessions, the training set data was trained with copy-paste adaptive enhancement disabled and only the training set data was uniformly cropped to 480×480.

Images