A method for infrared small target detection based on local maximum grayscale difference convolution layer

Abstract

The present invention discloses an infrared small target detection method based on a local maximum grayscale difference convolution layer, which converts the expert prior into an interpretable convolution layer. According to the characteristics of the local significance of infrared small targets, a local maximum grayscale difference convolution layer is designed, which can replace the convolution layer in any infrared small target detection network, and realize the robust detection of the network under weak supervision learning. At the same time, based on the local maximum grayscale difference convolution layer, a local maximum grayscale difference residual convolution block is constructed, which can replace the residual convolution block in any infrared small target detection network, and realize the robust weak small target detection based on point supervision.

Description

Technical Field

[0001] The invention relates to target detection technology, in particular to an infrared small target detection method based on a local maximum grayscale difference convolution layer. Background Art

[0002] Infrared small target detection aims to separate the target from the cluttered background in the infrared image. Due to the small size, little texture and weak energy of infrared small targets, infrared small target detection has always been a challenging task. Although the deep learning-based method has achieved better performance than the traditional model-driven method, it has strong unexplainability. For a common convolutional network, all its feature learning comes from the clustering summary of labeled samples that imply human cognition. In order to obtain more realistic target features, the sample labeling requirements are very high. The more detailed the labeling and the larger the sample size, the more conducive it is to learn the real features and reduce the risk of incorrect clustering. In this way, the dataset labeling task is very large, and the time and labor costs are high. Therefore, how to learn more real target features based on point labeling, achieve more reliable clustering, and enhance the robustness of detection is a problem that needs to be explored continuously.

[0003] When there are few supervisory signals, it is more difficult for the network to fully learn the spatiotemporal characteristics of the target. Prior knowledge is a cognitive information about weak target features based on expert experience. When training the network with point supervision, the training samples contain very little human cognitive information, which makes it difficult for the network to grasp the true characteristics of the target. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an infrared small target detection method based on a local maximum grayscale difference convolution layer in view of the shortcomings of the existing technology, so that the network can learn as many target features as possible and improve the accuracy of small target detection.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a small infrared target detection method, comprising the following steps:

[0006] S1, obtain multiple infrared training images containing small targets;

[0007] S2. Using the infrared training image as an input of an infrared small target detection network, training the infrared small target detection network, and obtaining a trained infrared small target detection network; the infrared small target detection network comprises a plurality of cascaded residual convolution blocks, wherein at least one residual convolution block for feature extraction is a local maximum grayscale difference residual convolution block; or

[0008] At least one convolution layer in at least one residual convolution block for feature extraction is a local maximum grayscale difference convolution layer;

[0009] The local maximum grayscale difference residual convolution block includes a cascaded local maximum grayscale difference convolution layer and a universal convolution layer, the local maximum grayscale difference convolution layer includes a maximum pooling layer, a first batch of normalization layers, a first activation function and a first convolution layer connected in sequence; the universal convolution layer includes a second convolution layer, a second batch of normalization layers and a second activation function connected in sequence.

[0010] The present invention designs a local maximum grayscale difference convolution layer, which can replace the convolution layer in any infrared small target detection network to achieve robust detection of the network under weak supervision learning. Based on the local maximum grayscale difference convolution layer, the present invention constructs a local maximum grayscale difference residual convolution block, which can replace the residual convolution block in any infrared small target detection network to achieve robust point-supervised small target detection. The present invention can enable the network to learn as many target features as possible, thereby improving the accuracy of small target detection.

[0011] The method of the present invention further comprises:

[0012] S3. Using the infrared image to be detected as the trained infrared small target detection network, detect the small targets in the infrared image.

[0013] In the present invention, the local maximum grayscale difference convolution layer performs the following operations:

[0014] Features of the extracted infrared training images Perform the maximum pooling operation to obtain the local maximum grayscale feature ; H is the height of the infrared image, W is the width of the infrared image, C is the number of channels of the infrared image, and R is a real number set;

[0015] Calculate the local maximum grayscale feature With features The difference is used to obtain the local maximum grayscale difference feature ;

[0016] From the local maximum grayscale difference feature Extracting local salient features of targets , for the local salient features of the target Convolution and batch normalization are performed in sequence, and the output of the batch normalization operation is used as the input of the activation function, and the output of the activation function is concatenated with the feature , and obtain the spatial features ;

[0017] The spatial domain features are obtained As the input of the next residual convolution block or the next network layer of the infrared small target detection network.

[0018] In the present invention, the local maximum grayscale difference residual convolution block / convolution layer is used to replace the residual convolution block or convolution layer in the backbone network for feature extraction in the infrared small target detection network.

[0019] In the present invention, the local maximum grayscale difference feature The specific acquisition process includes: performing a batch normalization operation on the difference, using the result of the batch normalization operation as the input of the activation function, and obtaining the local maximum grayscale difference feature ; Local maximum grayscale difference feature The expression is: ; and denote batch normalization and activation function respectively.

[0020] In the present invention, the target local salient features The expression is: ;in, represents the convolution operation, and denote batch normalization and activation function respectively.

[0021] In the present invention, the spatial feature The expression is: ; It means that convolution operation, batch normalization and ReLU activation are performed in sequence. represents the output of the local maximum grayscale difference convolution layer, that is, .

[0022] In the present invention, the first activation function and the second activation function are both ReLU activation functions.

[0023] In the present invention, the infrared small target detection network can adopt a Res-Unet network.

[0024] Compared with the prior art, the present invention has the following beneficial effects: the present invention can help the network perceive the grayscale difference around the pixel, the edge of the image target processed by the local maximum grayscale difference convolution layer will be enhanced, and the smooth part of the background will be suppressed. This convolution layer can be replaced by the 2D convolution layer in any infrared small target detection network to improve the detection performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 (a) is the original image, and Figure 1 (b) is the local maximum grayscale difference image (local enlarged image);

[0026] Figure 2 This is the principle of the local maximum grayscale difference convolution layer in the embodiment of the present invention;

[0027] Figure 3This is a structural diagram of a local maximum grayscale difference residual convolution block according to an embodiment of the present invention;

[0028] Figure 4 Three modes of replacing the residual convolution block in Res-Unet according to the embodiments of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The embodiment of the present invention converts the expert prior into an interpretable convolution layer. According to the characteristics of the local saliency of infrared small targets, a local maximum grayscale difference convolution layer is designed. The convolution layer can replace the convolution layer in any infrared small target detection network to achieve robust detection of the network under weak supervision learning. At the same time, based on the local maximum grayscale difference convolution layer, a local maximum grayscale difference residual convolution block is constructed, which can replace the residual convolution block in any infrared small target detection network to achieve robust point-supervised weak small target detection.

[0031] This implementation incorporates prior knowledge into the design of a special convolutional layer structure. Small targets appear as local small bright spots with high-frequency characteristics in the image, and the central energy is higher than the surrounding background. Therefore, the edge of the target is usually a small closed ring structure, which helps to distinguish the target from the background. Through experiments, it is found that finding the local maximum grayscale difference of the target helps to extract the target edge information, as shown in Figure 1 (a) and Figure 1 (b).

[0032] Using the above understanding, this embodiment designs a local maximum grayscale difference convolution layer , this convolutional layer can replace the convolutional layer in any infrared small target detection network. Its input features are the input features of the original convolutional layer to be replaced, that is, the output features of the previous layer structure in the network. The specific layer design considerations are as follows: Figure 2 shown.

[0033] For input features of any size , firstly, the sliding window size is ( The size can be set to an odd number such as 3, 5, etc. according to the degree of target edge diffusion), and the maximum pooling operation with a sliding window step size of 1 , get the local maximum grayscale feature of the same size as the input feature .like Figure 2As shown in the figure, the process is to obtain the intermediate matrix from the leftmost input feature. The maximum pooling operation is to take the maximum grayscale value in the left sliding window in the figure. The input feature needs to be padded accordingly to ensure that the input feature has the same scale as the local maximum grayscale feature.

[0034] Then, the difference between the local maximum grayscale feature and the input feature is calculated, and batch normalization and ReLU activation are performed to obtain the local maximum grayscale difference feature :

[0035]

[0036] in, and denote batch normalization and ReLU activation, respectively.

[0037] Finally, using the convolution kernel size , the step length is The general convolutional layer (generally =3, =1), from Further extract the local salient features of the target :

[0038] ;

[0039] in, Represents convolution.

[0040] like Figure 3 As shown in Figure 2, based on the local maximum grayscale difference convolution layer, a local maximum grayscale difference residual convolution block is constructed, which consists of two layers of convolution, namely: the pooling kernel is , the convolution kernel is Local maximum grayscale difference convolution layer , the kernel size is The general convolutional layer (generally =3). The input features are passed through the local maximum grayscale difference residual convolution block to obtain the spatial domain features. :

[0041]

[0042] in, It is a concatenation of convolution operation, batch normalization, and ReLU activation.

[0043] like Figure 3 , the local maximum grayscale difference operation is used to enhance the edge characteristics of the target, and the convolution operation is used to extract and fuse the characteristics. The resulting features The target edge information is included and will be input into the next layer of the network to complete the detection task.

[0044] This design can help the network perceive the grayscale differences around pixels. After being processed by the local maximum grayscale difference convolution layer, the edge of the image target will be enhanced, and the smooth part of the background will be suppressed. This convolution layer can be replaced by the 2D convolution layer in any infrared small target detection network to improve the detection performance of the network.

[0045] The following experiments use the public infrared small target video dataset NUDT-MIRSDT (https: / / github.com / TinaLRJ / Multi-frame-infrared-small-target-detection-DTUM).

[0046] All deep learning-based methods in the embodiments of the present invention are executed on PyTorch. The experimental computer has an Intel core i7-8700 CPU @3.2GHz, 64-GB memory, and an Nvidia GeForce 1080Ti GPU. All networks are trained using the HPM loss function to adapt to point supervision and optimized by the Adam optimizer. We initialize the weights of the convolutional layers using the Kaiming method and initialize the biases using uniformly distributed random initialization. The input image size of the network is 512×512. The batch size is set to 2 and the initial learning rate is set to 0.001. All training stops after 10 epochs.

[0047] The parameters are set as follows:

[0048]

[0049] In order to verify the effectiveness of the method of the embodiment of the present invention, we replace the corresponding residual convolution blocks in the Res-Unet network with the designed modules in different ways. We replace 1 module, 2 modules and 3 modules respectively, as shown in Figure 4 The comparison results are as follows.

[0050] Table 1 Comparison results of replacing 1 module, 2 modules and 3 modules

[0051]

[0052] From the results, we can see that without changing the overall structure of the network, only replacing some residual convolution blocks in the network with the designed local maximum grayscale difference residual convolution blocks, the target detection performance has been significantly improved, and the more replacements, the more obvious the performance improvement. This shows that the designed local maximum grayscale difference convolution layer, by introducing certain expert knowledge, helps the network learn more real features of the target.

[0053] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0054] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting small infrared targets based on a local maximum grayscale difference convolutional layer, characterized in that: The following steps are involved: S1, obtain multiple infrared training images containing small targets; S2. Using the infrared training image as an input of an infrared small target detection network, training the infrared small target detection network, and obtaining a trained infrared small target detection network; the infrared small target detection network comprises a plurality of cascaded residual convolution blocks, wherein at least one residual convolution block for feature extraction is a local maximum grayscale difference residual convolution block; or At least one convolution layer in at least one residual convolution block for feature extraction is a local maximum grayscale difference convolution layer; The local maximum grayscale difference residual convolution block includes a cascaded local maximum grayscale difference convolution layer and a universal convolution layer, wherein the local maximum grayscale difference convolution layer includes a maximum pooling layer, a first batch of normalization layers, a first activation function and a first convolution layer connected in sequence; the universal convolution layer includes a second convolution layer, a second batch of normalization layers and a second activation function connected in sequence; The local maximum grayscale difference convolution layer performs the following operations: Features of the extracted infrared training images Perform the maximum pooling operation to obtain the local maximum grayscale feature ; H is the height of the infrared image, W is the width of the infrared image, C is the number of channels of the infrared image, and R is a real number set; Calculate the local maximum grayscale feature With features The difference is used to obtain the local maximum grayscale difference feature ; From the local maximum grayscale difference feature Extracting local salient features of targets , for the local salient features of the target Convolution and batch normalization are performed in sequence, and the output of the batch normalization operation is used as the input of the activation function, and the output of the activation function is concatenated with the feature , and obtain the spatial features ; The spatial features As the input of the next residual convolution block or the next network layer of the infrared small target detection network.

2. The infrared small target detection method according to claim 1, characterized in that: Also includes: S3. Using the infrared image to be detected as the trained infrared small target detection network, detect the small targets in the infrared image.

3. The infrared small target detection method according to claim 1, characterized in that: Local maximum grayscale difference feature The specific acquisition process includes: performing a batch normalization operation on the difference, using the result of the batch normalization operation as the input of the activation function, and obtaining the local maximum grayscale difference feature ; Local maximum grayscale difference feature The expression is: ; and denote batch normalization and activation function respectively.

4. The infrared small target detection method according to claim 1, characterized in that: The local salient features of the target The expression is: ;in, represents the convolution operation, and denote batch normalization and activation function respectively.

5. The infrared small target detection method according to claim 1, characterized in that: The spatial characteristics The expression is: ; It means that convolution operation, batch normalization and ReLU activation are performed in sequence. represents the output of the local maximum grayscale difference convolution layer, that is, .

6. The infrared small target detection method according to any one of claims 1 to 5, characterized in that: The first activation function and the second activation function are both ReLU activation functions.

7. The infrared small target detection method according to any one of claims 1 to 5, characterized in that: The infrared small target detection network adopts the Res-Unet network.

Images