Infrared weak and small target detection method combining single-scale network feature extraction and fusion

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an infrared weak and small target detection method combining single-scale network feature extraction and fusion. In order to overcome the problem of small target detection caused by down-sampling, an encoder-decoder network and a single-scale network are effectively combined to integrate infrared image features, so that a small target can be accurately detected; the method comprises the following steps: capturing and fusing target features of different scales by constructing a multi-scale feature enhancement module; a single-scale up-sampling-free sub-network is designed, detail information is effectively reserved by creating a feature fusion block, and advanced features extracted on a large receptive field by an encoder and a decoder are fully utilized at the same time. Compared with other advanced methods, the infrared weak and small target detection method provided by the invention has obvious performance advantages on two public data of SIRST and IRSTD-1k.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an infrared dim small target detection method combining single-scale network feature extraction and fusion. Background Art

[0002] Infrared technology is widely used in various fields such as infrared guidance, missile defense system, urban security monitoring and early warning, ship monitoring and intelligent maritime defense, etc. due to its strong adaptability to changes in light and environment, as well as its unique cloud-seeing ability. In view of the huge potential of infrared technology in various fields, it has become the focus of academic and industrial attention.

[0003] However, since infrared images are usually obtained through long-distance imaging systems, the objects of interest usually occupy only a small part of the image and are often easily obscured by background noise. In addition, due to the lack of clear feature information such as color, texture, and structure, infrared weak target detection is extremely challenging, and false positives and false negatives are particularly obvious.

[0004] At present, infrared dim small target detection models based on deep learning have shown superior performance in infrared dim small target detection. According to the output results, they can be divided into two categories: detection-based methods and segmentation-based methods. The segmentation-based IRSTD method usually produces a binary mask as output. In these binary masks, 1 represents pixels belonging to the target and 0 represents background pixels. The result of the detection-based infrared dim small target detection model is a bounding box that captures the position and size of the detected target in the image.

[0005] However, no matter which deep learning-based model is used, the encoder-decoder architecture is particularly common. However, since the target is very small, after the downsampling operation of the mainstream encoder-decoder network, the small target will not only lose its spatial information, but also become blurred in the background, resulting in missed detection. The current network focusing on infrared weak small target detection uses various methods to alleviate the problems caused by downsampling and reduce the problem of target feature loss. However, downsampling still exists in the existing network and does not fundamentally solve this problem. In contrast, the single-scale architecture does not use downsampling, which can retain detail information, but the receptive field is very limited. How to overcome the inherent shortcomings of these architectures is still a problem that needs to be solved. Summary of the invention

[0006] In view of the shortcomings of the current infrared dim small target detection method based on deep learning, the present invention provides an infrared dim small target detection method combining single-scale network feature extraction and fusion. The method effectively combines the encoder-decoder architecture and the single-scale architecture to combine the advantages of both, extract more valuable information from infrared images, and can detect small targets more accurately.

[0007] The technical solution of the present invention is:

[0008] The infrared dim small target detection method combined with single-scale network feature extraction and fusion includes the following steps:

[0009] S1, collect infrared images for training;

[0010] S2, constructing a target detection model, including an encoder, a decoder, a feature enhancement module and a single-scale network; the training data needs to be preprocessed before being input into the target detection model, and the preprocessing method is to divide the original training image into four non-overlapping image blocks to obtain a first input image I1, divide the original training image into two non-overlapping image blocks to obtain a second input image I2, and use the original training image as a third input image I3;

[0011] The model iterates the encoder, feature enhancement module, and decoder. In the first iteration, the encoder input is the first input image I1, and in the second iteration, the input is the output result of the decoder in the first iteration and the second input image I2. In these two iterations, the encoder downsamples the input image to capture high-level features. The encoder consists of three downsampling blocks, each of which consists of two channel attention modules and a downsampling operation. The output of the encoder is enhanced by the feature enhancement module and then input to the decoder for decoding.

[0012] In the first iteration, the feature image patch that is input to the feature enhancement module by the encoder is defined as X i , i∈{1,2,3,4},X i The following processing is performed in the feature enhancement module:

[0013] X i After the original convolution, we get O i,1 :

[0014] O i,1 =f 3×3 (X i )

[0015] where f 3×3 (·) is the original 3*3 convolution;

[0016] X i After dilation convolution, we get O i,2 :

[0017] O i,2 =g 3×3 (X i ,dilation=3)

[0018] where g 3×3(·,dilation=3) represents dilated convolution with a dilation rate of 3;

[0019] X i After 1×1 convolution, large core attention module, GELU and 1×1 convolution, O i,3 :

[0020]

[0021] Among them, LKA represents the large kernel attention module, GELU indicates that the activation function used is the Gaussian error linear unit. In LKA, the idea of ​​decomposing large kernel convolution is used to decompose the 21x21 convolution into a combination of 5x5 depth convolution, 7x7 depth expansion convolution and 1x1 convolution to capture long-range dependencies. represents the i-th 1×1 convolutional layer, ψ 5×5 (·) represents a 5×5 depthwise convolution, φ 7×7 (·) denotes a 7×7 depthwise dilated convolution with a dilation rate of 3;

[0022] Then O i,1 , O i,2 , O i,3 , and X i Connect to get F i :

[0023] F i =concatenate(O i,1 ,O i,2 ,O i,3 ,X i )

[0024] Last F i After 1×1 convolution, it is combined with X i Fusion to obtain P i :

[0025] P i =X i +f 1×1 (F i )

[0026] Since the input images I1 and I2 are different, in the second iteration, the encoder inputs the image block M into the feature enhancement module i , i∈{1,2}, the operations taken for each image block and the operations on X in the first iteration i The output P of the feature enhancement module is the same as iInput to the decoder, which consists of three upsampling blocks. Each upsampling block includes two channel attention modules and an upsampling operation. The output of the first decoder will be added element by element with the second input image I2 as the input of the second encoder, and the output of the second decoder will be input into the single-scale non-downsampling subnet together with the third input image I3 for feature extraction and fusion;

[0027] The single-scale network is a subnet that does not perform downsampling at all. The subnet consists of four feature extraction and fusion modules. The input X L0 is the third input image I3, and the input X H0 is the output of the second decoder, the output is X L1 and X H1 :

[0028] X LH0 =Conv1(X L0 )

[0029] X HL0 =Conv2(X H0 )

[0030] X HH0 =Conv3(X H0 )

[0031]

[0032] X L1 =BN(ReLU(X LL0 +X HL0 ))

[0033] X H1 =BN(ReLU(X LH0 +X HH0 ))

[0034] Among them, BN stands for batch normalization, and ReLU refers to the rectified linear unit that introduces nonlinearity into the model. i∈{1,2,3} represents 1x1 convolution, while Conv i (·), i∈{1,2,3} represents 3x3 convolution;

[0035] The input X of the second feature extraction and fusion module L1 and X H1 is the output of the first feature extraction and fusion module, and the output of the second feature extraction and fusion module is X L2 , X H2 and X L0 , X H0 After adding each element, we get the input X of the third feature extraction and fusion module.L3 , X H3 , the output X of the third feature extraction and fusion module L4 , X H4 is the input of the fourth feature extraction and fusion module; the output H of the final spatial and semantic feature extraction and fusion module is:

[0036] H = concatenate(X L5 +X L3 ,X H5 +X H3 )

[0037] Where X L5 , X H5 is the output of the fourth feature extraction and fusion module;

[0038] The output features are processed using the sigmoid function. Pixels larger than the threshold are considered target pixels, and pixels smaller than the threshold are considered background pixels.

[0039] S3, using the training data to train the constructed target detection model to obtain a trained target detection model;

[0040] S4. Input the image to be detected into the trained target detection model for infrared weak target detection.

[0041] The beneficial effects of the present invention are as follows: the infrared dim small target detection method combined with single-scale network feature extraction and fusion provided by the present invention can effectively solve the problem that small targets are easily submerged in the background and detail information is easily lost during the downsampling process. Different from the traditional but staged model, the model effectively combines the encoder-decoder architecture and the single-scale non-upsampling sub-network to integrate the advantages of different frameworks to complete accurate infrared dim small target detection. The method constructs a U-Net-like network, and designs a multi-scale feature extraction module based on large kernel convolution and dilated convolution to optimize feature extraction and introduces a sub-network that does not perform downsampling to effectively retain detail information and complete feature fusion. The method can achieve good small target detection performance on both SIRST and IRSTD-1k public datasets, and has obvious performance improvements compared to other cutting-edge infrared dim small target detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the structure of an infrared dim small target detection model combining single-scale network feature extraction and fusion in an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of a feature enhancement module based on large kernel convolution in an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of spatial and semantic feature extraction and fusion module in a single-scale subnet in an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of infrared dim small target detection results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0047] refer to Figure 1 In one embodiment of the present invention, the present invention provides an infrared dim small target detection method combining single-scale network feature extraction and fusion, comprising the following steps:

[0048] Step 1: By collecting relevant public data sets (for example, selecting and collecting images from the two public data sets of SIRST and IRSTD-1K, and performing data augmentation on the images, such as flipping and cropping, to make the training set richer, thereby improving the generalization ability of the method), and conducting model training, the images in the data set are normalized. Specifically, the mean and variance of each data set are calculated, and then the pixel value of each image is subtracted from the mean by channel and divided by the variance. This can accelerate the convergence of the model.

[0049] Step 2: The present invention further processes the image preprocessed in step 1 using a multi-image block hierarchy method. This method divides the input infrared image into multiple non-overlapping image blocks of different sizes and inputs them into the model respectively. This allows the model to focus on regions of different sizes in turn, from smaller image blocks to the entire original image. This design allows the model to gradually extract features from local details and finally conduct a comprehensive analysis combined with global information.

[0050] Step 3: Use a U-Net-like encoder-decoder network to encode and decode the input infrared image block to extract image information in a wider receptive field, including the following steps:

[0051] Step 3.1: Before the image block is fed into the backbone network, the original convolution and channel attention modules are first used to preliminarily extract and enhance the features of infrared dim small targets, while capturing the basic structural information of the image. After such preprocessing, the image retains its original resolution, but the number of channels is increased, providing richer information for subsequent feature fusion.

[0052] Step 3.2: A U-Net-like network is used as the backbone network. It gradually downsamples the input image through the encoder to expand the receptive field and capture global information; while the decoder gradually upsamples the features and combines low-level features with high-level features through skip-layer connections. Unlike the traditional U-Net, the network here uses channel attention modules in the encoder and decoder to replace some convolutional layers to further improve the efficiency and accuracy of feature extraction. The encoder and decoder are composed of three downsampling blocks and upsampling blocks, respectively, each of which contains two channel attention modules and a sampling operation (downsampling or upsampling).

[0053] Step 3.3: A feature enhancement module based on large kernel convolution is specially integrated, and its specific structure is as follows Figure 2 As shown in the figure. For each input image block, this module captures and fuses contextual features of different scales through multi-scale convolution operations. Specifically, for each image block, the module first uses standard convolution to capture short-range dependencies, then captures medium-range dependencies through dilated convolution, and finally captures long-range dependencies using the strategy of decomposing large kernel convolution. These features of different scales are spliced ​​together, reduced in dimension through 1x1 convolution, and fused with the original feature map to finally generate updated image block features. Through this multi-scale feature enhancement and fusion strategy, the module can more effectively extract and integrate key information in the image, thereby improving the detection performance of infrared dim small targets.

[0054] Step 4: Figure 3 The spatial and semantic feature extraction and fusion modules in the single-scale subnet shown in FIG. 1 are used to further extract and fuse image features, thereby avoiding the problem of detail information loss in the traditional downsampling process, and finally generating output features, such as Figure 4 As shown, the single-scale subnet is specifically processed as follows:

[0055] Step 4.1: In this stage, the input image first goes through a preprocessing step. This includes preliminary feature extraction of the image using raw convolution and channel attention modules.

[0056] Step 4.2: The third stage subnet consists of four feature extraction and fusion modules, which are responsible for further extracting and fusing feature information from different sources while maintaining high resolution. The features from step 3 are rich in semantic information because of the expansion of the receptive field, while the original image is considered to be rich in spatial information.

[0057] Step 4.3: The feature map output from the third stage is processed using the sigmoid function. The sigmoid function maps the feature value to a range of 0 to 1, and then divides the pixels into target pixels and background pixels by setting a threshold. Pixels greater than the threshold are considered targets, while pixels less than the threshold are considered background.

[0058] Step 5: Use the data set provided in S1 to perform end-to-end training on the network model constructed by S2 to S4, and optimize the model by setting the loss function and introducing the supervision mechanism until its performance is stable. Afterwards, use the test set to evaluate the trained model. The test results of the present invention are compared with the test results of the other six methods as follows:

[0059] Table 1 Comparison of the detection results of the infrared weak target detection method used and the other six methods

[0060]

[0061] The results show that the infrared dim small target detection method proposed in the present invention that combines single-scale network feature extraction and fusion has higher accuracy and lower false detection rate.

Claims

1. An infrared small target detection method combining single-scale network feature extraction and fusion, characterized in that: The following steps are involved: S1, collect infrared images for training; S2. Build a target detection model, including encoder, decoder, feature enhancement module and single-scale network; The training data needs to be preprocessed before being input into the target detection model. The preprocessing method is to divide the original training image into four non-overlapping image blocks to obtain the first input image I1, divide the original training image into two non-overlapping image blocks to obtain the second input image I2, and use the original training image as the third input image I3; The model iterates the encoder, feature enhancement module, and decoder. In the first iteration, the encoder input is the first input image I1, and in the second iteration, the input is the output result of the decoder in the first iteration and the second input image I2. In these two iterations, the encoder downsamples the input image to capture high-level features. The encoder consists of three downsampling blocks, each of which consists of two channel attention modules and a downsampling operation. The output of the encoder is enhanced by the feature enhancement module and then input to the decoder for decoding. In the first iteration, the feature image patch that is input to the feature enhancement module by the encoder is defined as X i , i∈{1, 2, 3, 4}, X i The following processing is performed in the feature enhancement module: X i After the original convolution, we get O i，1 : O i，1 =f 3×3 (X i ) where f 3×3 (·) is the original 3*3 convolution; X i After dilation convolution, we get O i，2 : O i,2 =g 3×3 (X i ,dilation=3) where g 3×3 (·, dilation=3) represents a dilated convolution with a dilation rate of 3; X i After 1×1 convolution, large core attention module, GELU and 1×1 convolution, O i，3 : Among them, LKA represents the large kernel attention module, GELU indicates that the activation function used is the Gaussian error linear unit. In LKA, the idea of ​​decomposing large kernel convolution is used to decompose the 21x21 convolution into a combination of 5x5 depth convolution, 7x7 depth expansion convolution and 1x1 convolution to capture long-range dependencies. represents the i-th 1×1 convolutional layer, ψ 5×5 (·) represents a 5×5 depthwise convolution, φ 7×7 (·) denotes a 7×7 depthwise dilated convolution with a dilation rate of 3; Then O i，1 , O i，2 , O i，3 , and X i Connect to get F i : F i =concatenate(O i，1 ,O i,2 ,O i，3 ,X i ) Last F i After 1×1 convolution, it is combined with X i Fusion to obtain P i : P i =X i +f 1×1 (F i ) Since the input images I1 and I2 are different, in the second iteration, the encoder inputs the image block M into the feature enhancement module i , i∈{1,2}, the operations taken for each image block and the operations on X in the first iteration i The operation is the same as that of the feature enhancement module; the output P i Input to the decoder, which consists of three upsampling blocks. Each upsampling block includes two channel attention modules and an upsampling operation. The output of the first decoder will be added element by element with the second input image I2 as the input of the second encoder, and the output of the second decoder will be input into the single-scale non-downsampling subnet together with the third input image I3 for feature extraction and fusion; The single-scale network is a subnet that does not perform downsampling at all. The subnet consists of four feature extraction and fusion modules. The input X L0 is the third input image I3, and the input X H0 is the output of the second decoder, the output is X L1 and X H1 : X LH0 =Conv1(X L0 ) X HL0 =Conv2(X H0 ) X HH0 =Conv3(X H0 ) X L1 =BN(ReLU(X LL0 +X HL0 )) X H1 =BN(ReLU(X LH0 +X HH0 )) Among them, BN stands for batch normalization, and ReLU refers to the rectified linear unit that introduces nonlinearity into the model. i∈{1, 2, 3} represents 1x1 convolution, while Conv i (·), i∈{1, 2, 3} represents 3x3 convolution; The input X of the second feature extraction and fusion module L1 and X H1 is the output of the first feature extraction and fusion module, and the output of the second feature extraction and fusion module is X L2 , X H2 and X L0 , X H0 After adding each element, we get the input X of the third feature extraction and fusion module. L3 , X H3 , the output X of the third feature extraction and fusion module L4 , X H4 is the input of the fourth feature extraction and fusion module; the output H of the final spatial and semantic feature extraction and fusion module is: H=concatenate(X L5 +X L3 ,X H5 +X H3 ) Where X L5 , X H5 is the output of the fourth feature extraction and fusion module; The output features are processed using the sigmoid function. Pixels larger than the threshold are considered target pixels, and pixels smaller than the threshold are considered background pixels. S3, using the training data to train the constructed target detection model to obtain a trained target detection model; S4. Input the image to be detected into the trained target detection model for infrared weak target detection.

Images