Infrared weak and small target detection method based on semantic guidance fusion network

Abstract

The invention provides an infrared weak and small target detection method based on a semantic guidance fusion network. An encoder extracts image semantic features through a convolutional neural network; shallow semantic features are guided by using spatial distribution of deep semantic features through a semantic-guided adaptive fusion module, and target pixels are segmented through a segmentation detection head. According to the method, the problem of deep semantic feature target space information loss caused by downsampling and the problem that target backgrounds are similar and difficult to distinguish due to complex backgrounds can be relieved, so that model false alarms are reduced while space detail information is enhanced. The feature maps of different semantic levels are effectively fused, and it is ensured that the fused feature maps have rich semantic information and accurate spatial positioning information. And the detection performance of the model in a complex scene is improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for detecting infrared dim small targets. Background Art

[0002] Currently, target detection technology in visible light scenes is developing rapidly. However, it is affected by light, so the detection accuracy is not high at night or in scenes with poor lighting conditions. Infrared target detection can solve this problem, but due to the limited resolution of infrared sensors and the long imaging distance, infrared images often show small and dim targets, complex backgrounds, and low signal-to-noise ratio. Infrared weak target detection is a technology that segments targets from infrared images with complex backgrounds. This technology has a wide range of applications, such as maritime rescue, military reconnaissance, fire warning, and other fields.

[0003] Infrared weak target detection technology can be divided into two categories based on technical characteristics: traditional detection methods based on image features and target detection methods based on deep learning. Traditional target detection methods rely on analyzing the features of image data and manually formulating corresponding detection templates for detection. The advantage of this method is that its operation is simple and the execution speed is fast, but its disadvantage is also obvious, that is, its detection accuracy and scene generalization ability are poor. In order to make up for these shortcomings, in recent years, with the increase in data volume and the improvement of computing power resources, detection methods based on deep learning, specifically methods based on convolutional neural networks, have shown remarkable performance. This method trains the model with a large amount of infrared data, thereby enhancing the model's recognition ability for infrared weak targets and further improving the accuracy of target detection. However, although this deep learning-based method has improved the accuracy of target detection to a certain extent, it still has some problems. Due to the extremely small size of the target, the neural network will lose part of the target when extracting deep semantic features, which will affect the detection performance of the model. At the same time, due to the complexity of the infrared image background, the neural network may mistakenly identify some noise with shapes or high brightness as targets, thereby increasing the false alarm rate of the model. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides an infrared dim small target detection method based on a semantically guided fusion network. A semantically guided adaptive fusion network framework for infrared dim small target detection is proposed, which includes four parts, an encoder, a decoder, a multi-branch enhancement module and a semantically guided adaptive fusion module. The encoder extracts image semantic features through a convolutional neural network; the information of small targets in deep semantic features is enhanced by introducing sub-branches in the encoder; secondly, the spatial distribution of deep semantic features is used to guide shallow semantic features through a semantically guided adaptive fusion module to improve the distinction between targets and backgrounds, enhance the model's ability to identify targets, and enhance the semantic information and spatial information of feature maps through cross-layer fusion feature maps, thereby improving the detection performance of the model. Finally, the target pixels are segmented out by a segmentation detection head. The present invention can alleviate the problem of missing spatial information of deep semantic feature targets caused by downsampling and the problem of similarity and difficulty in distinguishing between targets and backgrounds due to complex backgrounds, thereby reducing model false alarms while enhancing spatial detail information.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Semantic-guided adaptive fusion network design;

[0007] In order to combine spatial detail information with semantic information well, a multi-resolution auxiliary enhancement encoder (MAE Encoder) is designed to gradually enhance multi-level deep features through low-level feature maps to alleviate the problem of information loss of small targets in the deep layer of the network; considering the complex background and targets of different sizes in infrared weak target images, the features extracted by the multi-resolution auxiliary enhancement encoder are processed by sequentially stacked window transformer layers (SwinTransformer Layer) and atrous spatial pyramid pooling (ASPP) to perceive the context and obtain global information for better feature representation; finally, an adaptive fusion decoder (AdaFusDecoder) is designed to aggregate a series of features extracted by the encoder. Its overall framework is as follows: Figure 1As shown in the figure, given an infrared image, it is first sent to MAE Encoder to extract multi-scale features, and then the feature representation is enhanced by perceptual context information. The multi-scale features extracted by MAEEncoder and the perceptual context information enhanced features are output to AdaFus Decoder together. AdaFus Decoder is used to aggregate cross-layer features and obtain accurate features with both semantic and spatial detail information. Finally, the fused features are sent to the detection head to accurately locate and segment the target.

[0008] Step 2: Loss function;

[0009] A staged loss function is selected for training the model. Specifically, first use the first-stage SoftIoULoss for training, and record the loss value of each round of training. If the loss value within 50 rounds fluctuates within 0.05, the model is considered to have converged. Then use the second-stage binary cross entropy loss (BCELoss) to replace SoftIoULoss for fine-tuning. Replace the loss function to iteratively train the model after the first-stage convergence for 10 rounds to fine-tune the model parameters for accurate pixel classification.

[0010] In step 1, the specific designs of each part are as follows, including the design of the network and the selection of the loss function:

[0011] Step 1-1: Multi-resolution auxiliary enhancement encoder (MAE Encoder);

[0012] Given an infrared image I∈R H×W First, a stem convolution is used to increase the number of channels, and then a series of features are obtained through stacked convolution-batch normalization-activation layers (Convolotion-BatchNorm-Relu, Conv-BN-ReLU) and group residual blocks (Group Residual Block, GRB) Each convolutional layer has a convolution with a stride of 2 to downsample the feature map. GRB is used because it realizes the interaction between feature map channels in the same horizontal convolutional layer, improves the utilization rate of encoder channel feature information, and improves its feature extraction ability. Considering the information loss in the deep layer of the network, the residual block is used to fuse Enhance deep multi-level features, namely:

[0013]

[0014] Where RB(·) and Down(·) represent residual blocks and downsampling, respectively. S represents the encoder depth of the model. In the framework diagram, S=4. After obtaining multi-level features, the deep representation of the perceptual context is refined through the serialization modeling of the window transformer layer. Then, the feature map is sent to ASPP to aggregate multi-scale features, which is described as follows:

[0015]

[0016] in Corresponding to the features extracted by the S-th layer network in the encoder, when S=4, it corresponds to Figure 1 In F b It represents the output features after the window transformer layer (swin) serialization modeling perception context and the atrous spatial pyramid pooling (ASPP) aggregation multi-scale feature processing, that is, Figure 1 F b ;

[0017] Step 1-2: Adaptive Fusion Decoder (AdaFus Decoder);

[0018] The adaptive fusion decoder aims to fuse the multi-scale features extracted from the MAE encoder. The adaptive fusion decoder includes a semantic-guided cascade fusion (SGCF) module, a non-local block (NonLocal) and a residual block (RB). The semantic-guided cascade fusion module is used for adaptive multi-scale fusion, and the non-local block calculates the correlation between any two positions to achieve long-distance dependency modeling. After multi-scale fusion and long-distance modeling, the residual block is introduced to gradually refine the multi-scale features to achieve accurate detection. The SGCF module is the core of the decoder.

[0019] Step 1-3: Split the head;

[0020] The segmentation head is a stacked convolution-activation function layer (Convolution-Sigmoid, Conv1-Sigmoid), in which the 1×1 convolution layer Conv1 is used for channel transformation, which converts the number of feature map channels into the number of target categories, where the number of target categories is 1. The predicted probability is then obtained through the Sigmoid function, and finally the threshold thresh is used to distinguish the foreground pixels belonging to the target from the background. Specifically, those in the probability map that are greater than the threshold are considered as targets, otherwise they are considered as background.

[0021] The threshold thresh is set to 0.5.

[0022] The SoftIoU loss is defined as follows:

[0023]

[0024] Among them, TP represents the total number of pixels correctly predicted by the model, T represents the total number of pixels belonging to the target, P represents the total number of pixels predicted by the model as the target, σ represents the balance coefficient, where N is the total number of pixels, 0 is the yth i The category of pixels, p i is the predicted probability value of the i-th sample.

[0025] In the step 1-2, since features at different levels have different characteristics, high-level features are good at positioning, while low-level features are helpful for accurately segmenting the target, a semantic guided cascade fusion (SGCF) module is designed to adaptively fuse multi-layer features to avoid confusion. High-level features come from the decoding end, while low-level and middle-level features come from the encoding end. Superscripts e and d are used to represent the encoder and decoder, and subscripts L, m and h are used to represent low-level features, middle-level features and high-level features, respectively. Figure 2 As shown, the SCGF module uses high-level features from the decoder Mid-level features from the encoder and low-level features As input; first use channel attention (i.e. Figure 2 Channel Attention) Enhancement get Considering Rich semantics in Guiding mid-level features Specifically, parallel global maximum pooling (GloabalMaxPooling, GMP) and global average pooling (GlobalAveragePooling, GAP) are used to pool features. The pooled features are connected along the channel, and then the guidance weights are generated through the Convolution-BatchNorm-Sigmoid (CBS) activation function, which is used to adjust the mid-level features. The spatial distribution of

[0026]

[0027] Where Conv1(·), BN(·) and σ(·) represent 1×1 convolution, batch normalization and sigmoid activation function respectively. and After addition, the features are sent to the weight generation module to obtain the adaptive fusion weight. In the weight generation module, the added features pass through two parallel sub-branches, one of which contains two 1×1 convolutional layers and the other consists of GAP and a fully connected (FC) layer. The fusion weight W is obtained by combining the outputs of the two sub-branches. hm ; The process is expressed as:

[0028]

[0029] Where FC(·) represents the fully connected layer, σ and Conv1 represent the sigmoid activation function and 1×1 convolution respectively, and then use the generated weight W hm Fusion of high-level and mid-level features; namely:

[0030]

[0031] Where "⊙" represents the Hadamard product. express and The fused features;

[0032] Likewise, using Guide low-level features, before fusion, first pass through spatial attention (i.e. Figure 2 The spatial attention operation in the To obtain At the same time, the fused features Generate weight guidance through parallel GMP and GAP connected to CBS blocks The spatial distribution of

[0033]

[0034] Next Steps And the fused features After adding, they are sent to the weight generation module to obtain the adaptive fusion weight;

[0035]

[0036] Finally, the output of SGCF is fused and Get, that is:

[0037]

[0038] In SGCF, two cascaded fusion modules are used to make full use of the rich spatial details of low-level features and the accurate semantic information of high-level features, which greatly enhances the feature representation ability of small targets and thus improves the detection performance of small infrared targets.

[0039] An electronic device, comprising:

[0040] one or more processors;

[0041] Memory;

[0042] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to execute the method as described above.

[0043] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0044] The beneficial effect of the present invention is that the model of the present invention introduces additional branches in the encoder part to supplement the deep feature information of the network. This design can effectively alleviate the problem that in the process of the encoder extracting semantic features from shallow to deep layers, part of the spatial information is lost due to downsampling, which in turn affects the feature representation of the target in the deep network. This additional branch supplement network helps to enhance the model's ability to identify and accurately locate small targets while maintaining the depth and complexity of the network.

[0045] The semantically guided cascade adaptive module designed by the model of the present invention can effectively fuse feature maps of different semantic levels. It ensures that the fused feature map has both rich semantic information and accurate spatial positioning information. In addition, by using deep semantic features to guide the semantic distribution reconstruction of shallow features, the module not only improves the distinction between the target and the background in the shallow features, but also strengthens the feature representation of the target. This improves the detection performance of the model in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural diagram of the network model constructed by the method of the present invention.

[0047] Figure 2 It is a structural diagram of the semantically guided cascade adaptive fusion module of the present invention.

[0048] Figure 3 It is a comparison chart of the detection results of the method of the present invention and different classical methods in the embodiments. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and embodiments. Specific embodiment:

[0051] (1) Dataset preparation:

[0052] First, the public infrared dim target dataset NUDT is divided into a training set and a test set with a division ratio of 1:1. Thus, the training set and the test set are obtained, that is, 663 training sets and 664 test sets. Before model training, the training set images are resized to 256×256, and the single channel is expanded to RGB3 channels. Then, the changed images are randomly flipped and rotated for data enhancement; for the test set, only the size and channel transformation need to be changed before model prediction.

[0053] At this point, the dataset preparation step is completed.

[0054] (2) Construction of complementary fusion network model

[0055] Network structure diagram Figure 1 , the important parameters for building the network include:

[0056] 1) Encoder MAE Encoder parameters: depth is [2, 2, 4, 2], that is, the number of stacked GRB modules in each level; width is [32, 64, 128, 256], that is, the number of channels of the module in each level;

[0057] 2) Semantic-guided cascade fusion module:

[0058] Its structure can be seen Figure 2 Among them, GAP uses torch.nn.GlobalAveragePooling(), GMP uses torch.nn.GlobalAveragePooling(), BN uses torch.nn.BatchNorm2d(), and FC uses torch.nn.Linear(). This module uses the decoder deep feature F h d , from the encoder's F m e and F l e As input, the three feature maps are fused through this module, and then the fused feature maps are used as input and sent to the decoder for further decoding. The upsampling and downsampling operations involved are all implemented by bilinear interpolation.

[0059] (3) Network training

[0060] 1) Training parameter setting: Four NVIDIA GeForce GTX 3090 GPUs with 24G video memory were used for training. At the same time, the Adam optimizer was used to update the parameters in the PyTorch framework. The number of training epochs was set to 500, the batch size of training was set to 48, and the initial learning rate was set to 0.00005. The learning rate was decayed to 1 / 10 of the original at the 200th epoch and the third epoch. The validation set was used to test the network effect at the end of every 10 epochs, and the network parameters with the best effect were saved.

[0061] 2) Stop training condition: the network's loss function reaches convergence.

[0062] (4) Network testing

[0063] 4.1) Input image: Resize the image to be tested to 256×256 and expand the channels to RGB3 channels.

[0064] 4.2) Load the best network parameters saved during the training phase or the network parameters saved in the last epoch, and test the input image.

[0065] 4.3) Network output results: The final output results are saved in the result folder in the form of pictures.

[0066] like Figure 3 As shown, some comparison diagrams of detection results of the method of the present invention and other classical methods are shown, including ACMNet, ISTDU-Net, UIUNet and DNANet. The first and second columns respectively represent the input infrared image and the real annotated target, and the following columns correspond to the predicted images of each method. These images cover single targets and multiple targets. From the results, the present invention has better detection performance and can avoid missed detection of dim and small targets (as shown in the first, second and sixth rows, where the yellow dotted box represents missed targets). Secondly, when the target has obvious shape features, the prediction results of the method of the present invention are also closer to the true value (as shown in the third row); in addition, the false alarm of the method of the present invention is lower in complex scenes (as shown in the fifth row). In order to more intuitively compare the effectiveness of the present invention, the comparison results of multiple methods on multiple data sets are given, as shown in Table 1:

[0067] Table 1: Detection results of the proposed method (SCAFNet) and other classical methods on multiple data sets

[0068]

[0069] The present invention can achieve optimal or suboptimal results for multiple evaluation indicators on multiple data sets, which fully demonstrates the effectiveness and generalizability of the method proposed in the present invention.

Claims

1. A method for infrared dim small target detection based on semantic guidance fusion network, characterized by The steps include: Step 1: Semantic-guided adaptive fusion network design; In order to combine spatial detail information with semantic information well, a multi-resolution auxiliary enhancement encoder is designed. The multi-level deep features are enhanced step by step through low-level feature maps to alleviate the problem of information loss of small targets in the deep layer of the network. Considering the complex background and targets of different sizes in infrared weak target images, the features extracted by the multi-resolution auxiliary enhancement encoder are processed by sequentially stacked window transformer layers and void spatial pyramid pooling to perceive the context and obtain global information for better feature representation. Finally, an adaptive fusion decoder is designed to aggregate a series of features extracted by the encoder. In the overall framework, given an infrared image, it is first sent to the MAE Encoder to extract multi-scale features, and then the feature representation is enhanced by perceiving context information. The multi-scale features extracted by the MAE Encoder and the perceiving context information enhanced features are output to the AdaFus Decoder together. The AdaFus Decoder is used to aggregate cross-layer features and obtain accurate features with both semantic and spatial detail information. Finally, the fused features are sent to the detection head to accurately locate and segment the target. Step 2: Loss function; A staged loss function is selected for training the model. Specifically, first use the first-stage SoftIoULoss for training, and record the loss value of each round of training. If the loss value within 50 rounds fluctuates within 0.05, the model is considered to have converged. Then use the second-stage binary cross entropy loss to replace SoftIoULoss for fine-tuning. Replace the loss function to iteratively train the model after the first-stage convergence for 10 rounds to fine-tune the model parameters for accurate pixel classification.

2. The infrared dim small target detection method based on semantic guidance fusion network according to claim 1 is characterized in that: In the step 1, the specific designs of each part are as follows, including the design of the network and the selection of the loss function: Step 1-1: multi-resolution auxiliary enhancement encoder; Given an infrared image I∈R H×W First, a stem convolution is used to increase the number of channels, and then a series of features are obtained through stacked convolution-batch normalization-activation layers and group residual blocks. Each convolutional layer has a convolution with a stride of 2 to downsample the feature map. GRB is used because it realizes the interaction between feature map channels in the same horizontal convolutional layer, improves the utilization rate of the encoder channel feature information, and improves its feature extraction capability. Considering the information loss in the deep layer of the network, the residual block is used to fuse Enhance deep multi-level features, namely: Where RB(·) and Down(·) represent residual blocks and downsampling respectively, and S represents the encoder depth of the model. After obtaining multi-level features, the deep representation of the perceptual context is refined through the serialization modeling of the window transformer layer; then the feature map is sent to ASPP to aggregate multi-scale features, which is described as follows: in Corresponding to the features extracted by the S-th layer network in the encoder, F b It represents the output feature F after the window transformer layer serialization modeling perception context and the void space pyramid pooling aggregation multi-scale feature processing. b ; Step 1-2: Adaptive fusion decoder; The adaptive fusion decoder aims to fuse the multi-scale features extracted from the MAE encoder. The adaptive fusion decoder includes a semantically guided cascade fusion module, a non-local block and a residual block. The semantically guided cascade fusion module is used for adaptive multi-scale fusion. The non-local block calculates the correlation between any two positions to achieve long-distance dependency modeling. After multi-scale fusion and long-distance modeling, the residual block is introduced to gradually refine the multi-scale features to achieve accurate detection. The SGCF module is the core of the decoder. Step 1-3: Split the head; The segmentation head is a stacked convolution-activation function layer, in which the 1×1 convolution layer Conv1 is used for channel transformation, which converts the number of feature map channels into the number of target categories, where the number of target categories is 1. Then the prediction probability is obtained through the Sigmoid function, and finally the threshold thresh is combined to distinguish the foreground pixels belonging to the target from the background. The probability map with a value greater than the threshold is regarded as the target, otherwise it is regarded as the background.

3. The infrared dim small target detection method based on semantic guidance fusion network according to claim 2 is characterized in that: The threshold thresh is set to 0.

5.

4. The infrared dim small target detection method based on semantic guidance fusion network according to claim 2 is characterized in that: In the step 1-2, a semantically guided cascade fusion module is designed to adaptively fuse multi-layer features to avoid confusion, where high-level features come from the decoding end, and low-level and mid-level features come from the encoding end; superscripts e and d are used to represent the encoder and decoder, and subscripts L, m and h are used to represent low-level features, mid-level features and high-level features, respectively; As shown in Figure 2, the SCGF module uses high-level features from the decoder Mid-level features from encoder and low-level features As input; first use channel attention enhancement get use Guiding mid-level features Specifically, we use parallel global max pooling and global average pooling to pool features. The pooled features are connected along the channel, and then the guidance weights are generated through convolution-batch normalization-activation function. The guidance weights are used to adjust the middle-level features. The spatial distribution of Where Conv1(·), BN(·) and σ(·) represent 1×1 convolution, batch normalization and sigmoid activation function respectively. and After addition, the features are sent to the weight generation module to obtain the adaptive fusion weight. In the weight generation module, the added features pass through two parallel sub-branches, one of which contains two 1×1 convolutional layers and the other consists of GAP and a fully connected layer. The fusion weight W is obtained by combining the outputs of the two sub-branches. hm ; The process is expressed as: Where FC(·) represents the fully connected layer, σ and Conv1 represent the sigmoid activation function and 1×1 convolution respectively, and then use the generated weight W hm Fusion of high-level and mid-level features; namely: Where "⊙" represents the Hadamard product. express and The fused features; Likewise, using To guide low-level features, before fusion, low-level features are first enhanced through spatial attention operations To obtain At the same time, the fused features Generate weight guidance through parallel GMP and GAP connected to CBS blocks The spatial distribution of Next Steps And the fused features After adding, they are sent to the weight generation module to obtain the adaptive fusion weight; Finally, the output of SGCF is fused and Get, that is: In SGCF, two cascaded fusion modules are used to make full use of the rich spatial details of low-level features and the accurate semantic information of high-level features, which greatly enhances the feature representation ability of small targets and thus improves the detection performance of small infrared targets.

5. The infrared dim small target detection method based on semantic guidance fusion network according to claim 2 is characterized in that: The SoftIoU loss is defined as follows: Among them, TP represents the total number of pixels correctly predicted by the model, T represents the total number of pixels belonging to the target, P represents the total number of pixels predicted by the model as the target, σ represents the balance coefficient, where N is the total number of pixels, 0 is the yth i The category of pixels, p i is the predicted probability value of the i-th sample.

6. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 5.

Images