A method for detecting small infrared targets based on time-domain profile features

Abstract

The present invention discloses an infrared small target detection method based on temporal profile features. Long sequences are input into a temporal differential convolutional layer and three spatial-temporal differential residual blocks to extract rough spatiotemporal features and locate salient areas. Then, the features are processed using TPM to obtain temporal profile features. After feature fusion and segmentation heads, the final confidence map of all input frame images is generated. The temporal profile dimension makes it unnecessary to extract fine spatiotemporal features, so TP‑Net can achieve excellent performance with higher efficiency.

Description

Technical Field

[0001] The invention relates to target detection technology, in particular to an infrared small target detection method based on time domain profile characteristics. Background Art

[0002] Infrared small target (IRST) detection aims to identify small targets from various infrared backgrounds and is widely used in fields such as drone surveillance. Infrared small targets are small in size, have no shape and texture information, and change dynamically, and are sometimes even submerged in complex background clutter. In deep learning-based methods, the main information used is spatial saliency features and temporal saliency features. Recently, some researchers have found that these feature dimensions are not sufficient for infrared weak small target (IRSDT) detection. In practical applications, IRSDT detection is crucial to capture targets as early and far as possible. Since the intensity of IRSDT is not only insignificant in the spatial domain, but also in the temporal domain, it is challenging to achieve robust detection of IRSDT.

[0003] As the academic community pays more attention to multi-frame IRSDT (MIRSDT) detection, temporal saliency features and motion features are used to improve the detection performance of dim targets with spatial saliency features. These feature dimensions perform well in detecting salient and continuously moving targets, but when the target is faint or even flickering, these features will not be significant, resulting in a serious degradation in detection performance. In the real world, IRDST may be submerged in random noise and dynamic clutter, and accurate prediction of targets and clutter depends on long-term observation information. Existing feature dimensions rarely pay attention to this information, so there is an urgent need for a new feature dimension to explore long-term information to achieve breakthroughs in IRSDT detection. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an infrared small target detection method based on time domain profile characteristics in view of the shortcomings of the existing technology, so as to improve the accuracy of small target detection and reduce calculation parameters.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for detecting small infrared targets based on time domain profile features, comprising the following steps:

[0006] S1, obtaining image sequence data;

[0007] S2, extracting inter-frame difference features of the image sequence data in time frame order;

[0008] S3, using the extracted inter-frame difference features as inputs of N spatiotemporal difference modules to extract spatiotemporal features;

[0009] S4, performing time-domain probe matching on the spatiotemporal features to obtain a series of matching features;

[0010] S5. Fuse all matching features to obtain the final confidence map of all input frame images.

[0011] The present invention designs three-dimensional differential convolutions of different dimensions to replace general convolutions to highlight small targets, extracts spatiotemporal features and locates significant areas through time domain and spatial domain differential convolutions; the present invention designs a time domain probe and a time domain probe mechanism, and learns and extracts the characteristics of the target in the time domain profile through the time domain probe mechanism and TP-Net to achieve effective IRST detection. The TP-Net of the present invention can process hundreds of frames of images at a time.

[0012] The convolution operation of the time-domain difference convolution layer The expression is as follows:

[0013] ;

[0014] in, is the first The pixel matrix of the frame image, is the convolution kernel of the time-domain difference convolution layer No. weight matrix, S is the image sequence data, k represents the size of the sliding window, and l is the time domain length of the inter-frame difference feature.

[0015] The spatiotemporal difference module includes a temporal difference convolution layer and a spatial difference convolution layer which are connected in sequence.

[0016] The temporal expansion rate of the temporal difference convolution layer is 2, and the spatial expansion rate of the spatial difference convolution layer is 2.

[0017] The temporal difference convolution layer and the spatial difference convolution layer both include a convolution layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0018] The convolution operation of the spatial difference convolution layer The expression is as follows:

[0019] ;

[0020] Where c is the index of the center of the space, is the input feature of the spatial difference convolution layer, is the first pixel vectors, represents the center pixel vector of the image in the sliding window, express Center and center pixel The corresponding weight vector is, is the convolution kernel of the spatial difference convolution layer The A weight vector is used to perform dot product operations with the corresponding vector of the image during the convolution process, representing the weights of the convolution kernel at different positions.

[0021] The specific implementation process of step S4 includes:

[0022] The spatiotemporal characteristics Replace with ; , , C represents the number of channels, T represents the time dimension, H and W represent the height and width of the feature map respectively;

[0023] Use the following formula to obtain matching features : ;in, represents the permutation operation of the matrix, is the correlation matrix.

[0024] In step S5, the specific implementation process of fusing all matching features includes:

[0025] Performing a batch normalization operation on the series of matching features;

[0026] Input the matching features after batch normalization operation into the activation function;

[0027] A convolution operation is performed on the output of the activation function to obtain a final confidence map of all input frame images.

[0028] The activation function is the ReLU activation function.

[0029] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a time-domain probe mechanism (TPM) to sense different changes, match and distinguish targets and clutter in the time-domain profile dimension. TPM uses multiple learnable correlation matrices as time-domain probes to match the time-domain profile features of the target pixel by pixel. Based on TPM, we propose a simple time-domain probe network (TP-Net), which extracts rough spatiotemporal features from the input multi-frame images and locates the salient areas, then uses TPM to process the features to obtain the time-domain profile features, and generates the final confidence map after feature fusion and segmentation head, thereby eliminating the need to extract fine spatiotemporal features. Comparative experimental results on the NUDT-MIRSDT dataset and the IRDST dataset show that TP-Net has achieved a significant performance breakthrough in small target detection (SNR≤3), and achieved state-of-the-art performance with significantly reduced computational parameters (29.2% / 38.2% of the lightest multi-frame / single-frame methods, respectively) and inference time (4.5% / 46.6% of the fastest multi-frame / single-frame methods, respectively). BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a diagram of the TP-Net network architecture of an embodiment of the present invention;

[0031] Figure 2 This is a structural diagram of the time-domain differential convolution layer in an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the architecture of the time probe TPM according to an embodiment of the present invention;

[0033] Figure 4 The results on the NUDT-MIRSDT dataset are compared intuitively for the embodiments of the present invention; (a1-a3) are the results on the NUDT-MIRSDT (SNR≤3) subset, (b1-b4) are the results on the NUDT-MIRSDT (SNR>3) subset, and the target area is enlarged in the upper right corner;

[0034] Figure 5 (a)-Figure 5 (c) show the ROC performance of different methods on the NUDT-MIRSDT dataset; Figure 5 (a) NUDTMIRSDT (SNR≤3), Figure 5 (b) NUDT-MIRSDT (SNR>3), Figure 5 (c) NUDT-MIRSDT (ALL). DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The embodiment of the present invention provides an infrared small target detection method based on time domain profile features. The network TP-Net of this embodiment includes feature extraction, TPM, and fusion and detection parts, such as Figure 1 to Figure 3 As shown in the figure. The architecture is simple and efficient. First, the long sequence is input into the time-domain difference convolution (TD-Conv) layer and three space-time difference residual blocks (STD-ResBlocks) to extract coarse spatiotemporal features and locate salient areas. Then, the features are processed using TPM to obtain the temporal profile features. After the feature fusion and segmentation head, the final confidence map of all input frame images is generated. The temporal profile dimension makes it unnecessary to extract fine spatiotemporal features, so TP-Net can achieve excellent performance with higher efficiency.

[0037] The purpose of feature extraction is to highlight the target and obtain its basic spatiotemporal features. In order to preserve the details of the target (especially weak targets), all 3D convolutions in this part are differential convolutions, including TD-Conv and spatial domain differential convolution (SD-Conv).

[0038] First, hundreds of frames of images Input a convolution kernel with a large TD-Conv layer It can help preserve and emphasize the temporal saliency of weak targets by applying convolution operations to inter-frame differential feature maps. Figure 1 As shown in the figure, the inter-frame difference feature is generated by calculating the difference between a certain frame feature and the features of two adjacent frames. The convolution calculation formula in TD-Conv can be described as:

[0039] ;

[0040] in is the first The pixel matrix of the frame image, yes No. A weight matrix.

[0041] The obtained features are processed by three STD-ResBlocks in sequence. Each STD-ResBlock contains a TD-Conv layer with a time domain expansion rate of 2 and a spatial domain with expansion rate 2 and Convolution Kernel SD-Conv layer Each convolutional layer consists of convolution, batch normalization, and ReLU activation operations.

[0042] SD-Conv can apply convolution operations to the center pixel difference feature map to help maintain and emphasize the spatial saliency of dark objects. This process can be described as:

[0043] ;

[0044] in represents the index of the center of the space, is the input feature of this layer, is the first pixel vectors, yes The A weight vector.

[0045] In order to check whether the signal of a certain pixel contains the target signal, we design a time-domain probe matching to match the target features in the time-domain profile dimension. Since the autocorrelations of the target signal and the clutter signal are significantly different, we use a learnable correlation matrix To realize time domain detection, the matrix can reflect the correlation of signals at different times.

[0046] Specifically, the input features First replaced by , and then apply the time domain probe to this feature pixel by pixel and channel by channel to obtain the matching feature :

[0047]

[0048] in Represents a matrix permutation operation.

[0049] According to the target motion process, the target signal includes at least three stages. In order to represent the time domain profile characteristics in detail, we need a set of time domain probes, so TPM has A time domain probe. Figure 3 As shown, The pixel probe has The correlation matrix (i.e. After time domain probe matching, a series of matching features can be obtained (i.e. ).

[0050] 1.%2 Feature Fusion

[0051] In order to fuse the matching information from different channels and time domain probes, we Convolutional Layer Acting on matching features, this process can be described as:

[0052]

[0053]

[0054] in , and They represent convolution operation, batch normalization and ReLU activation operation respectively.

[0055] We evaluate the method of this embodiment on the NUDT-MIRSDT dataset and the IRDST dataset. The IRDST simulation dataset contains 102077 frames from 316 sequences. The NUDT-MIRSDT dataset contains 120 sequences, each of which includes 100 frames. The test set of the NUDT-MIRSDT dataset is divided into two subsets according to their SNR. One consists of 8 sequences with SNR lower than 3, and the other consists of 12 sequences with SNR ranging from 3 to 10.

[0056] The detection rate (Pd), false alarm rate (Fa) and area under the ROC curve (AUC) were used as evaluation indicators of IRST detection performance. Pd and Fa are the recall rate and single-frame image threshold crossing rate at a specific threshold, and AUC is the area under the receiver operating characteristic (ROC) curve, which reflects the detection performance of the method at different thresholds.

[0057] All deep learning based methods are implemented in PyTorch on a computer equipped with an Intel Xeon Gold 182 6328H CPU @ 2.80 GHz and two Tesla V100s PCIe 32 GB GPUs. Specifically, the network is trained for 32 epochs with a batch size of 4. We use the Adam optimizer with an initial learning rate of 0.001 and the learning rate is decayed every 10 epochs with a decay rate of 0.7. We initialize the weights of the convolutional layers using the Kaiming method.

[0058] The input of TP-Net is 100 consecutive frame images (T=100), and the number of pixel probes is set to 8 (m=8). For data augmentation, we randomly select any frame as the first frame. If the number of subsequent frames is less than 99, the image with all zero values ​​is padded. We use random cropping with a patch size of 128×128 and ensure that the probability of the existence of an object in the sample is greater than 75%. The loss calculation adopts the Soft-IoU loss function. In the inference stage, we set the repetition rate to 10% to ensure that the objects at both ends of the input sequence can be well detected.

[0059] The method of this embodiment is compared with the most advanced IRST detection method on the NUDT-MIRSDT dataset and the IRDST simulation dataset. The quantitative results are shown in Table 1 and the qualitative results are shown in Table 2. Figure 4 As shown in Figures 5(a) to 5(c).

[0060] We compare the qualitative results of different detection methods tested on the NUDT-MIRSDT dataset, such as Figure 4 Compared with other methods that use spatial saliency features, temporal saliency features, and motion features, our TP-Net achieves better results in weak target and high fluctuation scenarios by introducing the temporal profile dimension. We further study the ROC of different methods on two subsets of NUDT-MIRSDT and the entire test set to demonstrate the superiority of the temporal profile dimension in IRSDT feature learning. As shown in Figure 5 (a)-Figure 5 (c), our TP-Net has a good performance in general. and extreme cases We achieve excellent detection performance in all cases, while the performance of other methods degrades significantly in extreme cases.

[0061] The quantitative results are shown in Table 1. Our TP-Net outperforms the state-of-the-art methods, especially on the IRSDT dataset. On the low SNR subset, our TP-Net outperforms the second best result by Improved , and Reduced Typically, multi-frame methods have lower inference speed than single-frame methods due to their higher computational complexity. However, among deep learning-based methods, our TP-Net achieves the least number of parameters and the best inference efficiency while maintaining the best performance.

[0062] Table 1. The results obtained by different state-of-the-art methods on the NUDT-MIRSDT dataset and the IRDST-SIMULATION dataset.

[0063]

[0064] In Table 1, the best results are shown in italic bold and the worst results are shown in bold.

[0065] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting small infrared targets based on time domain profile features, characterized in that: The following steps are involved: S1, obtaining image sequence data; S2, extracting inter-frame difference features of the image sequence data in time frame order; S3, using the extracted inter-frame difference features as inputs of N spatiotemporal difference modules to extract spatiotemporal features; S4, performing time-domain probe matching on the spatiotemporal features to obtain a series of matching features; The spatiotemporal characteristics Replace with ; , , C represents the number of channels, T represents the time dimension, H and W represent the height and width of the feature map respectively; Use the following formula to obtain matching features : ;in, represents the permutation operation of the matrix, is the correlation matrix; S5. Fuse all matching features to obtain the final confidence map of all input frame images.

2. The infrared small target detection method based on time domain profile characteristics according to claim 1 is characterized in that: In step S2, a time-domain difference convolution layer is used to obtain inter-frame difference features of the image sequence data.

3. The infrared small target detection method based on time domain profile characteristics according to claim 2 is characterized in that: The convolution operation of the time-domain difference convolution layer The expression is as follows: ； in, is the first The pixel matrix of the frame image, is the convolution kernel of the time-domain difference convolution layer No. weight matrix, S is the image sequence data, k represents the size of the sliding window, and l is the time domain length of the inter-frame difference feature.

4. The infrared small target detection method based on time domain profile characteristics according to claim 1 is characterized in that: The spatiotemporal difference module includes a temporal difference convolution layer and a spatial difference convolution layer which are connected in sequence.

5. The infrared small target detection method based on time domain profile characteristics according to claim 4 is characterized in that: The temporal expansion rate of the temporal difference convolution layer is 2, and the spatial expansion rate of the spatial difference convolution layer is 2.

6. The infrared small target detection method based on time domain profile characteristics according to claim 4 or 5, characterized in that: The temporal difference convolution layer and the spatial difference convolution layer both include a convolution layer, a batch normalization layer and a ReLU activation function connected in sequence.

7. The infrared small target detection method based on time domain profile characteristics according to claim 4 or 5, characterized in that: The convolution operation of the spatial difference convolution layer The expression is as follows: ； Where c is the index of the center of the space, is the input feature of the spatial difference convolution layer, is the first pixel vectors, represents the center pixel vector of the image in the sliding window, express Center and center pixel The corresponding weight vector is, is the convolution kernel of the spatial difference convolution layer The A weight vector is used to perform dot product operations with the corresponding vector of the image during the convolution process, representing the weights of the convolution kernel at different positions.

8. The infrared small target detection method based on time domain profile characteristics according to claim 1 is characterized in that: In step S5, the specific implementation process of fusing all matching features includes: Performing a batch normalization operation on the series of matching features; Input the matching features after batch normalization operation into the activation function; A convolution operation is performed on the output of the activation function to obtain a final confidence map of all input frame images.

9. The infrared small target detection method based on time domain profile characteristics according to claim 8 is characterized in that: The activation function is the ReLU activation function.

Images