A method and system for infrared small target detection based on context information

Abstract

This invention relates to a method and system for infrared small target detection based on contextual information, belonging to the field of computer vision technology. For an infrared small target dataset, a small target detection network is trained using classification loss functions, confidence loss functions, and position loss functions. Then, the trained small target detection network is used to extract features from infrared images, obtaining feature results. Finally, the extracted features are further fused, and infrared small target detection is performed on the fused features to obtain the final target detection result. Simultaneously, this invention proposes an infrared small target detection system based on contextual information. This invention does not rely on additional infrared image denoising, enhancement, or other processing modules; the training process is end-to-end, resulting in simple implementation, high performance, and strong robustness. The additional computational overhead of this invention is extremely low, which is beneficial for achieving low-latency, high-speed infrared small target detection, effectively improving the detection rate and reducing the false negative rate.

Description

Technical Field

[0001] This invention relates to a method and apparatus for detecting small targets in infrared images, specifically to a method and apparatus for detecting small infrared targets based on contextual information, belonging to the field of computer vision processing technology. Background Technology

[0002] Compared to visible light images, infrared images are unaffected by extreme weather and environments, can form images without external lighting, have strong detection capabilities, and long operating ranges. From infrared surveillance systems to infrared guidance systems, infrared images have significant research and application value in both civilian and military fields. However, compared to visible light images, infrared images suffer from drawbacks such as lower resolution, blurred images, and lower signal-to-noise ratios, and small objects are easily obscured by noise. Therefore, effectively detecting small targets in infrared images is a challenging task that has attracted widespread attention from the signal processing and computer vision communities.

[0003] Small object detection is a technique for detecting small objects in images. This technique can detect the category and location of small objects under natural lighting conditions. Currently, most small object detection methods are based on deep learning and deep convolutional neural networks, and this technology is widely used in fields such as surveillance and security, autonomous driving, and remote sensing satellites. According to the definition of the COCO dataset, objects smaller than 32×32 pixels are generally considered small objects. Small objects occupy a small percentage of pixels, and their detection performance is significantly lower compared to large objects. If the detection scene is complex, such as with objects occlusion, objects being obscured by the background, or densely packed objects, small objects are affected more severely than large objects, further increasing the difficulty of small object detection.

[0004] Contextual information is crucial because objects typically exist within a specific environment. Besides the object's inherent characteristics, there's a close connection between the object and its surroundings; this information is known as feature contextual information. Infrared images are heavily affected by noise and clutter, making small targets highly susceptible to interference. Therefore, utilizing other target-related information within the image, combined with the features of small targets, can effectively improve detection results and reduce the probability of missing small targets. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and deficiencies of existing infrared small target detection technologies. To solve the problems of insufficient utilization of local contextual information surrounding the target and global contextual information in the overall image, inadequate adaptability to variations in the features of different types of small targets, and improper fusion of shallow and deep features, this invention creatively proposes an infrared small target detection method and system based on contextual information. This invention effectively improves the performance of infrared small target detection and has good practical application results.

[0006] To achieve the above objectives, the present invention employs the following technical solutions.

[0007] A method for detecting small infrared targets based on contextual information includes the following steps:

[0008] Step 1: Obtain and process the infrared small target dataset.

[0009] Step 2: Train the small object detection network using the classification loss function, confidence loss function, and location loss function.

[0010] Step 3: Use the trained small target detection network to extract features from the infrared image to obtain the feature results. In this method, no preprocessing of the infrared image is required.

[0011] Step 4: Further fuse the extracted features, and perform infrared small target detection on the fused features to obtain the final target detection result.

[0012] To achieve the objectives described in this invention, this invention further proposes an infrared small target detection system based on contextual information, including an image processing module, a target information learning module, a feature extraction module, a feature fusion module, and a target detection module.

[0013] Beneficial effects

[0014] Compared with the prior art, the method and system of the present invention have the following advantages:

[0015] 1. This invention does not rely on additional infrared image denoising, enhancement, or other processing modules. The training process is performed end-to-end, making it simple to implement, high-performance, and robust.

[0016] 2. The invention has extremely low additional computational overhead, which is conducive to achieving low latency and high speed infrared small target detection, effectively improving the detection rate of small targets, reducing the false negative rate, and also improving the accuracy of targets of other scales. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention.

[0018] Figure 2 This is a schematic diagram of the feature extraction method described in this invention.

[0019] Figure 3 This is a schematic diagram of the framework and internal details of feature fusion described in the method of the present invention.

[0020] Figure 4 This is a flowchart of the system of the present invention. Detailed Implementation

[0021] To better illustrate the purpose and advantages of this invention, the invention will be described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, an infrared small target detection method based on context information includes the following steps:

[0023] Step 1: Obtain the infrared small target dataset and perform data augmentation processing.

[0024] High-quality datasets are essential for achieving good performance in deep learning-based infrared small target detection methods. However, existing infrared small target datasets are limited in number and size, with a poor proportion of small targets. Therefore, this invention first expands the infrared small target dataset through data augmentation, thereby enhancing the robustness of the entire detection method.

[0025] Specifically, in an image containing small infrared targets, small targets that do not overlap with other targets are identified and randomly copied and pasted to other locations in the image. These copied small targets must not obscure other targets and must maintain a distance from them.

[0026] Furthermore, based on copying and pasting small targets, other data augmentation operations can be superimposed (such as rotation, translation, scaling, cropping, mosaic enhancement, etc.), among which mosaic enhancement is preferred.

[0027] Step 2: Train the small object detection network using the classification loss function, confidence loss function, and location loss function.

[0028] Specifically, let the total loss function L(x,x') be expressed as:

[0029]

[0030] Where x and x' represent the predicted value and the actual value, respectively, and α box α obj α cls L represents the weights of the three loss functions. CIoU L obj L cls Let k and s represent the location loss function, confidence loss function, and classification loss function for the object detection task, respectively; 2 B represents the output feature map, the grid, and the number of anchors (i.e., positions) on each grid, respectively. kij This indicates whether the k-th output feature map, the i-th grid, and the j-th anchorbox are positive samples; 1 indicates a positive sample, and 0 indicates a negative sample. α k Used to balance the weights of output features at different scales.

[0031] Step 3: Use the trained small target detection network to extract features from the infrared image to obtain the feature results. Furthermore, no preprocessing of the infrared image is required.

[0032] Conventional infrared small target detection methods mainly extract the features inherent in the object itself, lacking the ability to acquire and analyze contextual information. Furthermore, the texture information of small targets is insufficient, and without the supplementation and enhancement of contextual information, the complex background of low-contrast infrared images can easily obscure them. The different shapes of small targets can also pose certain difficulties for detection.

[0033] Therefore, this method proposes to first extract features from the image. During the extraction process, the different contextual information requirements of features of different shapes are fully considered, and dynamic contextual information extraction is employed, such as... Figure 2 As shown, long-distance dependencies are established between various information segments of the features. The location encoding added to the input compensates for the lack of location information in the features, improving the problem of insufficient feature information for infrared targets at a distance. The deep input features are block-flattened into sequences and location information is introduced before being fed into a multi-head attention mechanism for weighted summation. Then, residual connections are used to optimize the results and accelerate convergence, through two fully connected layers followed by another residual connection. The subsequent layer is a deformable convolution, with a bias term added during convolution to ensure that even when objects of different shapes and sizes exist at the same location, their features can still be well represented.

[0034] Specifically, the process of extracting dynamic context information is as follows:

[0035] For an input feature F, its feature size is C×H×W, where C represents the number of channels, H represents the height, and W represents the width; given the block size P, C×H×W is divided into N P×P×C blocks, where P represents a block.

[0036] After obtaining N blocks, linearly transform them into N feature vectors of length, and add a flag vector x at the beginning of each feature vector. p ;

[0037]

[0038] F1 = E + F0

[0039] Where F0 represents the output vector result, W represents the Nth block. N The weight parameter is 'Concat[]', which is the concatenation operation.

[0040] The final result F0 is the output of the block embedding. After obtaining F0, the embedded features still lack the relative position information between blocks. Therefore, the position encoding information E is added to F0 to obtain F1, which represents the result after adding the position information.

[0041] The F1 algorithm, after embedding location information, is multiplied by three different parameter matrices, mapping them to a query matrix, a key-value matrix, and a value matrix. After attention processing, multiple attention results are obtained, representing different contextual information in the image. These attention results are concatenated and standardized to obtain the final summary of contextual information.

[0042]

[0043] head i =Attention(F1W) q F1W k F1W v )

[0044] F M =Concat[head] i ;head i ;head i ;...;head i W M

[0045] In this context, Attention() represents the attention mechanism operation, Q, K, and V represent the query matrix, the key-value matrix being queried, and the value matrix, respectively, and T represents the transpose operation. Indicates the scaling factor; F1 represents the result after adding position information, W q W k W v W M It is a learnable parameter matrix, Softmax indicates that the Softmax operation is performed, and head i F represents the output of multiple attention results. M This represents the output characteristics of multi-head attention. Concat represents the addition operation.

[0046] The feedforward neural network consists of two fully connected layers, and the multi-head attention output feature F after residual normalization. M The first fully connected layer maps the data to a high-dimensional space, while the second fully connected layer maps the data to the low-dimensional space, further preserving useful information. The process is as follows:

[0047] F2 = F M [0]+F1

[0048] X = F2W fc1 Wfc2 +F1

[0049] Where F2 represents the result after residuals, F M [0] represents the flag vector, X represents the output result, and W... fc1 W fc2 These are the weights of the two fully connected layers.

[0050] After processing the context information, the output X is dynamically adjusted using deformable convolution to connect different small targets with the context information:

[0051]

[0052] Where Y(p0) represents the deformable convolution output, X and Y are the input and output feature maps respectively, and p0 represents the position in the output feature map. n Let R represent adjacent positions, and R represent the range of real numbers. The function W() represents p. n The weight at point Δp. n It is the offset value, which is learned by performing parallel convolutions from the input features.

[0053] Step 4: Further fuse the extracted features, and perform infrared small target detection on the fused features to obtain the final target detection result.

[0054] Due to noise, the features of different small targets in infrared images vary greatly, which severely tests the model's feature fusion capabilities. Ordinary target detection models simply perform upsampling, convolution, and feature connection operations, without analyzing the spatial position features of objects or the semantic features of objects in the channel dimension. Or they can only fuse obvious features in the image, ignoring the information of small targets, resulting in low accuracy in the final detection of small targets.

[0055] Therefore, in this method, after feature extraction from the image, the extracted features are fused, and target detection is performed on the fused features. During the feature fusion process, a multi-information fusion layer is used to aggregate channel and spatial information from multiple features.

[0056] The aggregated features significantly improve the representation of the object's positional and semantic information. Adding a new feature scale during feature fusion supplements the deep features of small targets, enriching their detailed features.

[0057] To maximize the spatiotemporal information of objects during feature fusion and retain more target features, such as... Figure 3 As shown, the multi-information fusion layer performs information fusion operations at each feature scale.

[0058] The Multi-Information Fusion (MFM) module fuses information from different layers through multiple residual structures, as shown in the following structure: Figure 3 As shown in (c), it consists of three parts, the first being the IC layer, as follows: Figure 3 (b) indicates that the information is responsible for refining the features. Then, global pooling and max pooling are performed at the channel level, and the information is processed through a fully connected layer with shared weights. After multiplication and addition, the information is normalized by the softmax function to obtain the extracted channel information. This information is then multiplied with the input information to enhance the channel information.

[0059] After enhancing the channel information, global pooling and max pooling are performed separately at each location in the image. These sums are then combined using a 7×7 convolution to stack the features and normalized using the softmax function, achieving enhanced positional information. Finally, a 1×1 convolution is applied to further integrate the channel and spatial information.

[0060] Deep features contain rich semantic information, but this information is mostly target-related. Features of small targets are easily obscured by noise after multiple downsampling operations, making them difficult to locate. Shallow features, on the other hand, possess rich texture and location information for small targets. Furthermore, to effectively utilize shallow features to enhance the detail and location information of small targets, an additional feature scale is added to specifically focus on small objects, and a detection head is added to output the detection results. The related structure naming is as follows: Figure 3 As shown in (a), the outputs of the dynamic context information extraction module and the subsequent three multi-information fusion (MFM) modules are T5, T4, T3, and T2, respectively. The sizes of these outputs are 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the original image. Features of the same size connected to T5, T4, and T3 are denoted as R4, R3, and R2.

[0061] This method further upsamples the features when processing the feature map up to layer T3, and then adds it to layer T2. Simultaneously, layer T2 is connected to the features of the same size in the second layer of the backbone network. This improves the representation of small object details, conveys shallow detail information, and adds a small object detection head after layer T2 to reduce feature coupling between small objects and other objects in the same layer, thereby reducing the false negative rate of small objects, increasing the probability of detecting small objects, and mitigating the poor accuracy caused by excessively large scales. To correspond with the channels of subsequent networks, layer R2 is added after layer T2, connected to the features of layer T3 with the same dimension.

[0062] To achieve the objectives described in this invention, this invention further proposes an end-to-end infrared small target detection system based on context information, such as... Figure 4 As shown, it includes an infrared image processing module 10, a target information learning module 20, a feature extraction module 30, and a feature fusion and target detection module 40.

[0063] The infrared image processing module 10 is used to process the infrared image dataset used to train the small target detection model. This module can increase the number of small targets, enrich the changing scenarios of the dataset, and enhance the robustness of the model.

[0064] The small target information learning module 20 is used to guide the small target detection model to learn robust image features. This module uses information learning from the infrared small target dataset to train the model and outputs the trained small target detection model.

[0065] The image feature extraction module 30 utilizes the dynamic context information extraction module to extract target-related information and global information from image features, adapting to the contour changes of different small targets. Stable and clean small target features are extracted from the infrared image to achieve accurate infrared small target detection.

[0066] The feature fusion and target detection module 40 can fuse the extracted features. From the fused image features, it identifies and extracts the category, location, size, and shape of the target of interest, obtaining the final infrared small target detection result.

[0067] The connection relationships between the above modules are as follows:

[0068] The output of the infrared image processing module 10 is connected to the input of the small target information learning module 20.

[0069] The output of the small target information learning module 20 is connected to the input of the image feature extraction module 30.

[0070] The output of the image feature extraction module 30 is connected to the input of the feature fusion and target detection module 40.

Claims

1. A method for detecting small infrared targets based on context information, characterized in that, The method comprises the following steps: Step 1: Obtain an infrared small target data set and perform data enhancement processing; Step 2: Train the small target detection network using a classification loss function, a confidence loss function and a position loss function; Step 3: Extract features from the infrared image using the trained small target detection network to obtain feature results; First, the image is subjected to feature extraction; in the extraction process, dynamic context information extraction is used to establish long-distance dependencies between various information; after the input deep features are divided into blocks and flattened into a sequence and position information is introduced, the multi-head attention mechanism is used for weighted summation; Then, the residual connection is used to optimize the results and accelerate the convergence, and two fully connected layers are used and then the residual connection is used again; followed by a deformable convolution, and a bias term is added during the convolution; Step 4: Further fuse the extracted features, and perform infrared small target detection on the fused features to obtain the final target detection results; In the feature fusion process, the multi-information fusion layer is used to aggregate the channel and spatial information in the multiple features, and the multi-information fusion layer performs information fusion operation in each feature scale; The multi-information fusion module fuses the information of different layers through multiple residual structures, which includes three parts: the first is the IC layer, which is responsible for refining the information of the features, then global pooling and maximum pooling are performed on the channel level respectively, and the information is arranged through a shared weight fully connected layer, multiplied and added, and then normalized through a softmax function to obtain the extracted channel information, which is multiplied with the input information; After the channel information is extracted and enhanced, global pooling and maximum pooling are further performed on each position of the image, added, and then the convolution is used to stack the features and normalized through a softmax function to achieve the effect of enhancing the position information; finally, the channel and spatial information are integrated through convolution.

2. The method for detecting small infrared target based on context information according to claim 1, characterized in that, When performing data enhancement processing in step 1, find the small target that does not overlap with other targets in the image containing the infrared small target, and randomly copy and paste it to other positions of the image; wherein the copied small target does not block other targets and maintains a distance from other targets.

3. The method of claim 2, wherein the context information is obtained by using a plurality of pixels in the image.

3. The method of claim 2, wherein the context information is obtained by using a plurality of pixels in the image. On the basis of copying and pasting the small target, further superimpose other data enhancement operations, including rotation translation, scaling cropping, mosaic enhancement.

4. The method of claim 1, wherein the context information is obtained from a context database. 5 In step 2, the total loss function L(x, x') is expressed as: where x, x' represent the predicted value and the true value respectively, a box , a obj , a cls represent the weights of three loss functions respectively, L CIoU , L obj , L cls represent the position loss function, the confidence loss function and the classification loss function of the target detection task respectively; k, s 2 , B represent the output feature map, the grid and the number of positions on each grid respectively, I kij represents whether the kth output feature map, the ith grid and the jth anchor box are positive samples, if they are positive samples, then 1, if they are negative samples, then 0; a k is used to balance the weights of output features of different scales.

5. The method of claim 1, wherein the context information is obtained from a context database. 5 In step 3, the process of dynamic context information extraction is as follows: For the input feature F, the feature size is CxHxW, where C represents the number of channels, H represents the height, and W represents the width; given the size P of the block, CxHxW is divided into N PxPxC blocks, where P represents the block; After obtaining the N blocks, linearly transform them into N characteristic vectors of length, and add a flag bit vector x at the beginning of the vector p ; F1 = E + F0 wherein F0represents a vector result of output, represents the Nth block, W N is a weight parameter, and Concat[] is a concatenation operation; and the final F0is an output result of the block embedding. Add the position encoding information E to F0 to obtain F1, which represents the result after adding the position information; After embedding the position information, F1 is multiplied by three different parameter matrices to map to the query matrix, the key-value matrix to be queried and the value matrix; after attention mechanism processing, multiple attention results are obtained to represent different context information in the image; the attention results are spliced and standardized to obtain the final context information summary result: head i = Attention(F1W q ; F1W k ; F1W v ) F M = Concat[head i ; head i ; head i ;... ; head i ] W M Wherein, Attention() represents an attention mechanism operation, Q, K and V represent a query matrix, a key-value matrix to be queried and a value matrix respectively, T represents a transposition operation, represents a scaling factor; F1 represents a result after adding position information, W q , W k , W v , W M is a learnable parameter matrix, Softmax represents performing a Softmax operation, head i represents an output of multiple attention results, F M represents a multi-head attention output feature; Concat represents an addition operation; The feedforward neural network includes two fully connected layers, and the residual normalized multi-head attention output feature F M The low-dimensional space is mapped by the second fully connected layer, further retaining useful information, and the process is as follows: F2 = F M [0] + F1 X = F2W fc1 W fc2 +F1 wherein F2 represents a result after residual, F M [0] is a flag bit vector, X is an output result, W fc1 , W fc2 are weights of two full connection layers; After processing the context information, the output result X is dynamically adjusted by deformable convolution to adjust the effective information and contact the relationship between different small targets and context information: where Y(p0) represents the deformable convolution output result, X and Y are input and output feature maps respectively, p0 represents a position in the output feature, p n represents an adjacent position, R represents a real number range; the function W() represents a weight at p n ; Δp n is an offset value, which is learned by performing parallel convolution on the input feature.

6. The method for detecting small infrared targets based on context information according to claim 1, characterized in that, A feature scale is added in step 4 to focus on small objects; a detection head is added to output the detection result; The outputs of the dynamic context information extraction module and the subsequent three multi-information fusion modules are T5, T4, T3, and T2, and the sizes of these outputs are 1 / 32, 1 / 16, 1 / 8, and 1 / 4 of the original image, respectively. The same size features connected to T5, T4, and T3 are denoted as R4, R3, and R2. When the feature map is processed to the T3 layer, the feature is upsampled and added to the T2 layer after upsampling, and the T2 layer is connected to the same size feature of the second layer of the backbone network; a small target detection head is connected after the T2 layer, and the R2 layer is connected to the T3 layer with the same dimension.

7. A context information based infrared small target detection system implementing the method of claim 1, characterized by The infrared image processing module (10), the target information learning module (20), the feature extraction module (30), and the feature fusion and target detection module (40) are included. The infrared image processing module (10) is used to process the infrared image dataset for training the small target detection model. The small target information learning module (20) is used to guide the small target detection model to learn robust image features; this module uses the infrared small target dataset to train the model using information learning, and outputs the trained small target detection model; The image feature extraction module (30) extracts the target surrounding information and global related information in the image features using the dynamic context information extraction module, and adapts to the contour changes of different small targets; it extracts stable and clean small target features from the infrared image; The feature fusion and target detection module (40) fuses the extracted features, identifies and extracts the class position size shape of the target of interest from the fused image features, and obtains the final infrared small target detection result; The connection relationship between the above modules is as follows: The output end of the infrared image processing module (10) is connected to the input end of the small target information learning module (20); The output end of the small target information learning module (20) is connected to the input end of the image feature extraction module (30); The output end of the image feature extraction module (30) is connected to the input end of the feature fusion and target detection module (40).

Images