Infrared small target detection method based on semi-supervised multi-view prototype learning

Abstract

The invention discloses an infrared small target detection method based on semi-supervised multi-view prototype learning, and the method comprises the steps: firstly constructing an infrared motion small target detection network based on a teacher-student network, designing a bidirectional time motion sensor, and fully simulating a motion normal form of a target through forward and backward sensing, and generating a multi-view feature prototype as a soft label to explore a position label, and in combination with target perception motion reconstruction loss, achieving the purpose of detecting a mobile infrared small target on a large-scale unmarked video data set. The method shows higher accuracy and robustness in the aspect of infrared motion small target detection, can effectively detect the small target appearing in the infrared motion small target sequence image collected by the infrared imaging equipment, and has wide application prospects in the important application fields of unmanned aerial vehicle supervision, security and protection and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an infrared small target detection method based on semi-supervised multi-view prototype learning. Background Art

[0002] Infrared moving small target detection has always been an important and challenging problem in the field of computer vision. Different from existing target detection tasks, it has high concealment, strong adaptability and excellent anti-interference ability, and has a wide range of important applications such as traffic management and maritime rescue. Therefore, improving performance has been a continuous focus of computer vision research. However, compared with general objects, infrared small targets often have two unique characteristics. One is the lack of obvious edge and texture features due to the long imaging distance. Another reason is that the dim-small targets in these images or videos usually have low contrast and signal-to-noise ratio against complex backgrounds. In addition, due to the particularity of infrared images and the limitations of expensive equipment, there is currently a lack of large-scale labeled datasets.

[0003] In order to solve the problem of difficult infrared small target detection, researchers have made various attempts, which can be divided into single-frame and multi-frame schemes according to the number of frames processed. Single-frame schemes only use the visual features of a single image, and they usually have the obvious advantages of high detection speed and low model complexity. Although these methods have achieved remarkable performance, they often lead to detection errors in complex scenes. In contrast, multi-frame schemes extract target features from visual and motion patterns, and they usually focus on capturing target motion features from continuous video frames to improve detection accuracy.

[0004] According to whether training data is required, the current mainstream methods can be divided into model-driven methods and data-driven methods. In model-driven methods, researchers usually rely on complex manual feature extraction and model parameter adjustment to calculate the maximum contrast between the central pixel of the image and the neighboring area or use the structure of the infrared image to separate the background from the target. However, they rely heavily on prior knowledge and hand-crafted features and have almost no learning ability, which limits their adaptability to real complex scenes. In data-driven methods, target features are usually adaptively learned, which has achieved excellent performance and become the mainstream solution. However, almost all data-driven methods are based on fully supervised learning. This type of pipeline relies on a large number of training samples with high-quality annotations. Annotating large-scale target detection datasets is very expensive and time-consuming, especially for small moving infrared targets.

[0005] Currently, semi-supervised object detection methods have received extensive attention. They can train object detectors with a small number of labeled images and a large number of unlabeled images, however, this type of scheme usually deals with general objects with unique visual features. Since small infrared targets are usually dark, blurry, and have complex backgrounds, these methods often cannot effectively extract target features and correctly estimate pseudo labels. In addition, almost all semi-supervised object detection methods process independent images in a single-frame manner. They often ignore the rich temporal relationships in videos, which can serve as valuable clues for motion feature learning. Therefore, it is usually not suitable to directly apply existing methods of this type to mobile infrared small target detection.

[0006] In summary, the existing mobile infrared small target detection scheme has three obvious defects. First, the fully supervised method has limitations when faced with limited labeled data, and requires expensive label costs to improve performance. Second, for unlabeled data, it is difficult for ordinary models to achieve effective feature learning. Third, in the model training process, especially when using semi-supervised or weakly supervised methods, the quality of pseudo-labels (that is, labels predicted by the model) is often uneven, and there are noisy labels. These noisy labels will affect the training effect of the model and cause the model performance to deteriorate. Finally, in tasks such as video processing and motion capture, the motion patterns of target objects are often complex and changeable, and the existing loss functions may not be able to fully capture these motion patterns, resulting in the loss of target details during reconstruction or prediction. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes an infrared small target detection method based on semi-supervised multi-view prototype learning.

[0008] The technical solution adopted by the present invention is: a method for detecting small infrared targets based on semi-supervised multi-view prototype learning, and the specific steps are as follows:

[0009] S1. Construct an infrared moving small target detection network based on teacher-student network;

[0010] The target detection network is a teacher-student network, including: a teacher network, a student network, an abnormality-driven pseudo-label filtering module, and a target perception motion reconstruction module.

[0011] The teacher network and the student network both include: a feature extraction module and a multi-view prototype learning module. The feature extraction module includes: a basic feature extraction module and a bidirectional temporal motion sensor module; the multi-view prototype learning module includes: a prototype learning module, a detection head module, and a prototype pool module.

[0012] S2, collect infrared images, build an infrared image sequence data set, and divide it into a training set and a test set according to actual conditions;

[0013] S3, input the training set into the target detection network constructed in step S1 for training, the teacher network is updated from the student network through the EMA attention mechanism, and then the parameters of the target detection network are iteratively optimized based on the gradient descent method, and the optimal target detection network parameters are selected based on the test set to obtain the optimal target detection network;

[0014] S4. Based on the optimal target detection network obtained in step S3, the sequence image in the sampling window to be detected is input to obtain the final detection result.

[0015] Furthermore, the step S3 is specifically as follows:

[0016] S31, the training data set is input into the basic feature extraction module, Resnet50 is used for basic feature extraction, and then input into the bidirectional temporal motion sensor module for feature optimization;

[0017] First, we randomly extract two T-frame image sequences from the infrared video of the training set as input, namely, I U ={U1,U2,…,U T}, I L ={L1,L2,…,L T}.

[0018] Among them, I U represents the unlabeled dataset, I L represents a labeled dataset, and U T and L T It represents the keyframes that need to be detected, and the rest are reference frames that provide time information.

[0019] Then, the pre-trained ResNet50 is used as the basic feature extractor, and each frame of the image sequence is input into the extractor to obtain multi-frame visual features. and

[0020] in, represents the real number domain, n represents the number of feature extractor output layers, and T represents the time window size of frame sampling. C, H, and W represent the channel, height, and width of the feature tensor, respectively.

[0021] For the student network, the bidirectional temporal motion perception module receives multi-frame visual features X U and X L As its input, the spatiotemporal features M are generated respectively U and M L ; The teacher network only accepts X U Generate M U The details are as follows:

[0022] As a feature optimization module, the bidirectional temporal motion sensor module constructs a coupled bidirectional structure through the ConvGRU unit, simulating the use of spatiotemporal features in the past, present and future frames of global time encoding to fully perceive the motion trajectory of the target.

[0023] First, a coupled bidirectional structure is constructed using the ConvGRU unit. The ConvGRU cell includes a reset gate and an update gate

[0024] In the labeled data, multi-frame visual features X L ={X1,…,X t ,…X T} represents the input of ConvGRU. L ={H1,…,H t ,…H T} represents its hidden state. Then the overall update process of ConvGRU cells over time is expressed as follows:

[0025]

[0026] Where σ(·) and α(·) represent the sigmoid activation function and the tanh activation function, respectively. s (·) and f c (·) indicates a connection operation and a split operation respectively. “*” indicates a convolution operation. Represents element-wise product. and b represent the weight matrix and bias term in the corresponding convolutional layer.

[0027] The bidirectional temporal motion sensor module stacks two ConvGRU units front and back to perform global motion modeling by accumulating dual temporal motion dimensions. The calculation process expression is as follows:

[0028]

[0029] in, and They represent the forward and backward hidden states of the ConvGRU unit calculated by equation (1). t represents the visual features of the tth frame. And H t Represents the final output of the current cell and the input of the next cell.

[0030] Then we get the forward motion features of the labeled data Then get the reverse kinematic characteristics Similarly, the forward motion features of unlabeled data can be obtained and inverse kinematics features

[0031] Finally, these two types of motion patterns are aggregated through a convolutional layer with a 3×3 kernel and a stride of 1 to generate spatiotemporal features, which are expressed as follows:

[0032]

[0033] Among them, M U and M L Represent the multi-frame spatiotemporal features of unlabeled images and labeled images respectively.

[0034] S32, performing prototype learning on the feature information optimized by the bidirectional temporal motion sensor module in step S31 based on the multi-view prototype learning module;

[0035] The multi-view prototype learning module generates a multi-view feature prototype of the target into the prototype pool module through a set of prototype generators PG between high-level semantics and low-level details in the prototype learning module and the task alignment head of the task-aligned single-stage target detection TOOD in the detection head module, as follows:

[0036] The multi-scale keyframe features M are obtained through the bidirectional temporal motion perception module k ={M1,M2,…,M i ,…,M n} as the prototype generator input, M k It is M L or M U The last feature set of , n represents the number of feature extractor output layers.

[0037] For each prototype generator in the multi-view prototype learning module, the feature prototype of the previous layer and the key frame features corresponding to the current prototype generator are input into it to obtain the output of the next layer.

[0038] First, the feature prototype P i+1 Upsampling is performed to maintain the same i The same resolution, and concatenate them to obtain the fused feature F c .

[0039] Then, 3×3 depth convolution DW(·) and 1×1 convolution PW(·) are introduced to combine local and global target information to obtain an accurate offset map O m , the calculation process expression is as follows:

[0040]

[0041] Among them, f up (·) indicates bilinear interpolation operation.

[0042] Offset map in Pi+1 The upper guidance grid sampling obtains P′ i+1 , and add P′ i+1 The feature prototype and fusion feature F after sampling c , we get the target features of different views, and the calculation expressions are as follows:

[0043]

[0044] Among them, Grid(·) represents the grid sampling operation, F p Represents target features in different views.

[0045] Using non-local attention, the similarity between all positions in the feature prototype is calculated, and the expression is as follows:

[0046] Q,K,V=f p (F p )

[0047]

[0048] Among them, Q, K, V represent the query vector Q, key vector K and value vector V in the Transformer's self-attention mechanism, T represents the transposition operation, and f p (·) represents the projection operation implemented by 1×1 convolution, γ represents the weight coefficient, d represents the dimension, and P i Represents the new feature prototype of level i.

[0049] Similarly, M is processed in the same way k Finally, we obtain the multi-view feature prototype set of the target, namely P = {P1,…,P i ,…,P n}.

[0050] In each iteration, after the student network obtains the multi-view feature prototype set of the target, the student network uses the spatiotemporal features M of the labeled data L Initialize the multi-view prototype memory pool P M = {P1,P2,…P S}, then the multi-view feature prototype is lost through the multi-view prototype Guided learning of students on unlabeled data.

[0051] Where S represents the size of the multi-view prototype memory pool.

[0052] In each iteration, after the teacher network obtains the target multi-view feature prototype set, it uses P generated by the teacher network to update the multi-view prototype memory pool P through a first-in-first-out strategy. M , that is, using the spatiotemporal features M of the unlabeled data UUpdate the target's feature prototype.

[0053] S33, based on step S32, inputting the pseudo labels predicted by the unlabeled data in the teacher network into the abnormal driven pseudo label filtering module for filtering;

[0054] The anomaly-driven pseudo-label filtering module integrates isolation forest and Mahalanobis distance to obtain a dynamic threshold for detecting abnormal pseudo-labels, which is used to filter the pseudo-labels predicted by the teacher network on unlabeled data to obtain an unsupervised loss.

[0055] First, in each iteration, the center of all pseudo boxes is calculated and the pseudo label b is set i =(x l ,y l ,x r ,y r )'s center C i yes

[0056] Then calculate the Mahalanobis distance of each pseudo-frame, the expression is as follows:

[0057]

[0058] Among them, μ represents the average value of all center points, Ω represents the covariance matrix, and D M (C i ) means C i The Mahalanobis distance between all pseudo labels, and N represents the maximum total number of pseudo labels.

[0059] Then, an isolation forest is built on all pseudo labels, which contains several random trees to represent the data. Each node corresponds to a pseudo label. i The path length h i To calculate the anomaly score of each pseudo label, the expression is as follows:

[0060]

[0061] Among them, S a (C i ) means C i The anomaly score, E(·) represents the expected value, c(N) represents the normalized coefficient of the path length, and f h (N-1) represents the harmonic number, and β≈0.5772 represents the Euler constant.

[0062] Finally, combined with D M (C i ) and S a (C i ) to comprehensively detect abnormal pseudo labels, the expression is as follows:

[0063] S i =η1·D M (C i )+η2·S a (C i )#(9)

[0064] Among them, S i represents the final metric of the i-th pseudo label, η1 and η2 represent two weight coefficients.

[0065] S34, based on the target-aware motion reconstruction module, providing additional supervision to the teacher network and the student network by reconstructing the motion mask to obtain the reconstruction loss;

[0066] The bidirectional temporal motion sensor module outputs the motion feature set M k ={M1,M2,…,M n}, only convolutional layers and upsampling operations are used to reduce M k The channels of each level of motion features in the prediction mask G = {G1, G2, ..., G n}.

[0067] Then initialize a ground truth motion mask G of the same size as the input image g , all elements are set to 0. Then the target coordinates (x l ,y l ,x r ,y r ) projected onto the mask In the example, the target area in the mask is assigned a value of 1. The expression is as follows:

[0068]

[0069] Among them, j and k represent the mask G g The coordinate of a pixel in . Then the target-aware motion reconstruction loss is The definition expression is as follows:

[0070]

[0071] Where n represents the number of feature extractor output layers, G g represents the real mask, G i represents the prediction mask in G.

[0072] S35, defining a relevant loss function, performing target positioning detection, error loss calculation, and iterative optimization of target detection accuracy on the output features of steps S32, S33, and S34, that is, using a gradient descent method to iteratively optimize the target detection network parameters;

[0073] The detection loss of the detection head module includes: supervision loss Unsupervised loss Multi-view prototype learning loss The total detection loss definition expression is as follows:

[0074]

[0075] Among them, λ u and λ p Two hyperparameters representing the balanced loss term.

[0076] The detection head module adopts the task alignment head of task-aligned single-stage object detection TOOD, and adds a target-aware motion reconstruction loss term to the detection loss of TOOD. and The expression is as follows:

[0077]

[0078] in, represents the loss of labeled data, represents the loss of unlabeled data. λ1 and λ2 represent two weighting coefficients. represents a classification loss, and represents a regression loss.

[0079] Student Network Utilization Prototype Memory Pool P M The feature prototype in the ,guides students’ feature learning from multiple perspectives, i.e., using the features F obtained by students on unlabeled data s With P M The cosine similarity between all feature prototypes in is expressed as follows:

[0080]

[0081] Among them, F s That is, the fusion feature F obtained by the student network c , f cos (·) indicates the calculation of cosine similarity, “·” indicates inner product, and ∥·∥2 indicates l2 norm.

[0082] Finally, based on the calculated loss, the target detection network uses the back-propagation gradient descent method to iteratively optimize the parameters and save the training network parameters for each round.

[0083] S36: Based on the network parameters saved in step S35, select mAP based on the test data set 50 The network with the best accuracy is the optimal object detection network.

[0084] Beneficial effects of the present invention: The method of the present invention first constructs an infrared moving small target detection network based on a teacher-student network, designs a bidirectional temporal motion sensor, fully simulates the target's motion paradigm through forward and backward perception, generates multi-view feature prototypes as soft labels to explore position labels, and combines target-aware motion reconstruction loss to achieve the purpose of detecting moving infrared small targets on large-scale unlabeled video data sets. The method of the present invention constructs a new semi-supervised multi-view prototype learning framework with target-aware motion reconstruction, uses a teacher-student architecture to train sparsely labeled data, improves the feature learning ability of a large amount of unlabeled data, improves detection performance, effectively solves the limitations of fully supervised methods when facing limited labeled data, and reduces expensive label costs; secondly, a multi-view prototype learning method is introduced to guide feature learning of unlabeled data, which uses the cosine similarity between predicted features and feature prototypes to improve the feature representation ability of the target, and designs an abnormality-driven filtering strategy to refine pseudo-labels, accurately eliminate noise labels, improve the quality of pseudo-labels, and ensure stable stability. Finally, a new target-aware motion reconstruction loss is designed to provide additional supervision and adapt to target motion patterns to avoid target detail loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 The present invention is a flowchart of an infrared small target detection method based on semi-supervised multi-view prototype learning.

[0086] Figure 2 This is a network structure diagram of infrared moving small target detection based on a teacher-student network in an embodiment of the present invention.

[0087] Figure 3 4 is a block diagram of a bidirectional temporal motion sensor in an embodiment of the present invention.

[0088] Figure 4 is a structural diagram of a prototype learning module in a multi-view prototype learning module in an embodiment of the present invention;

[0089] Figure 5 It is a PR curve diagram on the data set in an embodiment of the present invention.

[0090] Figure 6 Graphs showing visualization results of detection boxes under three public data sets in the embodiments of the present invention. DETAILED DESCRIPTION

[0091] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0092] like Figure 1 As shown, a flow chart of an infrared small target detection method based on semi-supervised multi-view prototype learning of the present invention is shown, and the specific steps are as follows:

[0093] S1. Construct an infrared moving small target detection network based on teacher-student network;

[0094] like Figure 2 As shown, an intelligent detection network specifically used for infrared motion small target detection is constructed, that is, a target detection network based on the target detection network. The target detection network is a teacher-student network, including: a teacher network, a student network, an abnormality-driven pseudo-label filtering module, and a target perception motion reconstruction module;

[0095] The teacher network and the student network both include: a feature extraction module and a multi-view prototype learning module. The feature extraction module includes: a basic feature extraction module and a bidirectional temporal motion sensor module; the multi-view prototype learning module includes: a prototype learning module, a detection head module, and a prototype pool module;

[0096] S2, collect infrared images, build an infrared image sequence data set, and divide it into a training set and a test set according to actual conditions;

[0097] S3, input the training set into the target detection network constructed in step S1 for training, the teacher network is updated from the student network through the EMA attention mechanism, and then the parameters of the target detection network are iteratively optimized based on the gradient descent method, and the optimal target detection network parameters are selected based on the test set to obtain the optimal target detection network;

[0098] S4. Based on the optimal target detection network obtained in step S3, the sequence image in the sampling window to be detected is input to obtain the final detection result.

[0099] In this embodiment, step S3 is specifically as follows:

[0100] S31, the training data set is input into the basic feature extraction module, Resnet50 is used for basic feature extraction, and then input into the bidirectional temporal motion sensor module for feature optimization;

[0101] First, we randomly extract two T-frame image sequences from the infrared video of the training set as input, namely, I U ={U1,U2,…,U T}, I L ={L1,L2,…,L T}.

[0102] Among them, I U represents the unlabeled dataset, I L represents a labeled dataset, and U T and L T It represents the keyframes that need to be detected, and the rest are reference frames that provide time information.

[0103] Then, the pre-trained ResNet50 is used as the basic feature extractor, and each frame of the image sequence is input into the extractor to obtain multi-frame visual features. and

[0104] in, represents the real number domain, n represents the number of feature extractor output layers, and T represents the time window size of frame sampling. C, H, and W represent the channel, height, and width of the feature tensor, respectively.

[0105] For the student network, the bidirectional temporal motion perception module receives multi-frame visual features X U and X L As its input, the spatiotemporal features M are generated respectively U and M L ; The teacher network only accepts X U Generate M U The details are as follows:

[0106] In the bidirectional temporal motion sensor module, in order to simultaneously utilize the spatiotemporal features of the past, current, and future frames to fully perceive the motion trajectory of the target, such as Figure 2 As shown in the teacher network and student network sections, we designed Figure 3 The bidirectional temporal motion sensor shown in the figure, in this embodiment, the bidirectional temporal motion sensor will make full use of the potential of backward motion features and future frames. As a feature optimization module, the bidirectional temporal motion sensor module builds a coupled bidirectional structure through the ConvGRU unit, and simulates the use of spatiotemporal features in the past, present and future frames of the global time encoding to fully perceive the motion trajectory of the target.

[0107] First, a coupled bidirectional structure is constructed using the ConvGRU unit. The ConvGRU cell includes a reset gate and an update gate With these two gates, ConvGRU can resist noise interference through selective memory and forgetting.

[0108] In the labeled data, multi-frame visual features X L ={X1,…,X t ,…X T} represents the input of ConvGRU. L ={H1,…,H t ,…H T} represents its hidden state. Then the overall update process of ConvGRU cells over time is expressed as follows:

[0109]

[0110] Where σ(·) and α(·) represent the sigmoid activation function and the tanh activation function, respectively. s (·) and f c (·) indicates a connection operation and a split operation respectively. “*” indicates a convolution operation. Represents element-wise product. and b represent the weight matrix and bias term in the corresponding convolutional layer.

[0111] The bidirectional temporal motion sensor module stacks two ConvGRU units front and back to perform global motion modeling by accumulating dual temporal motion dimensions. The calculation process expression is as follows:

[0112]

[0113] in, and They represent the forward and backward hidden states of the ConvGRU unit calculated by equation (1). t represents the visual features of the tth frame. And H t Represents the final output of the current cell and the input of the next cell.

[0114] Then we get the forward motion features of the labeled data Then get the reverse kinematic characteristics Similarly, the forward motion features of unlabeled data can be obtained and inverse kinematics features

[0115] Finally, these two types of motion patterns are aggregated through a convolutional layer with a 3×3 kernel and a stride of 1 to generate spatiotemporal features, which are expressed as follows:

[0116]

[0117] Among them, M U and M L Represents the multi-frame spatiotemporal features of unlabeled and labeled images respectively. Each contains rich motion representation.

[0118] S32, performing prototype learning on the feature information optimized by the bidirectional temporal motion sensor module in step S31 based on the multi-view prototype learning module;

[0119] like Figure 4 As shown in the figure, the multi-view prototype learning module generates a multi-view feature prototype of the target into the prototype pool module through a set of prototype generators PG (the number is set according to the actual situation) between high-level semantics and low-level details in the prototype learning module and the task alignment head of the task-aligned single-stage object detection (TOOD) in the detection head module, as follows:

[0120] The multi-scale keyframe features M are obtained through the bidirectional temporal motion perception module k ={M1,M2,…,M i ,…,M n} as the prototype generator input, M k It is M L or M U The last feature set of , n represents the number of feature extractor output layers.

[0121] For each prototype generator in the multi-view prototype learning module, the feature prototype of the previous layer and the key frame features corresponding to the current prototype generator are input into it to obtain the output of the next layer.

[0122] In order to ensure the spatial consistency of feature fusion, we first i+1 Upsampling is performed to maintain the same i The same resolution, and concatenate them to obtain the fused feature F c .

[0123] Then, 3×3 depth convolution DW(·) and 1×1 convolution PW(·) are introduced to combine local and global target information to obtain an accurate offset map O m , the calculation process expression is as follows:

[0124]

[0125] Among them, f up (·) represents the bilinear interpolation operation. The motivation for generating offset maps for keyframe features at different scales is to ensure the diversity of multi-view feature prototypes. The offset map can be dynamically adjusted according to the target's motion mode and position information. i+1 The upper guidance grid sampling obtains P′ i+1 , to avoid information loss, and add P′ i+1 The feature prototype and fusion feature F after sampling c , we get the target features of different views, and the calculation expressions are as follows:

[0126]

[0127] Among them, Grid(·) represents the grid sampling operation, F p Represents target features in different views.

[0128] Since the scale of the target is small, it is challenging to capture the global dependency of the target. Inspired by the non-local block, this embodiment adopts non-local attention to solve this problem. By calculating the similarity between all positions in the feature prototype, the global understanding of the target can be improved and the background noise can be suppressed. The expression is as follows:

[0129] Q,K,V=f p (F p )

[0130]

[0131] Among them, Q, K, V represent the query vector Q, key vector K and value vector V in the Transformer's self-attention mechanism, T represents the transposition operation, and f p (·) represents the projection operation implemented by 1×1 convolution, γ represents the weight coefficient, d represents the dimension, and the use Normalized, P i Represents the new feature prototype of level i.

[0132] Similarly, M is processed in the same way k Finally, we obtain the multi-view feature prototype set of the target, namely P = {P1,…,P i ,…,P n}.

[0133] In each iteration, after the student network obtains the multi-view feature prototype set of the target, the student network uses the spatiotemporal features M of the labeled data L Initialize the multi-view prototype memory pool P M = {P1,P2,…P S}, then the multi-view feature prototype is lost through the multi-view prototype Guided learning of students on unlabeled data.

[0134] Where S represents the size of the multi-view prototype memory pool.

[0135] In each iteration, after the teacher network obtains the target multi-view feature prototype set, it uses P generated by the teacher network to update the multi-view prototype memory pool P through a first-in-first-out strategy. M (Prototype pool module), that is, using the spatiotemporal features M of unlabeled data U Update the target's feature prototype.

[0136] S33, based on step S32, inputting the pseudo labels predicted by the unlabeled data in the teacher network into the abnormal driven pseudo label filtering module for filtering;

[0137] This embodiment integrates isolation forest and Mahalanobis distance to obtain a dynamic threshold for detecting abnormal pseudo-labels. By analyzing the location and spatial distribution of the target, isolation forest can find abnormal pseudo-labels that are significantly different from the standard detection box. In addition, Mahalanobis distance can calculate the difference between the center of the detection box and the overall distribution, thereby further strengthening the filtering of abnormal pseudo-labels from a statistical perspective.

[0138] The anomaly-driven pseudo-label filtering module integrates isolation forest and Mahalanobis distance to obtain a dynamic threshold for detecting abnormal pseudo-labels, which is used to filter the pseudo-labels predicted by the teacher network on unlabeled data to obtain an unsupervised loss.

[0139] First, in each iteration, the center of all pseudo boxes is calculated and the pseudo label b is set i =(x l ,y l ,x r ,y r )'s center C i yes Then calculate the Mahalanobis distance of each pseudo-frame, the expression is as follows:

[0140]

[0141] Among them, μ represents the average value of all center points, Ω represents the covariance matrix, and D M (C i ) means C i The Mahalanobis distance between all pseudo labels, and N represents the maximum total number of pseudo labels.

[0142] Then, an isolation forest is built on all pseudo labels, which contains several random trees to represent the data. Each node corresponds to a pseudo label. i The path length h i To calculate the anomaly score of each pseudo label, the expression is as follows:

[0143]

[0144] Among them, S a (C i ) means C i The anomaly score, E(·) represents the expected value, c(N) represents the normalized coefficient of the path length, and f h (N-1) represents the harmonic number, and β≈0.5772 represents the Euler constant.

[0145] Finally, combined with D M (C i ) and S a (C i ) to comprehensively detect abnormal pseudo labels and avoid the limitations of a single method. The expression is as follows:

[0146] S i =η1·D M (C i )+η2·S a (C i )#(9)

[0147] Among them, S i represents the final metric of the i-th pseudo label, and η1 and η2 represent two weight coefficients. In this way, any proportion of noise labels can be filtered to ensure the stability and accuracy of semi-supervised learning.

[0148] S34, based on the target-aware motion reconstruction module, providing additional supervision to the teacher network and the student network by reconstructing the motion mask to obtain the reconstruction loss;

[0149] The object-aware motion reconstruction module serves as an auxiliary supervision branch, which uses this type of supervision to help the detection head adapt to the motion pattern of the object and alleviate the problem of losing object information in the feature map. The motion feature set M is output by the bidirectional temporal motion perception module. k ={M1,M2,…,M n}, only convolutional layers and upsampling operations are used to reduce M k The channels of each level of motion features in the prediction mask G = {G1, G2, ..., G n}.

[0150] Then initialize a ground truth motion mask G of the same size as the input image g , all elements are set to 0. Then the target coordinates (x l ,y l ,x r ,y r ) projected onto the mask In the example, the target area in the mask is assigned a value of 1. The expression is as follows:

[0151]

[0152] Among them, j and k represent the mask G g The coordinate of a pixel in . Then the target-aware motion reconstruction loss is The definition expression is as follows:

[0153]

[0154] Among them, n represents the number of feature extractor output layers, G g represents the real mask, G i represents the prediction mask in G.

[0155] S35, defining a relevant loss function, performing target positioning detection, error loss calculation, and iterative optimization of target detection accuracy on the output features of steps S32, S33, and S34, that is, using a gradient descent method to iteratively optimize the target detection network parameters;

[0156] The detection loss of the detection head module includes: supervision loss Unsupervised loss Multi-view prototype learning loss The total detection loss definition expression is as follows:

[0157]

[0158] Among them, λ u and λ p Denotes two hyperparameters of the balanced loss term. The detection head module adopts the task alignment head of task-aligned one-stage object detection (TOOD), which can improve the interaction between classification and localization, and adds a target-aware motion reconstruction loss term to the detection loss of TOOD. and The expression is as follows:

[0159]

[0160] in, represents the loss of labeled data, represents the loss of unlabeled data. λ1 and λ2 represent two weighting coefficients. represents a classification loss, and represents a regression loss.

[0161] Due to the limited labeled data in semi-supervised learning, the student’s learning process on unlabeled data is often susceptible to noisy pseudo-labels, resulting in unreliable feature learning. M The feature prototype in the ,guides students’ feature learning from multiple perspectives, i.e., using the features F obtained by students on unlabeled data s With P M The cosine similarity between all feature prototypes in is expressed as follows:

[0162]

[0163] Among them, F s That is, the fusion feature F obtained by the student network c , f cos (·) indicates the calculation of cosine similarity, “·” indicates inner product, and ∥·∥2 indicates l2 norm.

[0164] Finally, based on the calculated loss, the target detection network uses the back-propagation gradient descent method to iteratively optimize the parameters and save the training network parameters for each round.

[0165] S36: Based on the network parameters saved in step S35, select mAP based on the test data set 50The network with the best accuracy is the optimal object detection network.

[0166] mAP 50 Precision is a very important indicator in object detection, which is used to measure the average accuracy of the model when the IoU (intersection over union) threshold is 0.5.

[0167] This embodiment also takes the public infrared image sequence data sets DAUB, IRDST and ITSDT as examples to further illustrate the method of the present invention, as follows:

[0168] The training set of the DAUB dataset contains 10 videos with 8983 frames, and the test set contains 7 videos with 4795 frames. The IRDST dataset is divided into real datasets and synthetic datasets. The real dataset consists of 85 image sequences and 40656 pictures, of which 42 sequences with 20398 pictures are used as training sets, and the remaining 43 sequences with 20258 pictures are used as test sets; the ITSDT dataset consists of 87 image sequences and 21750 pictures, of which 40 sequences with 10000 pictures are selected as training sets, and 20 sequences with 5000 pictures are selected as test sets to form the new ITSDT-15K dataset.

[0169] (1) Initialization of the experiment;

[0170] Before inputting the infrared images, all images in the training set and the test set are converted to 512×512 pixels, and the sampling time window T1 is set to 5. In this embodiment, the model is trained with a batch size of 4 (3 unlabeled images and 1 labeled image) for 50,000 iterations. This embodiment uses SGD as the optimizer with a learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0001. The teacher network is updated by the EMA attention mechanism with a momentum of 0.9995. For the hyperparameters, in the equation, P M The size of S is 50. u and λ p Both are 2. In formula (13), λ1 and λ2 are set to 2 and 10 respectively. During testing, the IoU threshold of non-maximum suppression is 0.65, and only the predicted boxes with confidence higher than 0.001 are considered. All experiments are performed on two NVIDIA GeForce 4090 GPUs.

[0171] (2) Bidirectional temporal sensor feature extraction stage;

[0172] After the preprocessed image data is input into the model, Resnet50 is used to extract the labeled and unlabeled data features X L and X U , extract the global features M according to the steps of formula (1) and (2) respectivelyL and M U .

[0173] (3) Multi-view prototype learning stage;

[0174] Utilizing the spatiotemporal features of labeled data in the teacher network M L To initialize the multi-view prototype memory pool P M = {P1,P2,…P S}, the multi-view feature prototype can be obtained by the prototype loss Further guide the student's learning on unlabeled data. In each iteration, the teacher network will update the target's feature prototype through MPL (Multi-layer Perceptron).

[0175] (4) Anomaly-driven pseudo-label filtering stage;

[0176] By analyzing the location and spatial distribution of the target, Isolation Forest can find abnormal pseudo-labels that are significantly different from the standard detection box. In addition, the Mahalanobis distance can calculate the difference between the center of the detection box and the overall distribution. Through these two data calculations, the filtering of abnormal pseudo-labels can be further strengthened from a statistical perspective.

[0177] (5) Movement reconstruction stage;

[0178] Use convolutional layers and upsampling operations to reduce M k The prediction mask G = {G1, G2, ..., G n}, reconstruct the target perceived motion reconstruction loss

[0179] (6) Detection head and loss function;

[0180] This embodiment uses the task alignment head of TOOD and adds a target-aware motion reconstruction loss term to the detection loss of TOOD. Finally, the loss is calculated by equation (12), and the back-propagation gradient descent algorithm is used to effectively learn the model to obtain model parameters with high training performance, thereby achieving the purpose of effectively detecting small moving targets in infrared videos.

[0181] In order to verify the effectiveness of the proposed method in infrared moving small target positioning detection, training was performed on the public infrared sequence datasets IRDST and ITSDT-15K respectively. First, the proposed method was compared with some existing infrared small target detection methods (MAXMEAN, TopHat, RLCM, HBMLCM, PSTNN, WSLCM). In addition, 13 deep learning network methods (ACM, RISTD, ISNet, UIUNet, SANet, AGPCNet, RDIAN, DNANet, ResUnet*DTUM, SIRST5K, MSHNet, RPCANet, SSTNet) were also compared, and the results are shown in Table 1.

[0182] Table 1

[0183]

[0184] Table 1 shows the comparison of mAP50, precision, recall, and F1 indicators of each model on the public infrared sequence datasets DAUB, IRDST, and ITSDT-15K, where the best precision is marked in bold. Through the comparison, three obvious findings can be observed.

[0185] 1) The mAP50 index of the method of the present invention is always significantly better than all other current methods. For example, on the DAUB dataset, the mAP50 index of the method of the present invention is 94.26%, while the second highest index obtained by ST-Trans is 92.73%. The mAP50 index of the method of the present invention is 1.53% higher than that of the second highest method.

[0186] 2) On the IRDST and DAUB datasets, the proposed method outperforms most other methods in terms of accuracy. For example, in the ITSDT dataset, the Pr index of the proposed method is the second highest, with a value of 85.36%. On the DAUB dataset, the Pr index of the proposed method is the second highest, with a value of 97.08%, second only to 97.75% of ST-Trans.

[0187] 3) For the F1 index, the method of the present invention is always better than all other currently compared methods. For example, on the ITSDT-15K dataset, the F1 value of the method of the present invention is 88.69%, which is better than the second highest SSTNet's 87.50%. On the IRDST dataset, the F1 value of the method of the present invention is 86.11%, which is better than the second highest SSTNet's 83.91%. This shows that although the precision and recall rate of the method of the present invention are not always better than other methods, the comprehensive detection ability of the present invention is stronger.

[0188] In addition, this embodiment compares the model complexity on ITSDT-15K: the experimental comparison of the method of the present invention (S=50, T1=5) and other methods is shown in Table 2. As can be seen, there are two clear findings. One is that since semi-supervised methods usually use a student-teacher architecture, their parameters are usually higher than those of fully supervised methods. For example, the minimum parameter of the multi-frame fully supervised method is 9.64M, and the minimum parameter of the semi-supervised method is 32.04M. The parameter of the method of the present invention is 52.74M, which is slightly higher than them.

[0189] Table 2

[0190]

[0191] In order to intuitively compare the overall performance of different detection methods, this example creates a PR curve based on precision and recall, and plots the PR curves on DAUB, ITSDT-15K, and IRDST respectively. Figure 5 As shown. Among them, Figure 5 (a) is the PR graph on DAUB. Figure 5 (b) is the PR graph on ITSDT-15K. Figure 5 (c) is the PR graph on IRDST. As can be seen from the figure, the curve of the method of the present invention is mainly located in the upper right corner of the two figures, which shows that the comprehensive performance of the model of the method of the present invention is better than that of other models.

[0192] In addition to the numerical performance comparison, in order to intuitively illustrate the detection accuracy of the model of the method of the present invention compared with the current most advanced model method, this embodiment visualizes three groups of small target detection results of four methods on the DAUB dataset, IRDST dataset and ITSDT-15K dataset, as shown in Figure 2. Figure 6 As shown. Among them, Figure 6 (a) is the detection visualization result on the DAUB dataset. Figure 6 (b) is the detection visualization result on the ITSDT-15K dataset. Figure 6(c) is the detection visualization result on the IRDST dataset. Through visual comparison, two obvious findings can be observed. First, in the detection results of the IRDST dataset, various methods almost have no false detection or missed detection. This may be due to the fact that the background of the randomly selected samples is relatively simple and there is no occlusion. However, even so, the method of the present invention can still generate a detection frame that is highly similar to the actual target shape and tightly surrounds the target, demonstrating its accuracy. On the other hand, in the detection results of the ITSDT-15K dataset, ACM and ConsistentTeacher failed to detect small targets and occlusions, while the method of the present invention can generate correct and tight detection bounding boxes, showing excellent performance in detecting small infrared moving targets.

[0193] In summary, the method of the present invention shows higher accuracy and robustness in infrared moving small target detection, and can effectively detect small targets appearing in infrared moving small target sequence images collected by infrared imaging equipment. This makes the method have broad application prospects in important application fields such as drone supervision and security.

[0194] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for infrared small target detection based on semi-supervised multi-view prototype learning, the specific steps are as follows: S1. Construct an infrared moving small target detection network based on teacher-student network; The target detection network is a teacher-student network, including: Teacher network, student network, anomaly-driven pseudo-label filtering module, target-aware motion reconstruction module; The teacher network and the student network both include: a feature extraction module and a multi-view prototype learning module; the feature extraction module includes: a basic feature extraction module and a bidirectional temporal motion sensor module; the multi-view prototype learning module includes: a prototype learning module, a detection head module, and a prototype pool module; S2, collect infrared images, build an infrared image sequence data set, and divide it into a training set and a test set according to actual conditions; S3, input the training set into the target detection network constructed in step S1 for training, the teacher network is updated from the student network through the EMA attention mechanism, and then the parameters of the target detection network are iteratively optimized based on the gradient descent method, and the optimal target detection network parameters are selected based on the test set to obtain the optimal target detection network; S4. Based on the optimal target detection network obtained in step S3, the sequence image in the sampling window to be detected is input to obtain the final detection result.

2. According to claim 1, a method for infrared small target detection based on semi-supervised multi-view prototype learning is characterized in that: The step S3 is specifically as follows: S31, the training data set is input into the basic feature extraction module, Resnet50 is used for basic feature extraction, and then input into the bidirectional temporal motion sensor module for feature optimization; First, we randomly extract two T-frame image sequences from the infrared video of the training set as input, namely, I U ={U1,U2,…,U T }, I L ={L1,L2,…,L T }; Among them, L U represents the unlabeled dataset, I L represents a labeled dataset, and U T and L T It represents the key frames that need to be detected, and the rest are reference frames that provide time information; Then, the pre-trained ResNet50 is used as the basic feature extractor, and each frame of the image sequence is input into the extractor to obtain multi-frame visual features. and in, represents the real number domain, n represents the number of feature extractor output layers, T represents the time window size of frame sampling; C, H and W represent the channel, height and width of the feature tensor respectively; For the student network, the bidirectional temporal motion perception module receives multi-frame visual features X U and X L As its input, the spatiotemporal features M are generated respectively U and M L ; The teacher network only accepts X U Generate M U ; The details are as follows: As a feature optimization module, the bidirectional temporal motion sensor module builds a coupled bidirectional structure through ConvGRU units, simulating the use of spatiotemporal features in the past, present, and future frames of global time encoding to fully perceive the motion trajectory of the target; First, a coupled bidirectional structure is constructed using the ConvGRU unit; the ConvGRU cell includes a reset gate and an update gate In the labeled data, multi-frame visual features X L ={X1,…,X t ,…X T } represents the input of ConvGRU; H L ={H1,…,H t ,…H T } represents its hidden state; then the overall update process of the ConvGRU cell over time is expressed as follows: Among them, σ(·) and α(·) represent the sigmoid activation function and tanh activation function respectively; f s (·) and f c (·) indicates a concatenation operation and a splitting operation respectively; "*" indicates a convolution operation, and ° indicates an element-wise product; and b represent the weight matrix and bias term in the corresponding convolutional layer; The bidirectional temporal motion sensor module stacks two ConvGRU units front and back to perform global motion modeling by accumulating dual temporal motion dimensions. The calculation process expression is as follows: in, and They represent the forward and backward hidden states of the ConvGRU unit calculated by formula (1); X t represents the visual features of the tth frame; and H t Represents the final output of the current unit and the input of the next unit; Then we get the forward motion features of the labeled data Then get the reverse kinematic characteristics Similarly, the forward motion features of unlabeled data can be obtained and inverse kinematics features Finally, these two types of motion patterns are aggregated through a convolutional layer with a 3×3 kernel and a stride of 1 to generate spatiotemporal features, which are expressed as follows: Among them, M U and M L Represent the multi-frame spatiotemporal features of unlabeled images and labeled images respectively; S32, performing prototype learning on the feature information optimized by the bidirectional temporal motion sensor module in step S31 based on the multi-view prototype learning module; The multi-view prototype learning module generates a multi-view feature prototype of the target into the prototype pool module through a set of prototype generators PG between high-level semantics and low-level details in the prototype learning module and the task alignment head of the task-aligned single-stage target detection TOOD in the detection head module, as follows: The multi-scale keyframe features M are obtained through the bidirectional temporal motion perception module k ={M1,M2,…,M i ,…,M n } as the prototype generator input, M k It is M L or M U The last feature set of , n represents the number of feature extractor output layers; For each prototype generator in the multi-view prototype learning module, the feature prototype of the previous layer and the key frame features corresponding to the current prototype generator are input into it to obtain the output of the next layer; First, the feature prototype P i+1 Upsampling is performed to maintain the same i The same resolution, and concatenate them to obtain the fused feature F c ; Then, 3×3 depth convolution DW(·) and 1×1 convolution PW(·) are introduced to combine local and global target information to obtain an accurate offset map O m , the calculation process expression is as follows: Among them, f up (·) indicates bilinear interpolation operation; Offset map in P i+1 The upper guidance grid sampling obtains P′ i+1 , and add P′ i+1 The feature prototype and fusion feature F after sampling c , we get the target features of different views, and the calculation expressions are as follows: Among them, Grid(·) represents the grid sampling operation, F p Representing target features in different views; Using non-local attention, the similarity between all positions in the feature prototype is calculated, and the expression is as follows: Among them, Q, K, V represent the query vector Q, key vector K and value vector V in the Transformer's self-attention mechanism, T represents the transposition operation, and f p (·) represents the projection operation implemented by 1×1 convolution, γ represents the weight coefficient, d represents the dimension, and P i represents the new feature prototype of level i; Similarly, M is processed in the same way k Each level feature in; Finally, the multi-view feature prototype set of the target is obtained, that is, P = {P1, ..., P i ,…,P n }; In each iteration, after the student network obtains the multi-view feature prototype set of the target, the student network uses the spatiotemporal features M of the labeled data L Initialize the multi-view prototype memory pool P M = {P1,P2,…P s }, then the multi-view feature prototype is lost through the multi-view prototype Guide students’ learning on unlabeled data; Where S represents the size of the multi-view prototype memory pool; In each iteration, after the teacher network obtains the target multi-view feature prototype set, it uses P generated by the teacher network to update the multi-view prototype memory pool P through a first-in-first-out strategy. M , that is, using the spatiotemporal features M of the unlabeled data U Update the target’s feature prototype; S33, based on step S32, inputting the pseudo labels predicted by the unlabeled data in the teacher network into the abnormal driven pseudo label filtering module for filtering; The anomaly-driven pseudo-label filtering module integrates isolation forest and Mahalanobis distance to obtain a dynamic threshold for detecting abnormal pseudo-labels, which is used to filter the pseudo-labels predicted by the teacher network on unlabeled data to obtain an unsupervised loss. First, in each iteration, the center of all pseudo boxes is calculated and the pseudo label b is set i =(x l ,y l ,x r ,y r )'s center C i yes Then calculate the Mahalanobis distance of each pseudo-frame, the expression is as follows: Among them, μ represents the average value of all center points, Ω represents the covariance matrix, and D M (C i ) means C i The Mahalanobis distance between all pseudo labels, N represents the maximum total number of pseudo labels; Then, an isolation forest is built on all pseudo labels, which contains several random trees to represent the data. Each node corresponds to a pseudo label. i The path length h i To calculate the anomaly score of each pseudo label, the expression is as follows: Among them, S a (C i ) means C i The anomaly score, E(·) represents the expected value, c(N) represents the normalized coefficient of the path length, and f h (N-1) represents the harmonic number, β≈0.5772 represents the Euler constant; Finally, combined with D M (C i ) and S a (C i ) to comprehensively detect abnormal pseudo labels, the expression is as follows: S i =η1·D M (C i )+η2·S a (C i )#(9) Among them, S i represents the final metric of the i-th pseudo label, η1 and η2 represent two weight coefficients; S34, based on the target-aware motion reconstruction module, providing additional supervision to the teacher network and the student network by reconstructing the motion mask to obtain the reconstruction loss; The bidirectional temporal motion sensor module outputs the motion feature set M k ={M1,M2,…,M n }, only convolutional layers and upsampling operations are used to reduce M k The channels of each level of motion features in the prediction mask G = {G1, G2, ..., G n }; Then initialize a ground truth motion mask G of the same size as the input image g , set all elements to 0; then set the target coordinates (x l ,y l ,x r ,y r ) projected onto the mask In the example, the target area in the mask is assigned a value of 1. The expression is as follows: Among them, j and k represent the mask G g The coordinate of a pixel in the image; then the target-aware motion reconstruction loss The definition expression is as follows: Where n represents the number of feature extractor output layers, G g represents the real mask, G i represents the prediction mask in G; S35, defining a relevant loss function, performing target positioning detection, error loss calculation, and iterative optimization of target detection accuracy on the output features of steps S32, S33, and S34, that is, using a gradient descent method to iteratively optimize the target detection network parameters; The detection loss of the detection head module includes: supervision loss Unsupervised loss Multi-view prototype learning loss The total detection loss definition expression is as follows: Among them, λ u and λ p Two hyperparameters representing the balanced loss term; The detection head module adopts the task alignment head of task-aligned single-stage object detection TOOD, and adds a target-aware motion reconstruction loss term to the detection loss of TOOD; and The expression is as follows: in, represents the loss of labeled data, represents the loss of unlabeled data; λ1 and λ2 represent two weighting coefficients; represents a classification loss, and represents a regression loss; Student Network Utilization Prototype Memory Pool P M The feature prototype in the ,guides students’ feature learning from multiple perspectives, i.e., using the features F obtained by students on unlabeled data s With P M The cosine similarity between all feature prototypes in is expressed as follows: Among them, F s That is, the fusion feature F obtained by the student network c , f cos (·) indicates the calculation of cosine similarity, "·" indicates inner product, and ||·||2 indicates l2 norm; Finally, according to the calculated loss, the target detection network uses the back-propagation gradient descent method to iteratively optimize the parameters and save the training network parameters for each round; S36: Based on the network parameters saved in step S35, select mAP based on the test data set 50 The network with the best accuracy is the optimal object detection network.

Images