Intensive motion infrared small target detection method based on three-domain feature learning

Abstract

The invention relates to the technical field of target detection, and discloses a dense motion infrared small target detection method based on three-domain feature learning, and the method comprises the steps: obtaining a to-be-detected infrared video; a to-be-detected infrared video is input into a pre-trained intensive motion infrared small target detection model to obtain a detection result, and the intensive motion infrared small target detection model comprises a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a time feature extraction module, a fusion enhancement network and a detection head. The input end of the backbone network is used for inputting an infrared video to be detected, and the detection head is used for outputting a detection result. According to the detection method, a dense-motion infrared small target model is adopted for dense-motion infrared small target detection, multi-domain feature complementary enhancement is adopted in the model, time, space and frequency domain features are fused mutually, the problem that a single-domain feature is insufficient in representation of the dense small target is solved, and the detection accuracy of the dense-motion infrared small target is improved. And the detection performance of the dense moving small target in the complex scene is obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method for detecting densely moving infrared small targets based on three-domain feature learning. Background Art

[0002] Infrared small target detection (ISTD) has a wide range of important applications in areas such as traffic management, maritime rescue, and military guidance. Compared to general targets, small targets in infrared images or videos typically have low contrast with the background and may even lack distinct color, shape, or texture features. Given these two characteristics, effectively learning and accurately detecting small targets in complex infrared application scenarios is often a challenging problem. Accurately locating and tracking small moving targets in infrared images and videos has long been a hot topic and a challenge in vision research. Summary of the Invention

[0003] The present invention aims to solve, at least to some extent, one of the technical problems in the related art. To this end, one purpose of the present invention is to propose a method for detecting densely moving infrared small targets based on three-domain feature learning, which solves the problem that single-domain features are insufficient for representing densely moving small targets and significantly improves the detection performance of densely moving small targets in complex scenes.

[0004] A second object of the present invention is to provide an electronic device.

[0005] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a method for detecting small infrared targets in dense motion based on three-domain feature learning, the method comprising: obtaining an infrared video to be tested, wherein the infrared video to be tested comprises N consecutive infrared video frames, where N is a positive integer; inputting the infrared video to be tested into a pre-trained model for detecting small infrared targets in dense motion to obtain a detection result, wherein the detection result comprises a detection bounding box of the infrared small target, a Gaussian distribution of the detection bounding box, a detection classification, and a detection probability; the model for detecting small infrared targets in dense motion comprises a backbone network, a spatial feature extraction model, and a spatial feature extraction module. Block, frequency domain feature extraction module, time feature extraction module, fusion enhancement network and detection head, the input end of the backbone network is used to input the infrared video to be tested, the output end of the backbone network is respectively connected to the input end of the spatial feature extraction module, the input end of the frequency domain feature extraction module and the input end of the time feature extraction module, the output ends of the spatial feature extraction module, the frequency domain feature extraction module and the time feature extraction module are all connected to the input end of the fusion enhancement network, the output end of the fusion enhancement network is connected to the input end of the detection head, and the detection head is used to output the detection result.

[0006] According to the densely moving infrared small target detection method based on three-domain feature learning in an embodiment of the present invention, a model of densely moving infrared small targets is used for target detection. The model uses multi-domain feature complementary enhancement and the fusion of time, space, and frequency domain features, which solves the problem of insufficient representation of dense small targets by single domain features and significantly improves the detection performance of dense small targets in complex scenarios.

[0007] In addition, the densely moving infrared small target detection method based on three-domain feature learning proposed in the above embodiment of the present invention may also have the following additional technical features: According to one embodiment of the present invention, the training process of the dense motion infrared small target detection model includes: obtaining a training data set, wherein the training data set includes a training infrared video and a labeling result corresponding to each infrared video frame in the training infrared video, the training infrared video includes N consecutive infrared video frames, and the labeling result includes a true bounding box of the infrared small target, a true bounding box Gaussian distribution, a true classification and a true probability; inputting the training infrared video in the training data set into the dense motion infrared small target detection model to obtain a prediction result, wherein the prediction result includes a predicted bounding box of the infrared small target, a predicted bounding box Gaussian distribution, a predicted classification and a predicted probability; according to the predicted bounding box, the predicted bounding box Gaussian distribution, the predicted classification and the predicted probability A boundary regression loss function is constructed based on the predicted bounding box Gaussian distribution, the true bounding box, and the true bounding box Gaussian distribution. A classification loss function is constructed based on the predicted classification and the true classification. A target probability loss function is constructed based on the true probability and the predicted probability. A forward optical flow loss function and a backward optical flow loss function are constructed based on the training infrared video of the input detection model. A detection model loss function is constructed based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function, and the backward optical flow loss function. The dense motion infrared small target detection model is trained based on the detection model loss function to obtain a trained dense motion infrared small target detection model.

[0008] According to one embodiment of the present invention, the fusion enhancement network includes a first fusion enhancement module, a second fusion enhancement module and a third fusion enhancement module, the output ends of the spatial feature extraction module and the frequency domain feature extraction module are connected to the input end of the first fusion enhancement module, the output ends of the frequency domain feature extraction module and the time feature extraction module are connected to the input end of the second fusion enhancement module, the output ends of the first fusion enhancement module and the second fusion enhancement module are connected to the input end of the third fusion enhancement module, and the output end of the third fusion enhancement module is connected to the input end of the detection head, and the training infrared video in the training data set is input into the dense motion infrared small target detection model, including: using the backbone network to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map group, wherein the infrared feature map group includes N infrared feature maps; using The spatial feature extraction module performs spatial feature extraction on the infrared feature map group to obtain a spatial feature map; the frequency domain feature extraction module is used to perform frequency feature extraction on each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps; the time feature extraction module is used to perform time feature extraction on the infrared feature map group to obtain a time feature map; the first fusion enhancement module is used to generate a space-frequency fusion feature map based on the spatial feature map and the frequency feature map group, the second fusion enhancement module is used to generate a time-frequency fusion feature map based on the frequency feature map group and the time feature map, and the third fusion enhancement module is used to generate a space-time-frequency fusion feature map based on the space-frequency feature map and the time-frequency feature map; the detection head is used to generate the prediction result based on the space-time-frequency fusion feature map.

[0009] According to one embodiment of the present invention, the backbone network includes a first branch, a second branch and a residual connection layer, and the first branch and the second branch are both connected to the residual connection layer, wherein the first branch includes a first convolution unit, a second convolution layer, an expansion convolution unit, a spatial attention layer and a channel attention layer connected in sequence, the first convolution unit includes 8 parallel first convolution layers, and the directions of convolution of the corresponding convolution kernels of the 8 first convolution layers are horizontally right, horizontally left, vertically upward, vertically downward, upper right, upper left, lower right and lower left respectively, the expansion convolution unit includes 4 parallel expansion convolution layers, and the second branch includes a third convolution layer and a fourth convolution layer connected in sequence, and the backbone network is used to extract features from each infrared video frame in the training infrared video, including: using the 8 parallel first convolution layers in the first convolution unit to calculate the grayscale difference between the center pixel and the boundary pixel of the infrared video frame in the corresponding direction to obtain 8 first feature maps; multiplying the two feature maps with opposite convolution directions in the 8 first feature maps element by element to obtain 4 difference maps Different feature maps; adding the four difference feature maps element-by-element in a vertical reverse direction in pairs to obtain two second feature maps; using the second convolution layer to perform feature extraction on the first spliced ​​feature map to obtain a primary feature map, wherein the first spliced ​​feature map is obtained by splicing the two second feature maps; performing residual connection processing on the primary feature map and the third feature map to obtain a fourth feature map, wherein the third feature map is extracted from the infrared video frame by the third convolution layer; respectively using the four parallel dilated convolution layers in the dilated convolution unit to perform dilated convolution processing on the fourth feature map to obtain four dilated feature maps; splicing the four dilated feature maps to obtain a secondary feature map; using the spatial attention layer to perform feature extraction on the secondary feature map to obtain a fifth feature map; using the channel attention layer to perform feature extraction on the fifth feature map to obtain a sixth feature map; using the residual connection layer to perform residual connection processing on the sixth feature map and the seventh feature map to obtain an infrared feature map, wherein the seventh feature map is extracted from the third feature map by the fourth convolution layer.

[0010] According to one embodiment of the present invention, the spatial feature extraction module includes a third branch, a fourth branch and a memory enhancement submodule, and the third branch and the fourth branch are both connected to the memory enhancement submodule, wherein the third branch includes a fifth convolutional layer, a sixth convolutional layer, a spatial enhancement submodule, and a first embedding layer connected in sequence, and the fourth branch includes a second embedding layer. The spatial feature extraction module is used to extract spatial features from the infrared feature map group, including: splicing the first N-1 infrared feature maps in the infrared feature map group to obtain a second spliced ​​feature map; using the fifth convolutional layer to perform feature extraction on the second spliced ​​feature map to obtain an eighth feature map; element-wise multiplying the eighth feature map with the Nth video feature map in the infrared feature map group and then splicing it with the Nth infrared feature map in the infrared feature map group to obtain a third spliced ​​feature map; using the sixth convolutional layer to perform feature extraction on the third spliced ​​feature map to obtain an initial feature map; using the spatial enhancement submodule to The initial feature map is spatially enhanced to obtain a spatially enhanced feature map, wherein the spatial enhancement submodule includes four sub-branches and a seventh convolutional layer, and the four sub-branches perform feature processing on the initial feature map respectively. After the processing results of sub-branch three and sub-branch four are residually connected, the seventh convolutional layer is used to extract features from the residual connection results, and the extraction result is element-by-element multiplied by the addition result to obtain a spatially enhanced feature map, and the addition result is the element-by-element addition result of the processing results of sub-branch one and sub-branch two, the sub-branch one includes a first maximum pooling layer and a first fully connected layer connected in sequence, the sub-branch two includes a first average pooling layer and a second fully connected layer connected in sequence, the sub-branch three includes a channel maximum pooling layer, and the sub-branch four includes a channel average pooling layer; the spatially enhanced feature map and the initial feature map are element-by-element added to obtain local-global spatial features; the first embedding layer is used to generate a first key-value pair according to the local-global spatial features, wherein the first key-value pair includes a key Sum ; Using the memory enhancement submodule, a spatial feature map is obtained according to the first key-value pair and the second key-value pair, wherein the second key-value pair is generated by the second embedding layer according to the Nth video feature map in the feature map group, and the second key-value pair includes a key Sum , wherein the memory enhancement submodule uses the softmax activation function to and key The product of is normalized, and the similarity obtained by normalization is compared with the value After multiplication, the value Perform residual connection processing, and the residual connection result is processed by the matching layer to obtain the spatial feature map.

[0011] According to one embodiment of the present invention, the frequency domain feature extraction module is used to extract the frequency features of each infrared feature image in the infrared feature image group, including: using Fourier transform to convert the features of the infrared feature image into the frequency domain to obtain the frequency domain information of the infrared feature image, the frequency domain information including amplitude and phase ; Using the first convolution group, according to the amplitude and magnitude mask The product of , obtains the amplitude feature, where the amplitude mask The amplitude is calculated by the eighth convolutional layer and the ninth convolutional layer connected in sequence. and the phase The splicing result is extracted; using the second convolution group, according to the phase and phase mask The product of the phase mask is obtained, where The amplitude is calculated by the eighth convolutional layer and the tenth convolutional layer connected in sequence. and the phase The splicing result is extracted; the inverse Fourier transform is used to transform the infrared feature map in the frequency domain to the original definition domain according to the amplitude feature and the phase feature to obtain the frequency feature map; wherein, the convolution group in the first convolution group and the second convolution group includes an eleventh convolution layer, a second maximum pooling layer, a second average pooling layer, a first depth-separable convolution layer and a first point-by-point convolution layer, and the eleventh convolution layer is used to extract features of the input product, and the second maximum pooling layer and the second average pooling layer are used to perform pooling processing on the feature extraction results respectively, and after the maximum pooling result and the average pooling result are residually connected, the first depth-separable convolution layer and the first point-by-point convolution layer are used in turn to process the residual connection results to obtain the output feature, and the input product includes the amplitude and magnitude mask The product and phase and phase mask The output feature includes an amplitude feature and a phase feature.

[0012] According to one embodiment of the present invention, the temporal feature extraction module includes a coarse motion estimation branch and a feature enhancement branch connected in sequence, the coarse motion estimation branch includes a feature splicing layer, a second convolution unit and a twelfth convolution layer connected in sequence, the feature enhancement branch includes a sixteenth convolution layer, a seventeenth convolution layer, an eighteenth convolution layer, a softmax activation function, a nineteenth convolution layer, a twentieth convolution layer, a flattening layer and a deformable attention submodule, a twenty-first convolution layer and a 3D convolution layer, and the use of the temporal feature extraction module to extract temporal features from the infrared feature map group includes: using the feature splicing layer to perform forward and backward splicing on two adjacent infrared feature maps in the infrared feature map group, respectively, A first set of inter-frame relationship graphs and a second set of inter-frame relationship graphs are obtained, wherein the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs each include four inter-frame relationship graphs; the second convolution unit is used to capture the inter-frame relationship of the same source points of the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs at different motion speeds, to obtain a first inter-frame relationship and a second inter-frame relationship, wherein the second convolution unit includes a thirteenth convolution layer, a first ReLU activation function, a fourteenth convolution layer, a second ReLU activation function, a fifteenth convolution layer and a third ReLU activation function connected in sequence; the twelfth convolution layer is used to perform feature extraction on the first inter-frame relationship and the second inter-frame relationship, to obtain four forward optical flows. and 4 backward optical flows ; The forward optical flow Towards After distortion, the forward optical flow Add together to get the forward optical flow across two frames ; The backward optical flow Towards After distortion, the backward optical flow Add up to get the backward optical flow across two frames ; Each infrared feature map in the infrared feature map group and the forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Align to the intermediate frame to obtain alignment features; combine the infrared feature map group, the alignment features, and the forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Perform connection processing to obtain connection features; use the sixteenth convolution layer to extract features from the connection features to obtain a ninth feature map; use the deformable attention submodule to process the sampling point offset, the standardized attention weight and the amortized feature to obtain a refined feature, wherein the sampling point offset is obtained according to the tenth feature map and the offset, wherein the tenth feature map is extracted from the ninth feature map by the seventeenth convolution layer, and the offset is obtained according to the forward optical flow , forward optical flow across two frames , backward optical flow and the backward optical flow across two frames It is calculated that the standardized attention weight is obtained by processing the ninth feature map through the sequentially connected tenth convolutional layer and the softmax activation function, and the flattened feature is obtained by processing the infrared feature map group through the sequentially connected nineteenth convolutional layer, the twentieth convolutional layer and the flattened layer; the twenty-first convolutional layer is used to extract the refined features to obtain the eleventh feature map; the 3D convolutional layer is used to process the features after residual connection of the infrared feature map group and the eleventh feature map to obtain N twelfth feature maps; residual connection is performed on N twelfth feature maps to obtain the time feature map.

[0013] According to one embodiment of the present invention, the first fusion enhancement module, the second fusion enhancement module and the third fusion enhancement module all include a fusion enhancement module, the fusion enhancement module includes a second depth-separable convolution layer, a second point-by-point convolution layer, a collaborative attention submodule, a gating submodule and a batch normalization layer connected in sequence, the fusion enhancement module is used to receive the first input feature map and the second input feature map, and output a fusion feature map, the fusion enhancement module is specifically used to: use the second depth-separable convolution layer and the second point-by-point convolution layer connected in sequence to extract the fourth splicing feature to obtain a thirteenth feature map, wherein the fourth splicing feature is obtained by the first The input feature map and the second input feature map are spliced ​​along the channel dimension; the internal features of the thirteenth feature map are fused using the collaborative attention submodule to obtain a fourteenth feature map; the thirteenth feature map and the fourteenth feature map are subjected to residual connection processing to obtain a fifteenth feature map; the gating submodule is used to generate the weight value gate of the fifteenth feature map; the first product and the second product are added to obtain sixteen features, wherein the first product is the product of the fifteenth feature map and the weight value gate, and the second product is the product of (1-gate) and the fourth spliced ​​feature; the sixteen features are processed using the batch normalization layer to obtain a fused feature map.

[0014] According to one embodiment of the present invention, the detection model loss function for: ; in, represents the detection model loss function, represents the boundary regression loss function, represents the classification loss function, represents the target probability loss function, represents the forward optical flow loss function, represents the backward optical flow loss function, represents the boundary regression loss hyperparameter, represents the classification loss hyperparameter, represents the target probability loss hyperparameter, represents the optical flow loss hyperparameter; The boundary regression loss function for: ; in, represents the boundary regression loss function, represents the intersection-over-union hyperparameter, represents the intersection-over-union loss function, represents the normalized Gaussian distance hyperparameter, represents the normalized Gaussian distance loss function, represents the predicted bounding box, represents the ground-truth bounding box, represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, represents the predicted bounding box Gaussian distribution, represents the Gaussian distribution of the true bounding box, represents a constant; The classification loss function for: ; in, represents the classification loss function, i represents the number of training infrared videos in the training dataset, c is the number of categories, For real classification, To predict classification; The target probability loss function for: ; in, represents the target probability loss function, i represents the number of training infrared videos in the training dataset, is the true probability, is the predicted probability; The forward optical flow loss function for: ; in, represents the forward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the forward true image, represents the forward estimated image; The backward optical flow loss function for: ; in, represents the backward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the backward true image, Represents the backward estimated image.

[0015] To achieve the above-mentioned objectives, the second aspect of the present invention proposes an electronic device, including a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the dense motion infrared small target detection method based on three-domain feature learning as described above is implemented.

[0016] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a method for detecting small infrared targets in dense motion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a densely moving infrared small target detection model according to an embodiment of the present invention; Figure 3 This is a training flow chart of a densely moving infrared small target detection model according to an embodiment of the present invention; Figure 4 This is a flow chart of processing a training infrared video by a dense motion small infrared target detection model according to an embodiment of the present invention; Figure 5 1 is a schematic diagram of a processing flow of a backbone network according to an embodiment of the present invention; Figure 6 Schematic diagram of each convolution kernel in the first convolution unit according to an embodiment of the present invention; Figure 7 1 is a schematic diagram of a processing flow of a spatial feature extraction module according to an embodiment of the present invention; Figure 8 1 is a schematic diagram of a processing flow of a frequency domain feature extraction module according to an embodiment of the present invention; Figure 9 1 is a schematic diagram of a processing flow of a coarse motion estimation branch according to an embodiment of the present invention; Figure 10 1 is a schematic diagram of a process flow for generating forward and backward optical flow across two frames according to an embodiment of the present invention; Figure 11 1 is a schematic diagram of a processing flow of a feature enhancement branch according to an embodiment of the present invention; Figure 12 is a schematic diagram of the processing flow of the fusion enhancement module according to one embodiment of the present invention; Figure 13 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0019] Artificial intelligence technology has developed rapidly in recent years, and data-driven methods based on deep learning have become the mainstream approach. However, most research focuses on single frames. Compared to single-frame image methods that rely on the differences between local targets and the background, small target detection in multi-frame videos can extract more information from the spatiotemporal domain, offering broader application prospects. Furthermore, most current infrared small target detection (ISTD) methods focus on sparse (i.e., only one or a few) target detection. In reality, densely packed targets are common in real-world infrared detection scenarios, and accurately identifying multiple small targets remains a challenge.

[0020] To solve the above problems, an embodiment of the present invention provides a method and electronic device for detecting small infrared targets in dense motion based on three-domain feature learning. The following will describe in detail the method and electronic device for detecting small infrared targets in dense motion based on three-domain feature learning in accordance with the embodiment of the present invention in combination with the drawings in the specification and the specific implementation methods.

[0021] Figure 1 FIG. 1 is a flow chart of a method for detecting small infrared targets in dense motion according to an embodiment of the present invention. Figure 1 As shown, a method for detecting small infrared targets in dense motion based on three-domain feature learning may include: S101, obtaining an infrared video to be tested, wherein the infrared video to be tested includes N consecutive infrared video frames, where N is a positive integer; S102: Input the infrared video to be tested into a pre-trained dense motion infrared small target detection model to obtain a detection result, wherein the detection result includes a detection bounding box of the infrared small target, a Gaussian distribution of the detection bounding box, a detection classification, and a detection probability.

[0022] Specifically, continuous frame videos are collected of small infrared targets with dense motion to be detected. It should be noted that the number of infrared video frames in the collected continuous frame videos may be greater than N. When the number of infrared video frames in the collected continuous frame videos is greater than N, the collected continuous frame videos are processed to generate multiple videos including N continuous infrared video frames. The generated videos are recorded as the infrared videos to be detected. The infrared video frames in the infrared video to be detected contain dense small infrared targets.

[0023] The infrared video to be tested is input into a pre-trained dense motion infrared small target detection model. The pre-trained dense motion infrared small target detection model detects the infrared small targets in the infrared video to be tested and outputs the detection results.

[0024] It should be noted that densely packed moving infrared small targets are a key scenario for infrared small target detection. To address the insufficient feature representation capabilities and low accuracy of traditional detection algorithms, the present invention designed a three-domain deep learning network. The model trained by this network (the densely packed moving infrared small target detection model) can accurately detect densely packed moving infrared small targets, improving detection accuracy, robustness, and scenario adaptability.

[0025] Specifically, if Figure 2 As shown, the densely moving infrared small target detection model in the embodiment of the present invention may include a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network and a detection head. The input end of the backbone network is used to input infrared video, and the output end of the backbone network is respectively connected to the input end of the spatial feature extraction module, the input end of the frequency domain feature extraction module and the input end of the temporal feature extraction module. The output ends of the spatial feature extraction module, the frequency domain feature extraction module and the temporal feature extraction module are all connected to the input end of the fusion enhancement network, and the output end of the fusion enhancement network is connected to the input end of the detection head. The detection head is used to output the detection result.

[0026] Certain targets in complex backgrounds often do not show a high degree of correlation in consecutive frames, while information from different fields is complementary. Introducing more information domains helps to more comprehensively characterize the features of small infrared targets. In order to extract more features to improve the accuracy of recognition, the dense motion infrared small target detection model in the embodiment of the present invention adopts three-domain feature learning to fuse the features extracted from the time, space, and frequency domains respectively. Specifically, the dense motion infrared small target detection model extracts the features of the target in the three domains through the three branches of time, space, and frequency domains, and then fuses the features to improve the detection model's accuracy for dense motion infrared small target detection. It has broad application prospects in scenarios such as night surveillance, search and rescue, and military applications.

[0027] In one embodiment of the present invention, Figure 3 As shown, the training process of the dense motion infrared small target detection model may include: S201, obtaining a training data set, wherein the training data set includes a training infrared video and a labeling result corresponding to each infrared video frame in the training infrared video, the training infrared video includes N consecutive infrared video frames, and the labeling result includes a true bounding box of a small infrared target, a true bounding box Gaussian distribution, a true classification, and a true probability; S202: Inputting the training infrared video in the training data set into the dense motion infrared small target detection model to obtain a prediction result, wherein the prediction result includes a predicted bounding box of the infrared small target, a predicted bounding box Gaussian distribution, a predicted classification, and a predicted probability; S203, constructing a boundary regression loss function based on the predicted bounding box, the predicted bounding box Gaussian distribution, the true bounding box, and the true bounding box Gaussian distribution, constructing a classification loss function based on the predicted classification and the true classification, constructing a target probability loss function based on the true probability and the predicted probability, constructing a forward optical flow loss function and a backward optical flow loss function based on the training infrared video of the input detection model, and constructing a detection model loss function based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function, and the backward optical flow loss function; S204: Train the densely moving infrared small target detection model according to the detection model loss function to obtain a trained densely moving infrared small target detection model.

[0028] Specifically, multiple long videos of small infrared targets in dense motion are collected and processed to generate a large video set consisting of N consecutive infrared video frames, which are recorded as the training infrared video set. Each infrared target in the training infrared video set is labeled. The labeled content includes at least the true bounding box of the infrared target, the true bounding box Gaussian distribution, the true classification, and the true probability.

[0029] As a specific embodiment, when N is 5, for a continuous frame video set, continuous infrared video frames from the time window T=5 are taken as training infrared videos, and the training infrared videos are used as input for training a dense motion infrared small target detection model.

[0030] After training the infrared video input dense motion infrared small target detection model, the dense motion infrared small target detection model extracts features from each infrared video frame in the training infrared video, fuses and detects the extracted features, and outputs the predicted bounding box, predicted bounding box Gaussian distribution, predicted classification, and predicted probability of the infrared small target.

[0031] A boundary regression loss function is constructed based on the predicted bounding box, its Gaussian distribution, the true bounding box, and its Gaussian distribution. A classification loss function is constructed based on the predicted classification and the true classification. A target probability loss function is constructed based on the true probability and the predicted probability. Forward and backward optical flow loss functions are constructed based on the training infrared video input to the detection model. A detection model loss function is constructed based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward and backward optical flow loss functions. A densely moving infrared small target detection model is trained based on the detection model loss function to obtain a trained densely moving infrared small target detection model.

[0032] In one embodiment of the present invention, Figure 4 As shown, the fusion enhancement network may include a first fusion enhancement module, a second fusion enhancement module and a third fusion enhancement module. The output ends of the spatial feature extraction module and the frequency domain feature extraction module are both connected to the input end of the first fusion enhancement module, the output ends of the frequency domain feature extraction module and the time feature extraction module are both connected to the input end of the second fusion enhancement module, the output ends of the first fusion enhancement module and the second fusion enhancement module are both connected to the input end of the third fusion enhancement module, and the output end of the third fusion enhancement module is connected to the input end of the detection head. Inputting the training infrared video in the training data set into the dense motion infrared small target detection model may include: S301, using the backbone network to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map group, wherein the infrared feature map group includes N infrared feature maps ; S302, using the spatial feature extraction module to extract spatial features from the infrared feature map group to obtain a spatial feature map ; S303, using the frequency domain feature extraction module to extract the frequency features of each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps ; S304, using the time feature extraction module to extract the time feature of the infrared feature image group to obtain the time feature image ; S305, using the first fusion enhancement module to generate a space-frequency fusion feature map based on the spatial feature map and the frequency feature map group, using the second fusion enhancement module to generate a time-frequency fusion feature map based on the frequency feature map group and the time feature map, and using the third fusion enhancement module to generate a space-time-frequency fusion feature map based on the space-frequency feature map and the time-frequency feature map; S306: Using the detection head, a prediction result is generated based on the space-time-frequency fusion feature map.

[0033] Specifically, the backbone network in this embodiment of the present invention is a difference amplification network specifically designed for detecting small targets. Each infrared video frame in the training infrared video is input into a weighted feature extractor (backbone network) to extract features and obtain an infrared feature map.

[0034] The spatial feature extraction module, frequency domain feature extraction module and time feature extraction module are three parallel branches, which respectively use the spatial, frequency and time domain feature learning strategies to extract features from the infrared feature map group to obtain the spatial feature map Fs, the frequency feature map group and the time feature map Ft.

[0035] The first fusion enhancement module fuses the spatial feature map Fs with the frequency feature map group to obtain a space-frequency fusion feature map. The second fusion enhancement module fuses the frequency feature map group with the time feature map Ft to obtain a time-frequency fusion feature map. The third fusion enhancement module fuses the spatial-frequency feature map and the time-frequency feature map to generate a space-time-frequency fusion feature map. The fully interacting space-time-frequency fusion feature map is passed to the detection head to obtain the final prediction result.

[0036] The first, second, and third fusion enhancement modules in the embodiment of the present invention all include a fusion enhancement module (FEM), which is used to eliminate potential feature mismatches between different fields and simultaneously fuse and enhance features from different fields.

[0037] In one embodiment of the present invention, Figure 5As shown, the backbone network may include a first branch, a second branch and a residual connection layer, and the first branch and the second branch are both connected to the residual connection layer, wherein the first branch includes a first convolution unit, a second convolution layer, an expansion convolution unit, a spatial attention layer and a channel attention layer connected in sequence, the first convolution unit includes 8 parallel first convolution layers, and the directions of convolution of the convolution kernels corresponding to the 8 first convolution layers are horizontally right, horizontally left, vertically upward, vertically downward, upper right, upper left, lower right and lower left respectively. The expansion convolution unit includes 4 parallel expansion convolution layers, and the second branch includes a third convolution layer and a fourth convolution layer connected in sequence. The backbone network is used to extract features from each infrared video frame in the training infrared video, which may include: Calculate the grayscale difference between the center pixel and the boundary pixel of the infrared video frame in the corresponding direction using the eight parallel first convolution layers in the first convolution unit to obtain eight first feature maps; Multiply the two feature maps with opposite convolution directions in the 8 first feature maps element by element to obtain 4 difference feature maps; Add the four difference feature maps element-by-element in the vertical reverse direction to obtain two second feature maps; The second convolutional layer is used to extract features from the first spliced ​​feature map to obtain a primary feature map, where the first spliced ​​feature map is obtained by splicing two second feature maps; Performing a residual connection process on the primary feature map and the third feature map to obtain a fourth feature map, wherein the third feature map is extracted from the infrared video frame by the third convolutional layer; The fourth feature map is subjected to dilated convolution processing by using four parallel dilated convolution layers in the dilated convolution unit to obtain four dilated feature maps; The four expanded feature maps are spliced ​​together to obtain a secondary feature map; Use the spatial attention layer to extract features from the secondary feature map to obtain the fifth feature map; The fifth feature map of the channel attention layer is used for feature extraction to obtain the sixth feature map; The residual connection layer is used to perform residual connection processing on the sixth feature map and the seventh feature map to obtain an infrared feature map, wherein the seventh feature map is extracted from the third feature map by the fourth convolution layer.

[0038] The 8 preset directions in the embodiment of the present invention are based on the horizontal right direction of the center of the convolution kernel as 0°, and are divided into one direction every 45° in the clockwise (or counterclockwise) direction. The angles of the 8 directions are: 0°: horizontal to the right (along the positive direction of the x-axis). 45°: tilted to the upper right (diagonal direction). 90°: vertically upward (along the positive direction of the y-axis). 135°: tilted to the upper left (diagonal direction). 180°: horizontal to the left (along the negative direction of the x-axis). 225°: tilted to the lower left (diagonal direction). 270°: vertically downward (along the negative direction of the y-axis). 315°: tilted to the lower right (diagonal direction). For the parameters of the convolution kernels corresponding to the 8 first convolution layers, see Figure 6 .

[0039] Specifically, the first branch and the second branch process the infrared video frame respectively.

[0040] In the second branch, the third convolutional layer performs feature extraction on the infrared video frame to obtain a third feature map, and the fourth convolutional layer performs feature extraction processing on the third feature map to obtain a seventh feature map.

[0041] In the first branch, 8 (3×3) first convolution layers with specific weights are used to perform convolution in the above 8 preset directions, and the grayscale difference between the center pixel and the boundary pixel in the corresponding direction of the infrared video frame is calculated to obtain 8 first feature maps. Then, the two feature maps with opposite convolution directions in the 8 first feature maps are multiplied element by element to amplify the difference, obtaining 4 difference feature maps. Then, the 4 difference feature maps are added in pairs along the vertical direction to obtain 2 second feature maps. The two second feature maps are spliced ​​and passed through the (3×3) second convolution layer to obtain the primary feature map. .

[0042] Initial feature map The fourth feature map is obtained after residual connection with the third feature map. The fourth feature map is directly spliced ​​after four parallel dilated convolutions to obtain a secondary feature map. Finally, the secondary feature map is processed by the spatial attention mechanism to obtain the fifth feature map, and the fifth feature map is processed by the channel attention mechanism to obtain the sixth feature map. The sixth feature map is connected to the seventh feature map through residual connection to obtain the infrared feature map. .

[0043] In this embodiment, the four parallel dilated convolutional layers in the dilated convolution unit are: a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer and a fourth dilated convolutional layer, see Figure 5 The kernel sizes of the first, second, third, and fourth dilated convolutional layers are 1, 3, 5, and 7, respectively.

[0044] In one embodiment of the present invention, Figure 7 As shown, the spatial feature extraction module may include a third branch, a fourth branch, and a memory enhancement submodule, and the third branch and the fourth branch are both connected to the memory enhancement submodule. The third branch may include a fifth convolutional layer, a sixth convolutional layer, a spatial enhancement submodule, and a first embedding layer connected in sequence, and the fourth branch may include a second embedding layer. Using the spatial feature extraction module to extract spatial features from the infrared feature map group may include: Splicing the first N-1 infrared feature maps in the infrared feature map group to obtain a second spliced ​​feature map; Using the fifth convolutional layer to extract features from the second concatenated feature map to obtain an eighth feature map; Multiplying the eighth feature map by the Nth video feature map in the infrared feature map group element-by-element, and then splicing the resultant with the Nth infrared feature map in the infrared feature map group to obtain a third spliced ​​feature map; Use the sixth convolutional layer to extract features from the third concatenated feature map to obtain an initial feature map; The spatial enhancement submodule is used to perform spatial enhancement processing on the initial feature map to obtain a spatially enhanced feature map, wherein the spatial enhancement submodule includes four sub-branches and a seventh convolutional layer, and the four sub-branches perform feature processing on the initial feature map respectively. After the processing results of sub-branch three and sub-branch four are residually connected, the seventh convolutional layer is used to extract features from the residual connection result, and the extraction result is multiplied element by element by the addition result to obtain a spatially enhanced feature map, and the addition result is the element-by-element addition result of the processing results of sub-branch one and sub-branch two. Sub-branch one may include a first maximum pooling layer and a first fully connected layer connected in sequence, sub-branch two may include a first average pooling layer and a second fully connected layer connected in sequence, sub-branch three may include a channel maximum pooling layer, and sub-branch four may include a channel average pooling layer; The spatial enhancement feature map and the initial feature map are added element by element to obtain the local-global spatial feature; Generate a first key-value pair based on the local-global spatial features using the first embedding layer, wherein the first key-value pair includes a key Sum ; Using the memory enhancement submodule, a spatial feature map is obtained according to the first key-value pair and the second key-value pair, wherein the second key-value pair is generated by the second embedding layer according to the Nth video feature map in the feature map group, and the second key-value pair includes the key Sum , where the memory enhancement submodule uses the softmax activation function to and key The product of is normalized, and the similarity obtained by normalization is compared with the value After multiplication, the value Perform residual connection processing, and the residual connection result is processed by the matching layer to obtain the spatial feature map.

[0045] It should be noted that spatial relationship modeling is the key to accurately predicting and tracking the target position. In order to consider the global context information of the key frame and the spatial dependency between adjacent frames as much as possible, the embodiment of the present invention sets a spatial feature extraction module.

[0046] For example, when N is 5, the spatial feature extraction module first uses the first four frames in the infrared feature map group as reference frames, with the last frame as the keyframe. The first four keyframes are concatenated and passed through a (3×3) convolutional layer (the fifth convolutional layer). These frames are then element-wise multiplied with the last keyframe before being concatenated. This is then passed through a (3×3) convolutional layer (the sixth convolutional layer) to produce the initial feature map. The initial feature map is then passed through the spatial enhancement submodule to extract internal feature correlations. The resulting spatially enhanced feature map is then added to the initial feature map to produce the local-global spatial features.

[0047] The spatial enhancement submodule in the embodiment of the present invention includes four sub-branches and a seventh convolutional layer, see Figure 7 , the four sub-branches perform feature processing on the initial feature map respectively, and after performing residual connection on the processing results of sub-branch three and sub-branch four, the seventh convolutional layer is used to extract features from the residual connection results, and the extraction results are residually connected with the addition results to obtain a spatial enhanced feature map. The addition result is the element-by-element addition result of the processing results of sub-branch one and sub-branch two. Sub-branch one may include the first maximum pooling layer and the first fully connected layer connected in sequence, sub-branch two may include the first average pooling layer and the second fully connected layer connected in sequence, sub-branch three may include a channel maximum pooling layer, and sub-branch four may include a channel average pooling layer.

[0048] Four parallel sub-branches are used to process the initial feature map. The initial feature map from sub-branch one is first processed through a global max pooling layer (the first max pooling layer) and a fully connected layer (the first fully connected layer). The initial feature map from sub-branch two is then processed through a global average pooling layer (the first average pooling layer) and a fully connected layer (the second fully connected layer). The processing results of sub-branches one and two are pixel-wise added together to produce a channel-wise attention feature map. The initial feature map from sub-branch three is then processed through a channel-wise max pooling layer, where the maximum value is taken along the channel dimension. The initial feature map from sub-branch four is then processed through a channel-wise average pooling layer, where the average value is taken along the channel dimension. The processing results of sub-branches three and four are concatenated and then processed through a 7x7 convolution (the seventh convolutional layer) to produce a spatial attention feature map. Finally, the channel-wise attention feature map and the spatial attention feature map are element-wise multiplied to produce a spatially enhanced feature map.

[0049] The local-global spatial features are stored in memory, the keyframe features are used as queries, and two different embedding layers (the first embedding layer and the second embedding layer) are used to generate the corresponding key-value pairs ( , , , ). Use the Memory Enhancement Unit (MEU) to expand the infrared small target features. First, key and key After multiplication, the Softmax activation function is used for normalization to calculate the similarity between them. Multiply to utilize the value Query the area related to the infrared small target feature. The product and value of After the connection is made, the matching layer is used to establish the connection between the key frame and the reference frame to obtain the spatial feature map .

[0050] In one embodiment of the present invention, Figure 8 As shown, using the frequency domain feature extraction module to extract frequency features from each infrared feature image in the infrared feature image group may include: The Fourier transform is used to convert the features of the infrared feature map into the frequency domain to obtain the frequency domain information of the infrared feature map. The frequency domain information includes the amplitude and phase ; Using the first convolution group, according to the amplitude and magnitude mask The product of , obtains the amplitude feature, where the amplitude mask The amplitude is calculated by the eighth and ninth convolutional layers connected in sequence. and phase The splicing results are extracted; Using the second convolution group, according to the phase and phase mask The product of , we get the phase feature, where the phase mask The amplitude is calculated by the eighth convolutional layer and the tenth convolutional layer connected in sequence. and phase The splicing results are extracted; Using inverse Fourier transform, the infrared feature map in the frequency domain is transformed to the original definition domain according to the amplitude and phase characteristics to obtain the frequency feature map ; Among them, the convolution group in the first convolution group and the second convolution group includes the eleventh convolution layer, the second maximum pooling layer, the second average pooling layer, the first depth-separable convolution layer and the first point-by-point convolution layer. The eleventh convolution layer is used to extract features from the input product, and the second maximum pooling layer and the second average pooling layer are used to perform pooling processing on the feature extraction results respectively. After the maximum pooling result and the average pooling result are residually concatenated, the first depth-separable convolution layer and the first point-by-point convolution layer are used in turn to process the residual concatenation results to obtain output features. The input product includes amplitude and magnitude mask The product and phase and phase mask The output features include amplitude features and phase features.

[0051] It should be noted that the frequency domain features provide a comprehensive understanding of small infrared targets at different frequencies, while also reducing image noise and interference.

[0052] Specifically, Fourier transform is used to convert each frame of infrared feature map into frequency domain, and the frequency domain information is further converted into Further decomposed into amplitude and phase For the amplitude and phase Splicing is performed and processed through two convolutional layers to obtain the amplitude mask Magnitude mask and amplitude Multiplication guides the input of amplitude, and the first convolution group guides the extraction of amplitude features to obtain amplitude features. For phase, a similar process is followed to obtain phase features. Use the inverse Fourier transform to transform the processed frequency domain features (amplitude features and phase features) back to their original definition domain to obtain the frequency domain features of the infrared feature map in the frequency domain. .

[0053] In this embodiment of the present invention, the convolution groups (CGs) in the first and second convolution groups perform the following processing on the input product: the input product is first processed by a normal (3×3) convolution (the eleventh convolution layer), then subjected to maximum pooling (the second maximum pooling layer) and average pooling (the second average pooling layer) to extract channel information. The outputs of the second maximum pooling layer and the second average pooling layer are then subjected to a residual connection. Finally, a (3×3) depthwise convolution (the first depthwise separable convolution layer) is used to independently learn spatial context information on each channel, and cross-channel mixing of (1×1) pointwise convolution (the first pointwise convolution layer) is used to aggregate pixels. The output of the first pointwise convolution layer is multiplied pixel by pixel with the input product to obtain the features output by the convolution group.

[0054] In one embodiment of the present invention, Figure 9-11 As shown, the temporal feature extraction module may include a coarse motion estimation branch and a feature enhancement branch connected in sequence. The coarse motion estimation branch may include a feature splicing layer, a second convolution unit, and a twelfth convolution layer connected in sequence. The feature enhancement branch may include a sixteenth convolution layer, a seventeenth convolution layer, an eighteenth convolution layer, a softmax activation function, a nineteenth convolution layer, a twentieth convolution layer, a flattened layer and a deformable attention submodule, a twenty-first convolution layer, and a 3D convolution layer. The temporal feature extraction module is used to extract temporal features from the infrared feature map group, which may include: Using the feature splicing layer, forward and backward splicing are performed on two adjacent infrared feature maps in the infrared feature map group to obtain a first set of inter-frame relationship maps and a second set of inter-frame relationship maps, wherein the first set of inter-frame relationship maps and the second set of inter-frame relationship maps each include four inter-frame relationship maps; Using a second convolution unit to capture the inter-frame relationships of homologous points in the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs at different motion speeds, respectively, to obtain a first inter-frame relationship and a second inter-frame relationship, wherein the second convolution unit includes a thirteenth convolution layer, a first ReLU activation function, a fourteenth convolution layer, a second ReLU activation function, a fifteenth convolution layer, and a third ReLU activation function connected in sequence; The twelfth convolutional layer is used to extract features of the first and second inter-frame relationships, and four forward optical flows are obtained. and 4 backward optical flows ; The forward optical flow Towards After distortion, the forward optical flow Add together to get the forward optical flow across two frames ; The backward optical flow Towards After distortion, the backward optical flow Add up to get the backward optical flow across two frames ; Each infrared feature map in the infrared feature map group and the forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Align to the middle frame to obtain the alignment features; Infrared feature map group, alignment features, forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Perform connection processing to obtain connection features; The connection features are extracted using the sixteenth convolutional layer to obtain the ninth feature map; The deformable attention submodule is used to process the sampling point offset, the standardized attention weight and the amortized feature to obtain the refined feature, where the sampling point offset is obtained according to the tenth feature map and the offset, where the tenth feature map is extracted from the ninth feature map by the seventeenth convolutional layer, and the offset is obtained according to the forward optical flow. , forward optical flow across two frames , backward optical flow and the backward optical flow across two frames The calculated,normalized attention weights are obtained by processing the ninth feature map through the,sequentially connected tenth convolutional layer and the softmax activation,function, and the flattened features are obtained by processing the infrared feature map group,through the sequentially connected nineteenth convolutional layer, the twentieth convolutional layer, and the flattened,layer; The 21st convolutional layer is used to extract the refined features to obtain the 11th feature map; The 3D convolution layer is used to process the features of the infrared feature map group and the eleventh feature map after residual connection to obtain N twelfth feature maps; Perform residual connection on N twelfth feature maps to obtain a temporal feature map.

[0055] The temporal feature extraction module in this embodiment of the present invention consists of two main components: a coarse motion estimation module and a feature enhancement module. The coarse motion estimation module extracts coarse inter-frame motion information of the target and generates forward and backward optical flows. The feature enhancement module then combines the forward and backward optical flows with deformable convolution to extract temporal features.

[0056] Coarse motion estimation: In order to extract the feature differences between frames, infrared features of 5 adjacent frames are used. Figure 1 As input. Using the feature concatenation layer ( Figure 9 (not shown) The two adjacent frames in the five infrared feature maps are spliced ​​in the channel dimension according to forward splicing and backward splicing, respectively, to obtain two sets of inter-frame relationship maps (the first set of inter-frame relationship maps and the second set of inter-frame relationship maps), each set of which includes four inter-frame relationship maps. The second convolution unit captures the inter-frame relationship of the first set of inter-frame relationship maps and the second set of inter-frame relationship maps at different motion speeds. Specifically, the thirteenth convolution layer with an expansion rate of 3, the fourteenth convolution layer with an expansion rate of 5, and the fifteenth convolution layer with an expansion rate of 1 are used to capture the inter-frame relationship of the same source points at different motion speeds. Finally, four forward optical flows are obtained through the (3×3) convolution layer (the twelfth convolution layer). , and four backward optical flows , These optical flows are combined with multi-frame infrared features Figure 1The images are sent to the feature enhancement module for alignment and enhancement of infrared small target features.

[0057] Feature Enhancement Module, see Figure 10 , first of all, the forward optical flow Use warp operation to make it Distort, and then compare the distorted result with Add together to get the forward optical flow across two frames The backward optical flow is processed in the same way. Use warp operation to make it Distort, and then compare the distorted result with Add up to get the backward optical flow across two frames .

[0058] Secondly, each infrared feature map (except the middle frame) and the optical flow in the corresponding direction (i.e. forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames ) Align to the intermediate frame through the warp operation to obtain the alignment features , Specifically, the first infrared feature map is aligned to the third frame through the forward optical flow from (1 to 3), the second infrared feature map is aligned to the third frame through the forward optical flow from (2 to 3), the fourth infrared feature map is aligned to the third frame through the backward optical flow from (4 to 3), and the fifth infrared feature map is aligned to the third frame through the backward optical flow from (5 to 3), driving the optical flow to the middle frame.

[0059] Then combine the infrared feature map and align the features , forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Expand along one dimension and splice together to form a connection feature , , where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0060] Connection features After a (3×3) convolutional layer (the sixteenth convolutional layer) feature extraction process, the ninth feature map is obtained, and the obtained ninth feature map is divided into two branches.

[0061] One of the branches is used to calculate the sampling point offset. Specifically, the 17th convolutional layer is used to extract the features of the 9th feature map to obtain the 10th feature map. (Forward optical flow , forward optical flow across two frames ) and backward optical flow (Backward Optical Flow , backward optical flow across two frames ) can calculate the offset. The tenth feature map and the offset are added to generate the sampling point offset , , where T represents the number of frames, H represents the height, W represents the width, M represents the number of attention heads, and P represents the number of sampling points.

[0062] The other branch generates normalized attention weights Specifically, the eighteenth convolutional layer and the softmax activation function are connected in sequence to extract the features of the ninth feature map and obtain the standardized attention weights , , where T represents the number of frames, H represents the height, W represents the width, M represents the number of attention heads, and P represents the number of sampling points.

[0063] The five infrared feature maps are processed by two layers of 3×3 convolution (the 19th convolution layer and the 20th convolution layer) and a flattening layer to be flattened into flat features. , , where C represents the number of channels, T, H, and W represent the number of frames, height, and width respectively. , sampling point offset , the normalized attention weight Processed by Deformable Attention Function (DAF), refined features are output , , where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0064] To facilitate network optimization, 5 infrared feature maps and refined features The residual connection is added and the multi-frame twelfth feature map is generated through a (1×1×5) 3D convolution (3D convolution layer). , the twelfth feature map of multiple frames After residual connection, the temporal feature map is obtained , , where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0065] In an embodiment of the present invention, in the deformable attention function, the sampling point offset Used to identify homologous pixels between adjacent frames, and the standardized attention weights Characterize the possibility of pixels as targets, amortize features Stores the initial information of the image. In subsequent use, the final time feature map The twelfth feature map of multiple frames needs to be Get it by splicing.

[0066] In one embodiment of the present invention, Figure 12 As shown, the first fusion enhancement module, the second fusion enhancement module and the third fusion enhancement module all include a fusion enhancement module, which may include a second depth-separable convolution layer, a second point-by-point convolution layer, a collaborative attention submodule, a gating submodule and a batch normalization layer connected in sequence. The fusion enhancement module is used to receive the first input feature map and the second input feature map, and output a fusion feature map. The fusion enhancement module is specifically used to: Extracting features from the fourth spliced ​​feature using a second depthwise separable convolutional layer and a second pointwise convolutional layer connected sequentially to obtain a thirteenth feature map, wherein the fourth spliced ​​feature is obtained by splicing the first input feature map and the second input feature map along the channel dimension; The collaborative attention submodule is used to fuse the internal features of the thirteenth feature map to obtain the fourteenth feature map; Perform residual connection processing on the thirteenth feature map and the fourteenth feature map to obtain the fifteenth feature map; Use the gating submodule to generate the weight value gate of the fifteen feature maps; Add the first product and the second product to obtain sixteen features, where the first product is the product of the fifteenth feature map and the weight value gate, and the second product is the product of (1-gate) and the fourth splicing feature; The sixteen features are processed using a batch normalization layer to obtain a fused feature map.

[0067] The present invention utilizes spatial, temporal, and frequency domain features of infrared targets. However, due to the different learning paths of features in different domains, feature mismatches often occur. Therefore, a Fusion Enhancement Module (FEM) is designed to further compensate for these features, fusing and enhancing the differences and commonalities between features in different domains.

[0068] The Fusion Enhancement Module (FEM) concatenates the two input domain features (the first and second input feature maps) along the channel dimension to generate the fourth concatenated feature map. This concatenated feature map is then processed using depthwise convolution (the second depthwise separable convolution layer) and pointwise convolution (the second pointwise convolution layer) to generate the thirteenth feature map. The thirteenth feature map is then fused with internal features using the Co-Attention (co_att) submodule to generate the fourteenth feature map. The thirteenth and fourteenth feature maps are then connected using a residual connection to generate the fifteenth feature map.

[0069] The collaborative attention submodule in the embodiment of the present invention: the input feature map (the thirteenth feature map) is subjected to global average pooling (the third average pooling layer) to adjust the image size to , where T represents the number of frames and C represents the number of channels. Then, two layers of (1×1) convolution (the 22nd and 23rd convolutional layers) are used to adjust the channel weight information. During the processing, the number of channels changes from C to C / 4 and finally to C.

[0070] Next, the fifteen feature maps pass through a gating submodule to obtain a weight value, gate. The weight value gate is multiplied by the fifteen feature maps, and the result of multiplying (1-gate) by the fourth concatenated feature is added to obtain the final output.

[0071] It should be noted that the processing flow of the gating submodule is the same as that of the collaborative attention module, but the number of channels in the processing flow is changed so that the output of the gating submodule is a weight value , where T represents the number of frames.

[0072] In one embodiment of the present invention, the detection model loss function for: ; in, represents the detection model loss function, represents the boundary regression loss function, represents the classification loss function, represents the target probability loss function, represents the forward optical flow loss function, represents the backward optical flow loss function, represents the boundary regression loss hyperparameter, represents the classification loss hyperparameter, represents the target probability loss hyperparameter, represents the optical flow loss hyperparameter; ; in, represents the boundary regression loss function, represents the intersection-over-union hyperparameter, represents the intersection-over-union loss function, represents the normalized Gaussian distance hyperparameter, represents the normalized Gaussian distance loss function, represents the predicted bounding box, represents the ground-truth bounding box, represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, represents the predicted bounding box Gaussian distribution, represents the Gaussian distribution of the true bounding box, represents a constant; Classification loss function for: ; in, represents the classification loss function, i represents the number of training infrared videos in the training dataset, c is the number of categories, For real classification, To predict classification; Target probability loss function for: ; in, represents the target probability loss function, i represents the number of training infrared videos in the training dataset, is the true probability, is the predicted probability; Forward optical flow loss function for: ; in, represents the forward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the forward true image, represents the forward estimated image; Backward optical flow loss function for: ; in, represents the backward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the backward true image, Represents the backward estimated image.

[0073] In the embodiment of the present invention, the three-domain deep learning network realizes three-domain learning. In order to obtain better results, the loss function uses the modified YOLOX loss function and the optical flow loss for supervising optical flow. Detection model loss function as follows: ; in, represents the detection model loss function, is the boundary regression loss, is the classification loss, is the target probability loss, is the forward optical flow loss. is the backward optical flow loss. 、 、 and are the four hyperparameters of the balanced loss term.

[0074] As a boundary regression loss, IOU (Intersection over Union) loss is usually used for calculation. However, due to the small size of the target, the popular IOU loss may not be enough to capture the detailed information of the target distribution. Taking this into account, the difference between target regions is minimized by introducing the normalized Gaussian Wasserstein distance (NWD) loss. IOU loss helps the network accurately locate small targets, while NWD loss learns their distribution characteristics. Therefore, the boundary regression loss in the embodiment of the present invention is It is a regression loss that combines IOU loss and NWD loss.

[0075] ; in, is the predicted bounding box, is the bounding box of the marker. and Denotes the Gaussian distribution of the predicted box and the Gaussian distribution of the labeled box respectively. C is a constant related to the dataset. and are two hyperparameters that balance the loss term. q for forward optical flow and backward optical flow Use forward optical flow loss and backward optical flow loss Decompose the input multi-frame infrared video frame into forward images , backward image . Forward image pair Perform the warp operation to obtain the backward estimated image Similarly, the backward image pair Perform warp operation to obtain the forward estimated image The mean square error (MSE) is used as a metric to quantify the similarity between the original image and the reconstructed image, and the final forward optical flow loss is and backward optical flow loss Respectively expressed as: ; in, represents the forward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the forward true image, represents the forward estimated image.

[0076] ; in, represents the backward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the backward true image, Represents the backward estimated image.

[0077] The densely moving infrared small target detection algorithm based on three-domain feature learning proposed in the embodiments of the present invention significantly improves the detection performance of densely moving small targets in complex scenarios by fusing multi-dimensional features in time, space, and frequency. By using multi-domain feature complementary enhancement and fusing features in the time, space, and frequency domains, the problem of insufficient representation of dense small targets by single-domain features is resolved, significantly improving detection accuracy. The loss function uses a composite loss function that includes bounding box regression, front-to-back optical flow, and classification probability. Combining the IOU and normalized Gaussian Wasserstein distance (NWD) loss, it balances target positioning accuracy with distribution characteristic learning, optimizing the recall rate and positioning accuracy of small target detection. To address the difficult problem of missed detection and false detection of densely moving targets, the method proposed in the embodiments of the present invention can reduce the false detection rate in dense scenes and has broad application prospects.

[0078] To address the problem of weak correlation between features of densely moving small targets, an embodiment of the present invention proposes a three-domain feature collaborative learning strategy. It models global contextual relationships through non-local attention in the spatial domain, captures motion trajectories through deformable convolution guided by optical flow in the time domain, and extracts multi-dimensional spectral features through Fourier decomposition in the frequency domain. The complementarity of features in different domains is used to achieve accurate representation of densely moving small targets in complex scenarios, thus solving the problem of insufficient feature information in a single domain.

[0079] To address the problem of cross-domain feature fusion mismatch, an embodiment of the present invention designs a fusion enhancement module (FEM), which dynamically allocates weights to spatial-temporal-frequency domain features through the dual mechanisms of channel attention and spatial attention, and combines residual connections to compensate for feature differences, effectively eliminating cross-domain feature offset and improving the robustness of multi-domain feature fusion.

[0080] To address the pain point of insufficient positioning accuracy for dense small targets, an embodiment of the present invention designs a composite loss function that integrates the normalized Gaussian Wasserstein distance (NWD) loss and the IOU loss, taking into account both target distribution characteristics and bounding box regression accuracy. At the same time, forward and backward optical flow reconstruction losses are introduced to constrain motion feature learning, achieving a dual improvement in dense target detection accuracy and motion trajectory stability.

[0081] The present invention also provides an electronic device.

[0082] In this embodiment, if Figure 13 As shown, the electronic device 500 includes a processor 501, a memory 503 and a computer program stored in the memory. When the processor executes the computer program, the above-mentioned densely moving infrared small target detection method based on three-domain feature learning is implemented.

[0083] Figure 13 4 is a structural block diagram of a controller according to an embodiment of the present invention.

[0084] like Figure 13 As shown, controller 500 includes a processor 501 and a memory 503. Processor 501 and memory 503 are connected, for example, via a bus 502. Optionally, controller 500 may further include a transceiver 504. It should be noted that in practical applications, the number of transceivers 504 is not limited to one, and the structure of controller 500 does not constitute a limitation on the embodiments of the present invention.

[0085] Processor 501 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 501 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0086] Bus 502 may include a path for transmitting information between the above components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0087] Memory 503 is used to store a computer program corresponding to the method for detecting small, densely moving infrared targets based on three-domain feature learning according to the above-described embodiment of the present invention. The computer program is controlled and executed by processor 501. Processor 501 is used to execute the computer program stored in memory 503 to implement the contents of the above-described method embodiment. Figure 13 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0088] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0089] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0090] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0091] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0093] In the present invention, unless otherwise specified or limited, the terms "installed," "connected," "connect," "fixed," etc. should be understood in a broad sense. For example, they can refer to fixed connection, detachable connection, or integration; mechanical connection, electrical connection; direct connection, or indirect connection through an intermediate medium; internal communication between two components, or interaction between two components, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0094] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0095] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for detecting densely moving infrared small targets based on three-domain feature learning, characterized in that: The method comprises: Acquire an infrared video to be tested, wherein the infrared video to be tested includes N consecutive infrared video frames, where N is a positive integer; The infrared video to be tested is input into a pre-trained dense motion infrared small target detection model to obtain a detection result, wherein the detection result includes the detection bounding box of the infrared small target, the Gaussian distribution of the detection bounding box, the detection classification and the detection probability. The dense motion infrared small target detection model includes a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network and a detection head. The input end of the backbone network is used to input the infrared video to be tested, and the output end of the backbone network is respectively connected to the input end of the spatial feature extraction module, the input end of the frequency domain feature extraction module and the input end of the temporal feature extraction module. The output ends of the spatial feature extraction module, the frequency domain feature extraction module and the temporal feature extraction module are all connected to the input end of the fusion enhancement network. The output end of the fusion enhancement network is connected to the input end of the detection head, and the detection head is used to output the detection result.

2. The method for detecting small infrared targets in dense motion according to claim 1, characterized in that: The training process of the dense motion infrared small target detection model includes: Obtaining a training data set, wherein the training data set includes a training infrared video and a labeling result corresponding to each infrared video frame in the training infrared video, the training infrared video includes N consecutive infrared video frames, and the labeling result includes a true bounding box of a small infrared target, a true bounding box Gaussian distribution, a true classification, and a true probability; Inputting the training infrared video in the training data set into the dense motion infrared small target detection model to obtain a prediction result, wherein the prediction result includes a predicted bounding box, a predicted bounding box Gaussian distribution, a predicted classification, and a predicted probability of the infrared small target; Constructing a boundary regression loss function according to the predicted bounding box, the predicted bounding box Gaussian distribution, the true bounding box and the true bounding box Gaussian distribution, constructing a classification loss function according to the predicted classification and the true classification, constructing a target probability loss function according to the true probability and the predicted probability, constructing a forward optical flow loss function and a backward optical flow loss function according to the training infrared video of the input detection model, and constructing a detection model loss function according to the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function and the backward optical flow loss function; The densely moving infrared small target detection model is trained according to the detection model loss function to obtain a trained densely moving infrared small target detection model.

3. The method for detecting small infrared targets in dense motion according to claim 2, characterized in that: The fusion enhancement network includes a first fusion enhancement module, a second fusion enhancement module and a third fusion enhancement module. The output ends of the spatial feature extraction module and the frequency domain feature extraction module are both connected to the input end of the first fusion enhancement module, the output ends of the frequency domain feature extraction module and the time feature extraction module are both connected to the input end of the second fusion enhancement module, the output ends of the first fusion enhancement module and the second fusion enhancement module are both connected to the input end of the third fusion enhancement module, and the output end of the third fusion enhancement module is connected to the input end of the detection head. The training infrared video in the training data set is input into the dense motion infrared small target detection model, including: Using the backbone network to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map group, wherein the infrared feature map group includes N infrared feature maps; Using the spatial feature extraction module to extract spatial features from the infrared feature map group to obtain a spatial feature map; Using the frequency domain feature extraction module to extract frequency features from each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps; Using the time feature extraction module to extract time features from the infrared feature image group to obtain a time feature image; Using the first fusion enhancement module to generate a space-frequency fusion feature map according to the spatial feature map and the frequency feature map group, using the second fusion enhancement module to generate a time-frequency fusion feature map according to the frequency feature map group and the time feature map, and using the third fusion enhancement module to generate a space-time-frequency fusion feature map according to the space-frequency feature map and the time-frequency feature map; The detection head is used to generate the prediction result according to the space-time-frequency fusion feature map.

4. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The backbone network includes a first branch, a second branch and a residual connection layer, and the first branch and the second branch are both connected to the residual connection layer, wherein the first branch includes a first convolution unit, a second convolution layer, an expansion convolution unit, a spatial attention layer and a channel attention layer connected in sequence, the first convolution unit includes 8 parallel first convolution layers, and the directions of convolution of the corresponding convolution kernels of the 8 first convolution layers are horizontally right, horizontally left, vertically upward, vertically downward, upper right, upper left, lower right and lower left respectively. The expansion convolution unit includes 4 parallel expansion convolution layers, and the second branch includes a third convolution layer and a fourth convolution layer connected in sequence. The backbone network is used to extract features from each infrared video frame in the training infrared video, including: Calculating grayscale differences between central pixels and boundary pixels of the infrared video frame in corresponding directions using eight parallel first convolutional layers in the first convolutional unit to obtain eight first feature maps; Multiply the two feature maps with opposite convolution directions in the 8 first feature maps element by element to obtain 4 difference feature maps; Adding the four difference feature maps element-by-element in a vertical reverse direction in pairs to obtain two second feature maps; Using the second convolutional layer to perform feature extraction on the first spliced ​​feature map to obtain a primary feature map, wherein the first spliced ​​feature map is obtained by splicing two of the second feature maps; Performing a residual connection process on the primary feature map and the third feature map to obtain a fourth feature map, wherein the third feature map is extracted from the infrared video frame by the third convolutional layer; performing dilated convolution processing on the fourth feature map using four parallel dilated convolution layers in the dilated convolution unit respectively to obtain four dilated feature maps; Splicing the four expanded feature maps to obtain a secondary feature map; Performing feature extraction on the secondary feature map using the spatial attention layer to obtain a fifth feature map; Perform feature extraction using the fifth feature map of the channel attention layer to obtain a sixth feature map; The residual connection layer is used to perform residual connection processing on the sixth feature map and the seventh feature map to obtain an infrared feature map, wherein the seventh feature map is extracted from the third feature map by the fourth convolutional layer.

5. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The spatial feature extraction module includes a third branch, a fourth branch, and a memory enhancement submodule, wherein the third branch and the fourth branch are both connected to the memory enhancement submodule, wherein the third branch includes a fifth convolutional layer, a sixth convolutional layer, a spatial enhancement submodule, and a first embedding layer connected in sequence, and the fourth branch includes a second embedding layer. The spatial feature extraction module is used to extract spatial features from the infrared feature map group, including: splicing the first N-1 infrared feature images in the infrared feature image group to obtain a second spliced ​​feature image; Performing feature extraction on the second concatenated feature map using the fifth convolutional layer to obtain an eighth feature map; Multiplying the eighth feature map by the Nth video feature map in the infrared feature map group element by element, and then splicing the resultant with the Nth infrared feature map in the infrared feature map group to obtain a third spliced ​​feature map; Using the sixth convolutional layer to perform feature extraction on the third spliced ​​feature map to obtain an initial feature map; The spatial enhancement submodule is used to perform spatial enhancement processing on the initial feature map to obtain a spatially enhanced feature map, wherein the spatial enhancement submodule includes four sub-branches and a seventh convolutional layer, and the four sub-branches perform feature processing on the initial feature map respectively. After the processing results of sub-branch three and sub-branch four are residually connected, the seventh convolutional layer is used to perform feature extraction on the residual connection result, and the extraction result is multiplied element by element by the addition result to obtain a spatially enhanced feature map, and the addition result is the element-by-element addition result of the processing results of sub-branch one and sub-branch two. The sub-branch one includes a first maximum pooling layer and a first fully connected layer connected in sequence, the sub-branch two includes a first average pooling layer and a second fully connected layer connected in sequence, the sub-branch three includes a channel maximum pooling layer, and the sub-branch four includes a channel average pooling layer; Adding the spatial enhancement feature map and the initial feature map element by element to obtain a local-global spatial feature; Generate a first key-value pair based on the local-global spatial feature using the first embedding layer, wherein the first key-value pair includes a key Sum ; Using the memory enhancement submodule, a spatial feature map is obtained according to the first key-value pair and the second key-value pair, wherein the second key-value pair is generated by the second embedding layer according to the Nth video feature map in the feature map group, and the second key-value pair includes a key Sum , wherein the memory enhancement submodule uses the softmax activation function to and key The product of is normalized, and the similarity obtained by normalization is compared with the value After multiplication, the value Perform residual connection processing, and the residual connection result is processed by the matching layer to obtain the spatial feature map.

6. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The extracting frequency features of each infrared feature image in the infrared feature image group by using the frequency domain feature extraction module includes: The characteristics of the infrared characteristic image are converted into the frequency domain by Fourier transform to obtain the frequency domain information of the infrared characteristic image. The frequency domain information includes the amplitude and phase ; Using the first convolution group, according to the amplitude and magnitude mask The product of , obtains the amplitude feature, where the amplitude mask The amplitude is calculated by the eighth convolutional layer and the ninth convolutional layer connected in sequence. and the phase The splicing results are extracted; Using the second convolution group, according to the phase and phase mask The product of the phase mask is obtained, where The amplitude is calculated by the eighth convolutional layer and the tenth convolutional layer connected in sequence. and the phase The splicing results are extracted; Using inverse Fourier transform, according to the amplitude feature and the phase feature, the infrared feature map in the frequency domain is transformed to the original definition domain to obtain a frequency feature map; wherein, the convolution group in the first convolution group and the second convolution group includes an eleventh convolution layer, a second maximum pooling layer, a second average pooling layer, a first depth-separable convolution layer and a first point-by-point convolution layer, and the eleventh convolution layer is used to extract features of the input product, and the second maximum pooling layer and the second average pooling layer are used to perform pooling processing on the feature extraction results, and after residual concatenation is performed on the maximum pooling result and the average pooling result, the first depth-separable convolution layer and the first point-by-point convolution layer are used in turn to process the residual concatenation results to obtain output features, and the input product includes amplitude and magnitude mask The product and phase and phase mask The output feature includes an amplitude feature and a phase feature.

7. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The temporal feature extraction module includes a coarse motion estimation branch and a feature enhancement branch connected in sequence, the coarse motion estimation branch includes a feature splicing layer, a second convolution unit, and a twelfth convolution layer connected in sequence, the feature enhancement branch includes a sixteenth convolution layer, a seventeenth convolution layer, an eighteenth convolution layer, a softmax activation function, a nineteenth convolution layer, a twentieth convolution layer, a flattening layer and a deformable attention submodule, a twenty-first convolution layer, and a 3D convolution layer, and performing temporal feature extraction on the infrared feature map group using the temporal feature extraction module includes: Using the feature stitching layer, forward and backward stitching is performed on two adjacent infrared feature maps in the infrared feature map group to obtain a first set of inter-frame relationship maps and a second set of inter-frame relationship maps, wherein the first set of inter-frame relationship maps and the second set of inter-frame relationship maps each include four inter-frame relationship maps; The second convolution unit is used to capture the inter-frame relationships of homologous points in the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs at different motion speeds, respectively, to obtain a first inter-frame relationship and a second inter-frame relationship, wherein the second convolution unit includes a thirteenth convolution layer, a first ReLU activation function, a fourteenth convolution layer, a second ReLU activation function, a fifteenth convolution layer, and a third ReLU activation function connected in sequence; The twelfth convolutional layer is used to extract features of the first inter-frame relationship and the second inter-frame relationship respectively to obtain four forward optical flows. and 4 backward optical flows ; The forward optical flow Towards After distortion, the forward optical flow Add together to get the forward optical flow across two frames ; The backward optical flow Towards After distortion, the backward optical flow Add up to get the backward optical flow across two frames ; Each infrared feature map in the infrared feature map group and the forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Align to the middle frame to obtain the alignment features; The infrared feature map group, the alignment feature, and the forward optical flow , forward optical flow across two frames , backward optical flow , backward optical flow across two frames Perform connection processing to obtain connection features; Performing feature extraction on the connection features using the sixteenth convolutional layer to obtain a ninth feature map; The deformable attention submodule is used to process the sampling point offset, the standardized attention weight and the amortized feature to obtain a refined feature, wherein the sampling point offset is obtained according to the tenth feature map and the offset, wherein the tenth feature map is extracted from the ninth feature map by the seventeenth convolutional layer, and the offset is obtained according to the forward optical flow. , forward optical flow across two frames , backward optical flow and the backward optical flow across two frames The standardized attention weight is obtained by processing the ninth feature map through the sequentially connected tenth convolutional layer and the softmax activation function, and the flattened feature is obtained by processing the infrared feature map group through the sequentially connected nineteenth convolutional layer, the twentieth convolutional layer, and the flattened layer; Using the twenty-first convolutional layer to extract the refined features, to obtain an eleventh feature map; Using the 3D convolutional layer to process the features of the infrared feature map group and the eleventh feature map after residual connection, to obtain N twelfth feature maps; Perform residual connection on the N twelfth feature maps to obtain the temporal feature map.

8. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The first fusion enhancement module, the second fusion enhancement module and the third fusion enhancement module all include a fusion enhancement module, which includes a second depth-wise separable convolution layer, a second point-by-point convolution layer, a collaborative attention submodule, a gating submodule and a batch normalization layer connected in sequence. The fusion enhancement module is used to receive the first input feature map and the second input feature map, and output a fusion feature map. The fusion enhancement module is specifically used to: Performing feature extraction on the fourth splicing feature using the second depthwise separable convolutional layer and the second pointwise convolutional layer connected sequentially to obtain a thirteenth feature map, wherein the fourth splicing feature is obtained by splicing the first input feature map and the second input feature map along the channel dimension; Using the collaborative attention submodule to fuse the internal features of the thirteenth feature map to obtain a fourteenth feature map; Performing residual connection processing on the thirteenth feature map and the fourteenth feature map to obtain a fifteenth feature map; Using the gating submodule to generate weight values ​​gate of the fifteen feature maps; Add the first product and the second product to obtain sixteen features, wherein the first product is the product of the fifteen feature maps and the weight value gate, and the second product is the product of (1-gate) and the fourth splicing feature; The sixteen features are processed using the batch normalization layer to obtain a fused feature map.

9. The method for detecting small infrared targets in dense motion according to claim 3, characterized in that: The detection model loss function for: ; in, represents the detection model loss function, represents the boundary regression loss function, represents the classification loss function, represents the target probability loss function, represents the forward optical flow loss function, represents the backward optical flow loss function, represents the boundary regression loss hyperparameter, represents the classification loss hyperparameter, represents the target probability loss hyperparameter, represents the optical flow loss hyperparameter; The boundary regression loss function for: ; in, represents the boundary regression loss function, represents the intersection-over-union hyperparameter, represents the intersection-over-union loss function, represents the normalized Gaussian distance hyperparameter, represents the normalized Gaussian distance loss function, represents the predicted bounding box, represents the ground-truth bounding box, represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, represents the predicted bounding box Gaussian distribution, represents the Gaussian distribution of the true bounding box, represents a constant; The classification loss function for: ; in, represents the classification loss function, i represents the number of training infrared videos in the training dataset, c is the number of categories, For real classification, To predict classification; The target probability loss function for: ; in, represents the target probability loss function, i represents the number of training infrared videos in the training dataset, is the true probability, is the predicted probability; The forward optical flow loss function for: ; in, represents the forward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the forward true image, represents the forward estimated image; The backward optical flow loss function for: ; in, represents the backward optical flow loss function, Indicates the height of the image, represents the width of the image, t represents the time frame, Indicates the Frame No. Row and Column pixels, represents the backward true image, Represents the backward estimated image.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the method for detecting small infrared targets in dense motion based on three-domain feature learning according to any one of claims 1 to 9 is implemented.

Images