A method for detecting densely moving infrared small targets based on three-domain feature learning

Abstract

This invention relates to the field of target detection technology and discloses a method for detecting densely moving infrared small targets based on three-domain feature learning. The method includes: acquiring an infrared video to be tested; inputting the infrared video to be tested into a pre-trained densely moving infrared small target detection model to obtain detection results. The densely moving infrared small target detection model includes a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network, and a detection head. The input end of the backbone network is used to input the infrared video to be tested, and the detection head is used to output the detection results. This detection method uses a densely moving infrared small target model for densely moving infrared small target detection. This model uses multi-domain feature complementarity enhancement, fusing temporal, spatial, and frequency domain features, solving the problem of insufficient representation of densely moving small targets by single-domain features, and significantly improving the detection performance of densely moving small targets in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method for detecting densely moving infrared small targets based on three-domain feature learning. Background Technology

[0002] Infrared small target detection (ISTD) has wide and important applications in traffic management, maritime rescue, and military guidance. Compared to general targets, small targets in infrared images or videos typically have low contrast with the background and may even lack obvious color, shape, or texture features. Considering these two characteristics, in highly complex infrared application scenarios, how detection models can effectively learn and accurately detect weak targets is often a very challenging problem. How to accurately locate and track moving small targets in infrared images and videos has always been a hot topic and a difficult point in vision research. Summary of the Invention

[0003] This invention aims to at least partially address one of the technical problems in related technologies. To this end, one objective of this invention is to propose a method for detecting densely moving infrared small targets based on three-domain feature learning, which solves the problem of insufficient representation of densely moving small targets by single-domain features and significantly improves the detection performance of densely moving small targets in complex scenes.

[0004] The second objective of this invention is to provide an electronic device.

[0005] To achieve the above objectives, a first aspect of the present invention proposes a method for detecting densely moving infrared small targets based on three-domain feature learning. The method includes: acquiring an infrared video to be tested, wherein the infrared video to be tested comprises N consecutive infrared video frames, where N is a positive integer; inputting the infrared video to be tested into a pre-trained densely moving infrared small target detection model to obtain detection results, wherein the detection results include the detection bounding box of the infrared small target, the Gaussian distribution of the detection bounding box, the detection classification, and the detection probability; the densely moving infrared small target detection model includes a backbone network and a spatial feature extraction module. The system comprises a backbone network, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network, and a detection head. The input of the backbone network is used to input the infrared video to be tested. The output of the backbone network is connected to the inputs of the spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module, respectively. The outputs of the spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module are all connected to the input of the fusion enhancement network. The output of the fusion enhancement network is connected to the input of the detection head, which is used to output the detection result.

[0006] The method for detecting densely moving infrared small targets based on three-domain feature learning according to embodiments of the present invention uses a model of densely moving infrared small targets for target detection. This model uses multi-domain feature complementarity enhancement, and the time, space and frequency domain features are fused together, which solves the problem that single-domain features are insufficient for representing densely moving small targets, and significantly improves the detection performance of densely moving small targets in complex scenes.

[0007] In addition, the densely moving infrared small target detection method based on three-domain feature learning proposed in the above embodiments of the present invention may also have the following additional technical features:

[0008] According to an embodiment of the present invention, the training process of the densely moving infrared small target detection model includes: acquiring a training dataset, wherein the training dataset includes training infrared videos and labeling results corresponding to each infrared video frame in the training infrared videos, the training infrared videos include N consecutive infrared video frames, and the labeling results include the true bounding box of the infrared small target, the Gaussian distribution of the true bounding box, the true classification, and the true probability; inputting the training infrared videos in the training dataset into the densely moving infrared small target detection model to obtain prediction results, wherein the prediction results include the predicted bounding box of the infrared small target, the Gaussian distribution of the predicted bounding box, the predicted classification, and the predicted probability; and based on the predicted bounding box, the... A boundary regression loss function is constructed based on the predicted bounding box Gaussian distribution, the ground truth bounding box, and the ground truth bounding box Gaussian distribution. A classification loss function is constructed based on the predicted classification and the ground truth classification. A target probability loss function is constructed based on the ground truth probability and the predicted probability. Forward optical flow loss functions and backward optical flow loss functions are constructed based on the training infrared video input to the detection model. A detection model loss function is constructed based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function, and the backward optical flow loss function. The dense moving infrared small target detection model is trained based on the detection model loss function to obtain a trained dense moving infrared small target detection model.

[0009] According to an embodiment of the present invention, the fusion enhancement network includes a first fusion enhancement module, a second fusion enhancement module, and a third fusion enhancement module. The outputs of the spatial feature extraction module and the frequency domain feature extraction module are both connected to the input of the first fusion enhancement module. The outputs of the frequency domain feature extraction module and the temporal feature extraction module are both connected to the input of the second fusion enhancement module. The outputs of the first fusion enhancement module and the second fusion enhancement module are both connected to the input of the third fusion enhancement module. The output of the third fusion enhancement module is connected to the input of the detection head. The step of inputting the training infrared video in the training dataset into the dense moving infrared small target detection model includes: using the backbone network to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map group, wherein the infrared feature map group includes N infrared feature maps; using... The spatial feature extraction module extracts spatial features from the infrared feature map group to obtain a spatial feature map; the frequency domain feature extraction module extracts frequency features from each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps; the temporal feature extraction module extracts temporal features from the infrared feature map group to obtain a temporal feature map; the first fusion enhancement module generates a spatial-frequency fusion feature map based on the spatial feature map and the frequency feature map group; the second fusion enhancement module generates a temporal-frequency fusion feature map based on the frequency feature map group and the temporal feature map; the third fusion enhancement module generates a spatial-temporal-frequency fusion feature map based on the spatial-frequency feature map and the temporal-frequency feature map; and the detection head generates the prediction result based on the spatial-temporal-frequency fusion feature map.

[0010] According to an embodiment of the present invention, the backbone network includes a first branch, a second branch, and a residual connection layer. Both the first branch and the second branch are connected to the residual connection layer. The first branch includes a first convolutional unit, a second convolutional layer, a dilated convolutional unit, a spatial attention layer, and a channel attention layer connected in sequence. The first convolutional unit includes eight parallel first convolutional layers, with the directions of convolution corresponding to the convolution kernels being horizontal to the right, horizontal to the left, vertically upward, vertically downward, upper right, upper left, lower right, and lower left, respectively. The dilated convolutional unit includes four parallel dilated convolutional layers. The second branch includes a third convolutional layer and a fourth convolutional layer connected in sequence. The feature extraction of each infrared video frame in the training infrared video using the backbone network includes: calculating the grayscale difference between the center pixel and the boundary pixel of the infrared video frame in the corresponding direction using the eight parallel first convolutional layers in the first convolutional unit, obtaining eight first feature maps; and multiplying the two feature maps with opposite convolution directions element-wise among the eight first feature maps to obtain four difference feature maps. The four differential feature maps are summed element-wise pairwise along a vertical reverse direction to obtain two second feature maps. The first convolutional layer is used to extract features from the first stitched feature map to obtain a primary feature map, wherein the first stitched feature map is obtained by stitching together two second feature maps. The primary feature map and the third feature map are subjected to residual concatenation processing to obtain a fourth feature map, wherein the third feature map is extracted from the infrared video frame by the third convolutional layer. The fourth feature map is subjected to dilated convolution processing by four parallel dilated convolutional layers in the dilated convolutional unit to obtain four dilated feature maps. The four dilated feature maps are stitched together to obtain a secondary feature map. The spatial attention layer is used to extract features from the secondary feature map to obtain a fifth feature map. The channel attention layer is used to extract features from the fifth feature map to obtain a sixth feature map. The sixth and seventh feature maps are subjected to residual concatenation processing by the residual concatenation layer to obtain an infrared feature map, wherein the seventh feature map is extracted from the third feature map by the fourth convolutional layer.

[0011] According to an embodiment of the present invention, the spatial feature extraction module includes a third branch, a fourth branch, and a memory enhancement submodule. The third branch and the fourth branch are both connected to the memory enhancement submodule. The third branch includes a fifth convolutional layer, a sixth convolutional layer, a spatial enhancement submodule, and a first embedding layer connected sequentially. The fourth branch includes a second embedding layer. The spatial feature extraction of the infrared feature map group using the spatial feature extraction module includes: stitching the first N-1 infrared feature maps in the infrared feature map group to obtain a second stitched feature map; extracting features from the second stitched feature map using the fifth convolutional layer to obtain an eighth feature map; multiplying the eighth feature map element-wise with the Nth video feature map in the infrared feature map group and then stitching it with the Nth infrared feature map in the infrared feature map group to obtain a third stitched feature map; extracting features from the third stitched feature map using the sixth convolutional layer to obtain an initial feature map; and using the spatial enhancement submodule to extract features from the third stitched feature map. The initial feature map undergoes spatial enhancement processing to obtain a spatially enhanced feature map. The spatial enhancement submodule includes four sub-branches and a seventh convolutional layer. Each of the four sub-branches processes the initial feature map. The processing results of sub-branch three and sub-branch four are residually concatenated. The seventh convolutional layer then extracts features from the residual concatenation results. The extracted results are then multiplied element-wise with the summed results to obtain the spatially enhanced feature map. The summed results are the element-wise sums of the processing results of sub-branch one and sub-branch two. Sub-branch one includes a first max-pooling layer and a first fully connected layer connected in sequence; sub-branch two includes a first average pooling layer and a second fully connected layer connected in sequence; sub-branch three includes a channel max-pooling layer; and sub-branch four includes a channel average pooling layer. The spatially enhanced feature map and the initial feature map are then element-wise added to obtain local-global spatial features. The first embedding layer generates a first key-value pair based on the local-global spatial features. The first key-value pair includes a key... Sum Using the memory enhancement submodule, a spatial feature map is obtained based on the first key-value pair and the second key-value pair. The second key-value pair is generated by the second embedding layer based on the Nth video feature map in the feature map group. The second key-value pair includes a key... Sum The memory enhancement submodule utilizes the softmax activation function to activate the key. s and keys The product of is normalized, and the similarity obtained after normalization is compared with the value. Multiplication and value Residual connection processing is performed, and the results of the residual connection are processed by the matching layer to obtain a spatial feature map.

[0012] According to an embodiment of the present invention, the step of extracting frequency features from each infrared feature map in the infrared feature map group using the frequency domain feature extraction module includes: converting the features of the infrared feature map to the frequency domain using Fourier transform to obtain the frequency domain information of the infrared feature map, wherein the frequency domain information includes amplitude. and phase Using the first convolutional group, based on the amplitude and amplitude mask The product of these factors yields the amplitude features, wherein the amplitude mask... The amplitude is affected by the sequentially connected eighth and ninth convolutional layers. and the phase The splicing result is extracted; using the second convolutional group, based on the phase... and phase mask The product of these terms yields the phase feature, wherein the phase mask... The amplitude is affected by the sequentially connected eighth and tenth convolutional layers. and the phase The splicing result is extracted; using inverse Fourier transform, the infrared feature map in the frequency domain is transformed to the original domain based on the amplitude feature and the phase feature to obtain the frequency feature map; wherein, the convolutional groups in the first and second convolutional groups include an eleventh convolutional layer, a second max pooling layer, a second average pooling layer, a first depthwise separable convolutional layer, and a first pointwise convolutional layer. The eleventh convolutional layer is used to extract features from the input product, and the second max pooling layer and the second average pooling layer are used to pool the feature extraction results respectively. After concatenating the residuals of the max pooling results and the average pooling results, the first depthwise separable convolutional layer and the first pointwise convolutional layer are used sequentially to process the residual concatenation results to obtain the output features. The input product includes amplitude. and amplitude mask product, phase and phase mask The product of the amplitude and phase features is the output feature.

[0013] According to one embodiment of the present invention, the temporal feature extraction module includes a coarse motion estimation branch and a feature enhancement branch connected in sequence. The coarse motion estimation branch includes a feature stitching layer, a second convolutional unit, and a twelfth convolutional layer connected in sequence. The feature enhancement branch includes a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a softmax activation function, a nineteenth convolutional layer, a twentieth convolutional layer, a flattening layer, a deformable attention submodule, a twenty-first convolutional layer, and a 3D convolutional layer. The temporal feature extraction using the temporal feature extraction module to extract temporal features from the infrared feature map group includes: using the feature stitching layer to perform forward and backward stitching on adjacent infrared feature maps in the infrared feature map group, respectively. A first set of inter-frame relationship maps and a second set of inter-frame relationship maps are obtained, each consisting of four inter-frame relationship maps. The second convolutional unit is used to capture the inter-frame relationships of common points in the first and second sets of inter-frame relationship maps at different motion velocities, resulting in the first and second inter-frame relationships. The second convolutional unit comprises a thirteenth convolutional layer, a first ReLU activation function, a fourteenth convolutional layer, a second ReLU activation function, a fifteenth convolutional layer, and a third ReLU activation function, connected sequentially. The twelfth convolutional layer is used to extract features from the first and second inter-frame relationships, yielding four forward optical flow maps. and 4 backflow images The forward optical flow Towards After distortion, with the forward light flow The summation yields the forward optical flow across two frames. The backward optical flow Towards After distortion, with backward light flow Adding them together gives the backward optical flow across two frames. ; Each infrared feature map and forward optical flow in the infrared feature map group Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Align with the middle frame to obtain alignment features; combine the infrared feature map group, the alignment features, and the forward optical flow. Forward optical flow across two frames Backward optical flow Backward optical flow across two frames The connection processing is performed to obtain connection features; the connection features are then extracted using the sixteenth convolutional layer to obtain the ninth feature map; the deformable attention submodule is used to process the sampling point offset, standardized attention weights, and amortized features to obtain refined features, wherein the sampling point offset is obtained based on the tenth feature map and the offset, wherein the tenth feature map is extracted from the ninth feature map by the seventeenth convolutional layer, and the offset is based on the forward optical flow. Forward optical flow across two frames Backward optical flow Backward optical flow across two frames The calculation shows that the standardized attention weights are obtained by processing the ninth feature map through a sequentially connected tenth convolutional layer and a softmax activation function; the amortized features are obtained by processing the infrared feature map group through a sequentially connected nineteenth convolutional layer, the twentieth convolutional layer, and the amortized layer; the refined features are extracted using the twenty-first convolutional layer to obtain the eleventh feature map; the features after residual concatenation between the infrared feature map group and the eleventh feature map are processed using the 3D convolutional layer to obtain N twelfth feature maps; and residual concatenation is performed on the N twelfth feature maps to obtain the temporal feature map.

[0014] According to an embodiment of the present invention, the first fusion enhancement module, the second fusion enhancement module, and the third fusion enhancement module all include a fusion enhancement module. The fusion enhancement module includes a second depthwise separable convolutional layer, a second pointwise convolutional layer, a collaborative attention submodule, a gating submodule, and a batch normalization layer connected in sequence. The fusion enhancement module is used to receive a first input feature map and a second input feature map, and output a fused feature map. Specifically, the fusion enhancement module is used to: extract features from the fourth concatenated feature using the second depthwise separable convolutional layer and the second pointwise convolutional layer connected in sequence, to obtain a thirteenth feature map, wherein the fourth concatenated feature is derived from the first... The input feature map and the second input feature map are concatenated along the channel dimension to obtain the fourteenth feature map; the internal features of the thirteenth feature map are fused using the collaborative attention submodule to obtain the fourteenth feature map; the thirteenth feature map and the fourteenth feature map are subjected to residual connection processing to obtain the fifteenth feature map; the weight value gate of the fifteenth feature map is generated using the gating submodule; the first product and the second product are added to obtain the sixteenth feature map, wherein the first product is the product of the fifteenth feature map and the weight value gate, and the second product is the product of (1-gate) and the fourth concatenated feature map; the sixteenth feature map is processed using the batch normalization layer to obtain the fused feature map.

[0015] According to an embodiment of the present invention, the detection model loss function for:

[0016] ;

[0017] in, This represents the loss function of the detection model. This represents the boundary regression loss function. Represents the classification loss function. Represents the target probability loss function. This represents the forward optical flow loss function. This represents the backward optical flow loss function. This represents the hyperparameter of the boundary regression loss. This represents the hyperparameter of the classification loss. This represents the target probability loss hyperparameter. This represents the hyperparameter of optical flow loss;

[0018] The boundary regression loss function for:

[0019] ;

[0020] in, This represents the boundary regression loss function. This represents the intersection-union ratio hyperparameter. This represents the intersection-union ratio loss function. This represents the normalized Gaussian distance hyperparameter. This represents the normalized Gaussian distance loss function. Indicates the predicted bounding box. Represents the true bounding box. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This indicates a Gaussian distribution of the predicted bounding box. This represents the Gaussian distribution of the true bounding box. Represents a constant;

[0021] The classification loss function for:

[0022] ;

[0023] in, Let i represent the classification loss function, i represent the number of training infrared videos in the training dataset, and c represent the number of categories. For accurate classification, For predictive classification;

[0024] The target probability loss function for:

[0025] ;

[0026] in, Let i represent the target probability loss function, and let i represent the number of training infrared videos in the training dataset. For true probability, To predict probabilities;

[0027] The forward optical flow loss function for:

[0028] ;

[0029] in, This represents the forward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the forward-facing real image. Represents the forward-estimated image;

[0030] The backward optical flow loss function for:

[0031] ;

[0032] in, This represents the backward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the backward real image, This represents the backward estimated image.

[0033] To achieve the above objectives, a second aspect of the present invention provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the above-described method for detecting densely moving infrared small targets based on three-domain feature learning.

[0034] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0035] Figure 1 This is a flowchart of a densely moving infrared small target detection method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of a densely moving infrared small target detection model according to an embodiment of the present invention;

[0037] Figure 3 This is a flowchart of the training process for a densely moving infrared small target detection model according to an embodiment of the present invention;

[0038] Figure 4 This is a flowchart illustrating how a densely moving infrared small target detection model of an embodiment of the present invention processes training infrared video;

[0039] Figure 5 This is a schematic diagram of the processing flow of the backbone network according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of each convolution kernel in the first convolution unit according to an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the processing flow of the spatial feature extraction module according to an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the processing flow of the frequency domain feature extraction module according to an embodiment of the present invention;

[0043] Figure 9 This is a schematic diagram of the processing flow of the coarse motion estimation branch according to an embodiment of the present invention;

[0044] Figure 10 This is a schematic diagram of the processing flow for generating forward and backward optical flow across two frames according to an embodiment of the present invention.

[0045] Figure 11 This is a schematic diagram of the processing flow of the feature enhancement branch according to an embodiment of the present invention;

[0046] Figure 12 This is a schematic diagram of the processing flow of the fusion enhancement module according to an embodiment of the present invention;

[0047] Figure 13 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0049] In recent years, artificial intelligence technology has developed rapidly, and data-driven methods, primarily deep learning, have become the mainstream approach. However, most research focuses on single frames. In reality, compared to single-frame image methods that rely on the differences between local targets and the background, multi-frame video-based small target detection can extract more information from the spatiotemporal domain and has broader application prospects. Furthermore, most current infrared small target detection ISTD methods focus on sparse (i.e., only one or a few) target detection. In fact, dense targets are ubiquitous in real-world infrared detection scenarios, and accurately identifying multiple small targets remains a challenge.

[0050] To address the aforementioned problems, embodiments of the present invention provide a method and electronic device for detecting densely moving infrared small targets based on three-domain feature learning. The following will describe in detail the method and electronic device for detecting densely moving infrared small targets based on three-domain feature learning according to embodiments of the present invention, in conjunction with the accompanying drawings and specific implementation methods.

[0051] Figure 1 This is a flowchart of a densely moving infrared small target detection method according to an embodiment of the present invention. Figure 1 As shown, a method for detecting densely moving infrared small targets based on three-domain feature learning may include:

[0052] S101, acquire the infrared video to be tested, wherein the infrared video to be tested includes N consecutive infrared video frames, where N is a positive integer;

[0053] S102, the infrared video to be tested is input into the pre-trained dense moving infrared small target detection model to obtain the detection results, which include the detection bounding box of the infrared small target, the Gaussian distribution of the detection bounding box, the detection classification, and the detection probability.

[0054] Specifically, continuous frame video of densely moving small infrared targets to be detected is acquired. It should be noted that the number of infrared video frames in the acquired continuous frame video can be greater than N. When the number of infrared video frames in the acquired continuous frame video is greater than N, the acquired continuous frame video is processed to generate multiple videos containing N consecutive infrared video frames. This generated video is denoted as the infrared video to be tested. The infrared video frames in the infrared video to be tested contain densely moving small infrared targets.

[0055] The infrared video to be tested is input into a pre-trained dense moving infrared small target detection model. The pre-trained dense moving infrared small target detection model detects infrared small targets in the infrared video to be tested and outputs the detection results.

[0056] It should be noted that densely packed moving infrared small targets are an important scenario for infrared small target detection. To address the shortcomings of traditional algorithms in feature representation and accuracy, this invention designs a three-domain deep learning network. The model trained by this network (densely packed moving infrared small target detection model) can accurately detect densely packed moving infrared small targets, with improvements in detection accuracy, robustness, and scene adaptability.

[0057] Specifically, such as Figure 2 As shown, the densely moving infrared small target detection model in this embodiment of the invention may include a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network, and a detection head. The input end of the backbone network is used to input infrared video, and the output end of the backbone network is connected to the input ends of the spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module, respectively. The output ends of the spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module are all connected to the input end of the fusion enhancement network. The output end of the fusion enhancement network is connected to the input end of the detection head, and the detection head is used to output the detection results.

[0058] In complex backgrounds, certain targets often do not exhibit high correlation in consecutive frames. Information from different domains is complementary, and introducing more information domains helps to more comprehensively characterize the features of small infrared targets. To extract more features and improve recognition accuracy, the densely moving infrared small target detection model in this embodiment employs three-domain feature learning, fusing features extracted from the temporal, spatial, and frequency domains respectively. Specifically, the densely moving infrared small target detection model extracts target features in the three domains—temporal, spatial, and frequency—through three branches, and then fuses these features to improve the accuracy of the detection model for densely moving infrared small targets. This model has broad application prospects in scenarios such as night surveillance, search and rescue, and military applications.

[0059] In one embodiment of the present invention, such as Figure 3 As shown, the training process of the densely moving infrared small target detection model may include:

[0060] S201, Obtain the training dataset, which includes training infrared videos and the labeling results corresponding to each infrared video frame in the training infrared videos. The training infrared videos include N consecutive infrared video frames, and the labeling results include the true bounding boxes of infrared small targets, the Gaussian distribution of the true bounding boxes, the true classification, and the true probability.

[0061] S202, input the training infrared videos in the training dataset into the dense motion infrared small target detection model to obtain the prediction results, which include the predicted bounding box of the infrared small target, the Gaussian distribution of the predicted bounding box, the predicted classification, and the predicted probability.

[0062] S203, construct a boundary regression loss function based on the predicted bounding box, the Gaussian distribution of the predicted bounding box, the true bounding box, and the Gaussian distribution of the true bounding box; construct a classification loss function based on the predicted classification and the true classification; construct a target probability loss function based on the true probability and the predicted probability; construct a forward optical flow loss function and a backward optical flow loss function based on the training infrared video of the input detection model; and construct a detection model loss function based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function, and the backward optical flow loss function.

[0063] S204. Train the dense moving infrared small target detection model according to the detection model loss function to obtain the trained dense moving infrared small target detection model.

[0064] Specifically, multiple long videos of densely moving small infrared targets are acquired and processed to generate a large set of N consecutive infrared video frames, which is designated as the training infrared video set. The small infrared targets on each frame of the training infrared video set are labeled. The labeling includes at least the ground truth bounding box, the Gaussian distribution of the ground truth bounding box, the true classification, and the true probability of the small infrared target.

[0065] As a specific embodiment, when N is 5, for a set of continuous frame videos, continuous infrared video frames from time window T=5 are taken as training infrared videos, and these training infrared videos are used as inputs to train a densely moving infrared small target detection model.

[0066] After training the infrared video input dense motion infrared small target detection model, the dense motion infrared small target detection model extracts features from each infrared video frame in the training infrared video, fuses and detects the extracted features, and outputs the predicted bounding box, the Gaussian distribution of the predicted bounding box, the predicted classification, and the predicted probability of the infrared small target.

[0067] A boundary regression loss function is constructed based on the predicted bounding box, its Gaussian distribution, the ground truth bounding box, and its Gaussian distribution. A classification loss function is constructed based on the predicted and ground truth classifications. A target probability loss function is constructed based on the ground truth and predicted probabilities. Forward and backward optical flow loss functions are constructed based on the infrared video used to train the detection model. A detection model loss function is then constructed based on these boundary regression, classification, target probability, forward, and backward optical flow loss functions. The dense moving infrared small target detection model is trained using these detection model loss functions to obtain a trained dense moving infrared small target detection model.

[0068] In one embodiment of the present invention, such as Figure 4As shown, the fusion enhancement network may include a first fusion enhancement module, a second fusion enhancement module, and a third fusion enhancement module. The outputs of the spatial feature extraction module and the frequency domain feature extraction module are both connected to the input of the first fusion enhancement module. The outputs of the frequency domain feature extraction module and the temporal feature extraction module are both connected to the input of the second fusion enhancement module. The outputs of the first and second fusion enhancement modules are both connected to the input of the third fusion enhancement module. The output of the third fusion enhancement module is connected to the input of the detection head. The training infrared video from the training dataset is input into the dense moving infrared small target detection model, which may include:

[0069] S301, the backbone network is used to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map set, which includes N infrared feature maps. ;

[0070] S302, using the spatial feature extraction module to extract spatial features from the infrared feature map group, to obtain the spatial feature map. ;

[0071] S303, the frequency feature extraction module is used to extract the frequency features of each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps. ;

[0072] S304, The time feature extraction module is used to extract time features from the infrared feature map group to obtain the time feature map. ;

[0073] S305, the first fusion enhancement module generates a spatial-frequency fusion feature map based on the spatial feature map and the frequency feature map group, the second fusion enhancement module generates a time-frequency fusion feature map based on the frequency feature map group and the time feature map, and the third fusion enhancement module generates a spatial-time-frequency fusion feature map based on the spatial-frequency feature map and the time-frequency feature map;

[0074] S306 uses a detection head to generate prediction results based on a spatial-temporal-frequency fusion feature map.

[0075] Specifically, in this embodiment of the invention, the backbone network is a differential amplification network specifically designed for detecting small targets. Each infrared video frame from the training infrared video is input into a shared-weight feature extractor (backbone network) to extract features, resulting in an infrared feature map set.

[0076] The spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module are three parallel branches. They respectively use spatial, frequency, and temporal three-domain feature learning strategies to extract features from the infrared feature map group, resulting in spatial feature map Fs, frequency feature map group, and temporal feature map Ft.

[0077] The first fusion enhancement module fuses the spatial feature map Fs and the frequency feature map group to obtain a spatial-frequency fusion feature map. The second fusion enhancement module fuses the frequency feature map group and the temporal feature map Ft to obtain a temporal-frequency fusion feature map. The third fusion enhancement module fuses the spatial-frequency feature map and the temporal-frequency feature map to generate a spatial-temporal-frequency fusion feature map. The fully interacting spatial-temporal-frequency fusion feature map is then passed to the detection head to obtain the final prediction result.

[0078] The first, second, and third fusion enhancement modules in this embodiment of the invention all include a fusion enhancement module (FEM). The fusion enhancement module is used to eliminate potential feature mismatches between different domains, while fusing and enhancing features from different domains.

[0079] In one embodiment of the present invention, such as Figure 5 As shown, the backbone network may include a first branch, a second branch, and a residual connection layer. Both the first and second branches are connected to the residual connection layer. The first branch includes a first convolutional unit, a second convolutional layer, a dilated convolutional unit, a spatial attention layer, and a channel attention layer connected in sequence. The first convolutional unit includes eight parallel first convolutional layers, with the directions of convolution corresponding to the convolution kernels being horizontally right, horizontally left, vertically up, vertically down, upper right, upper left, lower right, and lower left, respectively. The dilated convolutional unit includes four parallel dilated convolutional layers. The second branch includes a third and a fourth convolutional layer connected in sequence. Feature extraction from each infrared video frame in the training infrared video using the backbone network may include:

[0080] The grayscale difference between the center pixel and the boundary pixel in the infrared video frame in the corresponding direction is calculated by using the eight parallel first convolutional layers in the first convolutional unit, resulting in eight first feature maps.

[0081] Multiply the two feature maps with opposite convolution directions from the eight first feature maps element by element to obtain four difference feature maps;

[0082] The four difference feature maps are added element-wise pairwise along a vertical reverse direction to obtain two second feature maps;

[0083] The first concatenated feature map is obtained by using the second convolutional layer to extract features from the first concatenated feature map. The first concatenated feature map is obtained by concatenating two second feature maps.

[0084] The primary feature map and the third feature map are subjected to residual connection processing to obtain the fourth feature map. The third feature map is extracted from the infrared video frame by the third convolutional layer.

[0085] The fourth feature map is processed by dilation convolution using four parallel dilation convolution layers in the dilation convolution unit, resulting in four dilation feature maps.

[0086] The four expanded feature maps are stitched together to obtain a secondary feature map.

[0087] The spatial attention layer is used to extract features from the secondary feature map to obtain the fifth feature map;

[0088] The sixth feature map is obtained by extracting features using the fifth feature map of the channel attention layer;

[0089] The sixth and seventh feature maps are processed by residual connection layer to obtain infrared feature maps. The seventh feature map is extracted from the third feature map by the fourth convolutional layer.

[0090] In this embodiment of the invention, the eight preset directions are defined with the center of the convolutional kernel pointing horizontally to the right as the 0° reference. Each direction is divided into 45° intervals in a clockwise (or counterclockwise) direction. The angles of the eight directions are as follows: 0°: Horizontal to the right (along the positive x-axis). 45°: Tilted to the upper right (diagonal direction). 90°: Vertically upward (along the positive y-axis). 135°: Tilted to the upper left (diagonal direction). 180°: Horizontal to the left (along the negative x-axis). 225°: Tilted to the lower left (diagonal direction). 270°: Vertically downward (along the negative y-axis). 315°: Tilted to the lower right (diagonal direction). The parameters of the convolutional kernels corresponding to the eight first convolutional layers are described below. Figure 6 .

[0091] Specifically, the first branch and the second branch process the infrared video frames respectively.

[0092] In the second branch, the third convolutional layer extracts features from the infrared video frame to obtain the third feature map, and the fourth convolutional layer performs feature extraction processing on the third feature map to obtain the seventh feature map.

[0093] In the first branch, eight (3×3) first convolutional layers with specific weights are convolved in the eight preset directions to calculate the grayscale difference between the center and boundary pixels of the infrared video frame in the corresponding directions, resulting in eight first feature maps. Next, the differences are amplified by element-wise multiplication of two feature maps with opposite convolution directions, resulting in four difference feature maps. These four difference feature maps are then added pairwise along the vertical direction to obtain two second feature maps. The two second feature maps are then concatenated and passed through a (3×3) second convolutional layer to obtain the primary feature map. .

[0094] Initial feature map The fourth feature map is obtained by residual concatenating it with the third feature map. This fourth feature map is then processed through four parallel dilated convolutions and directly concatenated to obtain the secondary feature map. Finally, the secondary feature map is processed by a spatial attention mechanism to obtain the fifth feature map, and the fifth feature map is processed by a channel attention mechanism to obtain the sixth feature map. The sixth and seventh feature maps are then residually concatenated to obtain the infrared feature map. .

[0095] In this embodiment, the four parallel dilated convolutional layers in the dilated convolutional unit are: a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer, and a fourth dilated convolutional layer, respectively. See [link to documentation]. Figure 5 The kernel sizes for the first, second, third, and fourth dilated convolutional layers are 1, 3, 5, and 7, respectively.

[0096] In one embodiment of the present invention, such as Figure 7 As shown, the spatial feature extraction module may include a third branch, a fourth branch, and a memory enhancement submodule. Both the third and fourth branches are connected to the memory enhancement submodule. The third branch may include a fifth convolutional layer, a sixth convolutional layer, the spatial enhancement submodule, and a first embedding layer connected sequentially. The fourth branch may include a second embedding layer. The spatial feature extraction module performs spatial feature extraction on the infrared feature map set, which may include:

[0097] The first N-1 infrared feature images in the infrared feature image group are stitched together to obtain the second stitched feature image;

[0098] The fifth convolutional layer is used to extract features from the second concatenated feature map to obtain the eighth feature map;

[0099] The eighth feature map is multiplied element-wise with the Nth video feature map in the infrared feature map group, and then concatenated with the Nth infrared feature map in the infrared feature map group to obtain the third concatenated feature map.

[0100] The sixth convolutional layer is used to extract features from the third concatenated feature map to obtain the initial feature map;

[0101] The initial feature map is spatially augmented using a spatial augmentation submodule to obtain a spatially augmented feature map. The spatial augmentation submodule includes four sub-branches and a seventh convolutional layer. The four sub-branches perform feature processing on the initial feature map respectively. The processing results of sub-branch three and sub-branch four are residually connected. The seventh convolutional layer is used to extract features from the residual connection results. The extracted results are then multiplied element-wise with the summed results to obtain the spatially augmented feature map. The summed results are the element-wise summed results of the processing results of sub-branch one and sub-branch two. Sub-branch one may include a first max pooling layer and a first fully connected layer connected in sequence. Sub-branch two may include a first average pooling layer and a second fully connected layer connected in sequence. Sub-branch three may include a channel max pooling layer. Sub-branch four may include a channel average pooling layer.

[0102] Element-wise addition of the spatial augmented feature map and the initial feature map yields the local-global spatial features;

[0103] The first key-value pair is generated based on local-global spatial features using the first embedding layer, wherein the first key-value pair includes a key. Sum ;

[0104] Using the memory enhancement submodule, a spatial feature map is obtained based on the first key-value pair and the second key-value pair. The second key-value pair is generated by the second embedding layer based on the Nth video feature map in the feature map group. The second key-value pair includes a key... Sum Among them, the memory enhancement submodule uses the softmax activation function to target the key. s and keys The product of is normalized, and the similarity obtained after normalization is compared with the value. Multiplication and value Residual connection processing is performed, and the results of the residual connection are processed by the matching layer to obtain a spatial feature map.

[0105] It should be noted that spatial relationship modeling is crucial for accurately predicting and tracking target locations. To take into account the global context information of keyframes and the spatial dependencies between adjacent frames as much as possible, this embodiment of the invention includes a spatial feature extraction module.

[0106] For example, when N is 5, in the spatial feature extraction module, the first four frames in the infrared feature map set are used as reference frames, and the last frame is used as a keyframe. The first four keyframes are concatenated and passed through a (3×3) convolutional layer (the fifth convolutional layer). Then, they are multiplied element-wise with the last keyframe and concatenated. After passing through another (3×3) convolutional layer (the sixth convolutional layer), an initial feature map is obtained. Next, the initial feature map is processed by the spatial enhancement submodule to extract the internal correlation of the features. The resulting spatially enhanced feature map is added to the initial feature map to obtain the local-global spatial features.

[0107] The spatial enhancement submodule in this embodiment of the invention includes four sub-branches and a seventh convolutional layer, see [link / reference]. Figure 7 The four sub-branches perform feature processing on the initial feature map respectively. After residual concatenation of the processing results of sub-branch three and sub-branch four, the seventh convolutional layer is used to extract features from the residual concatenation results. The extracted results are then residual concatenated with the summed results to obtain the spatially enhanced feature map. The summed results are the element-wise summed results of the processing results of sub-branch one and sub-branch two. Sub-branch one may include a first max pooling layer and a first fully connected layer connected in sequence. Sub-branch two may include a first average pooling layer and a second fully connected layer connected in sequence. Sub-branch three may include a channel max pooling layer. Sub-branch four may include a channel average pooling layer.

[0108] The initial feature map is processed using four parallel sub-branches. The initial feature map from sub-branch one is processed sequentially through a global max pooling layer (first max pooling layer) and a fully connected layer (first fully connected layer). The initial feature map from sub-branch two is processed sequentially through a global average pooling layer (first average pooling layer) and a fully connected layer (second fully connected layer). The processing results from sub-branch one and sub-branch two are summed pixel-wise to obtain the channel attention feature map. The initial feature map from sub-branch three is processed through a channel max pooling layer, taking the maximum value along the channel dimension. The initial feature map from sub-branch four is processed through a channel average pooling layer, taking the average value along the channel dimension. The processing results from sub-branch three and sub-branch four are concatenated and then processed through a 7x7 convolution (seventh convolutional layer) to obtain the spatial attention feature map. Finally, the channel attention feature map and the spatial attention feature map are multiplied element-wise to obtain the spatial enhancement feature map.

[0109] Local-global spatial features are stored in memory, keyframe features are used as queries, and two different embedding layers (first embedding layer and second embedding layer) are used to generate corresponding key-value pairs. , , , The Memory Enhancement Unit (MEU) is used to extend the features of small infrared targets. First, the key... s and keys After multiplication, the similarity is calculated by normalizing the results using the Softmax activation function. Then, the similarity is... Multiply to utilize the value Query regions related to the features of small infrared targets. Compare similarity with values. The product and value After the connection is established, a matching layer is used to establish the relationship between the keyframes and the reference frames, resulting in a spatial feature map. .

[0110] In one embodiment of the present invention, such as Figure 8 As shown, the frequency feature extraction module can be used to extract frequency features from each infrared feature map in the infrared feature map group, which may include:

[0111] Fourier transform is used to convert the features of the infrared feature map to the frequency domain, obtaining the frequency domain information of the infrared feature map, which includes the amplitude. and phase ;

[0112] Using the first convolutional group, based on the amplitude and amplitude mask The product of these factors yields the amplitude features, where the amplitude mask is... The amplitude is determined by the sequentially connected eighth and ninth convolutional layers. and phase The splicing result is extracted;

[0113] Using the second convolutional group, based on phase and phase mask The product of these terms yields the phase features, where the phase mask is... The amplitude is determined by the sequentially connected eighth and tenth convolutional layers. and phase The splicing result is extracted;

[0114] By employing inverse Fourier transform, based on amplitude and phase characteristics, the infrared feature map in the frequency domain is transformed back to the original domain to obtain the frequency feature map. The first and second convolutional groups each include an eleventh convolutional layer, a second max-pooling layer, a second average-pooling layer, a first depthwise separable convolutional layer, and a first pointwise convolutional layer. The eleventh convolutional layer extracts features from the input product. The second max-pooling layer and the second average-pooling layer then pool the extracted features. Residual concatenation of the max-pooling and average-pooling results is performed, followed by processing the residual concatenation results using the first depthwise separable convolutional layer and the first pointwise convolutional layer to obtain the output features. The input product includes amplitude. and amplitude mask product, phase and phase mask The product of these features results in output characteristics including amplitude and phase features.

[0115] It should be noted that frequency domain features provide a comprehensive understanding of small infrared targets at different frequencies, while also reducing image noise and interference.

[0116] Specifically, Fourier transform is used to convert each frame of infrared feature map to the frequency domain, and the frequency domain information is further processed. Further decomposed into amplitude and phase For amplitude With phase The data is spliced ​​together and processed through two convolutional layers to obtain the amplitude mask. Amplitude mask With amplitude Multiplication guides the input amplitude, and the first convolutional group guides the extraction of amplitude features, yielding the amplitude features. For phase, a similar process is followed to obtain phase features. An inverse Fourier transform is then used to transform the processed frequency domain features (amplitude and phase features) back to their original domain, obtaining the frequency domain features of the infrared feature map in the frequency domain. .

[0117] In this embodiment of the invention, the convolutional groups (CGs) in the first and second convolutional groups process the input product as follows: the input product is first processed by a normal (3×3) convolution (eleventh convolutional layer), then by max pooling (second max pooling layer) and average pooling (second average pooling layer) to extract channel information. The outputs of the second max pooling layer and the second average pooling layer are residually concatenated. Finally, a (3×3) depthwise convolution (first depthwise separable convolutional layer) is used to independently learn the spatial context information on each channel, and cross-channel mixing of a (1×1) pointwise convolution (first pointwise convolutional layer) is used to aggregate pixels. The output of the first pointwise convolutional layer is multiplied pixel by pixel with the input product to obtain the features output by the convolutional group.

[0118] In one embodiment of the present invention, such as Figure 9-11 As shown, the temporal feature extraction module may include a coarse motion estimation branch and a feature enhancement branch connected in sequence. The coarse motion estimation branch may include a feature splicing layer, a second convolutional unit, and a twelfth convolutional layer connected in sequence. The feature enhancement branch may include a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a softmax activation function, a nineteenth convolutional layer, a twentieth convolutional layer, a flattening layer, a deformable attention submodule, a twenty-first convolutional layer, and a 3D convolutional layer. The temporal feature extraction module can be used to extract temporal features from the infrared feature map set, including:

[0119] The feature stitching layer is used to stitch two adjacent infrared feature maps in the infrared feature map group forward and backward respectively to obtain the first set of inter-frame relationship maps and the second set of inter-frame relationship maps. The first set of inter-frame relationship maps and the second set of inter-frame relationship maps each include four inter-frame relationship maps.

[0120] The second convolutional unit is used to capture the inter-frame relationships of the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs at different motion speeds, so as to obtain the first inter-frame relationship and the second inter-frame relationship. The second convolutional unit includes a thirteenth convolutional layer, a first ReLU activation function, a fourteenth convolutional layer, a second ReLU activation function, a fifteenth convolutional layer and a third ReLU activation function connected in sequence.

[0121] The twelfth convolutional layer was used to extract features from the first and second inter-frame relationships, respectively, to obtain four images of forward optical flow. and 4 backflow images ;

[0122] Forward optical flow Towards After distortion, with the forward light flow The summation yields the forward optical flow across two frames. ;

[0123] Backward optical flow Towards After distortion, with backward light flow Adding them together gives the backward optical flow across two frames. ;

[0124] Each infrared feature map and forward optical flow in the infrared feature map group Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Align with the middle frame to obtain the alignment feature;

[0125] Infrared feature map group, aligned features, forward optical flow Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Perform connection processing to obtain connection characteristics;

[0126] The connection features are extracted using the sixteenth convolutional layer to obtain the ninth feature map;

[0127] A deformable attention submodule is used to process the sampling point offset, standardized attention weights, and amortized features to obtain refined features. The sampling point offset is obtained based on the tenth feature map and the offset itself. The tenth feature map is extracted from the ninth feature map by the seventeenth convolutional layer, and the offset is based on the forward optical flow. Forward optical flow across two frames Backward optical flow Backward optical flow across two frames The calculation shows that the standardized attention weights are obtained by processing the ninth feature map through a series of connected tenth convolutional layers and softmax activation functions, and the amortized features are obtained by processing the infrared feature map group through a series of connected nineteenth convolutional layers, twentieth convolutional layers and amortized layers.

[0128] The 21st convolutional layer is used to extract refined features, resulting in the 11th feature map;

[0129] The features after residual connection between the infrared feature map group and the eleventh feature map are processed using a 3D convolutional layer to obtain N twelfth feature maps.

[0130] Residual connections are performed on N twelfth feature maps to obtain time feature maps.

[0131] In this embodiment of the invention, the temporal feature extraction module comprises two main parts: a coarse motion estimation module and a feature enhancement module. The coarse motion estimation module is responsible for extracting coarse inter-frame motion information of the target, generating forward and backward optical flows. Then, the feature enhancement module combines the forward and backward optical flows with deformable convolution to extract temporal features.

[0132] Coarse motion estimation: In order to extract inter-frame feature differences, infrared features from 5 adjacent frames... Figure 1 It is used as input. The feature concatenation layer is utilized. Figure 9 (Not shown in the image) Five infrared feature images are stitched together along the channel dimension using both forward and backward stitching methods to obtain two sets of inter-frame relationship maps (the first set and the second set), each consisting of four images. The second convolutional unit captures the inter-frame relationships of points from the same source in both the first and second sets at different motion speeds. Specifically, a thirteenth convolutional layer with a dilation rate of 3, a fourteenth convolutional layer with a dilation rate of 5, and a fifteenth convolutional layer with a dilation rate of 1 are used to capture the inter-frame relationships of points from the same source at different motion speeds. Finally, four forward optical flow images are obtained through a (3×3) convolutional layer (the twelfth convolutional layer). , and four backflow optical sheets , These optical flows and multi-frame infrared features Figure 1It is fed into the feature enhancement module for the alignment and enhancement of infrared small target features.

[0133] Feature enhancement module, see Figure 10 First, the forward optical flow Use the warp operation to make it... Distort, then compare the distorted result with... The summation yields the forward optical flow across two frames. The same principle applies to backward optical flow processing. Use the warp operation to make it... Distort, then compare the distorted result with... The sum is used to obtain the backward optical flow across two frames. .

[0134] Secondly, each infrared feature map (excluding intermediate frames) and the optical flow in the corresponding direction (i.e., forward optical flow) Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Alignment features are obtained by aligning the warp operation to the intermediate frame. , Specifically, the first infrared feature image is aligned to the third frame via the forward light flow (1 to 3), the second infrared feature image is aligned to the third frame via the forward light flow (2 to 3), the fourth infrared feature image is aligned to the third frame via the backward light flow (4 to 3), and the fifth infrared feature image is aligned to the third frame via the backward light flow (5 to 3), driving the light flow towards the middle frame.

[0135] Then align the infrared feature maps and features. Forward optical flow Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Unfold along one dimension and piece them together to form a connection feature. , Where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0136] Connection characteristics After feature extraction processing by a (3×3) convolutional layer (the sixteenth convolutional layer), the ninth feature map is obtained, which is divided into two branches.

[0137] One branch is used to calculate the sampling point offset. Specifically, the seventeenth convolutional layer is used to extract features from the ninth feature map to obtain the tenth feature map. This is based on the forward optical flow. (Forward optical flow) Forward optical flow across two frames ) and backward optical flow (backward optical flow) Backward optical flow across two frames The offset can be calculated. The tenth feature map and the offset are added together to generate the sampling point offset. , Where T represents the number of frames, H represents the height, W represents the width, M represents the number of attention heads, and P represents the number of sampling points.

[0138] Another branch generates standardized attention weights. Specifically, the eighteenth convolutional layer and the softmax activation function, connected sequentially, extract features from the ninth feature map to obtain standardized attention weights. , Where T represents the number of frames, H represents the height, W represents the width, M represents the number of attention heads, and P represents the number of sampling points.

[0139] Five infrared feature images are processed through two 3×3 convolutional layers (the nineteenth and twentieth convolutional layers) and a spreading layer to be flattened into flattened features. , Where C represents the number of channels, and T, H, and W represent the number of frames, height, and width, respectively. Subsequently, the amortized features... Sampling point offset Standardized attention weights The refined features are processed using a Deformable Attention Function (DAF). , Where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0140] To facilitate network optimization, 5 infrared feature images were used. With refined features The residual connections are summed and then passed through a (1×1×5) 3D convolutional layer to generate multiple frames of the twelfth feature map. , Twelfth feature map of multiple frames The time feature map is obtained after residual connection. , Where T represents the number of frames, C represents the number of channels, H represents the height, and W represents the width.

[0141] In an embodiment of the present invention, in the deformable attention function, the sampling point offset The attention weights are used to identify pixels from the same source between adjacent frames, and the normalized attention weights are used to identify pixels from the same source between adjacent frames. Amortized features represent the probability of a pixel being a target. It stores the initial information of the image. In subsequent use, the final temporal feature map... The twelfth feature map of multiple frames needs to be processed. It is obtained by splicing together.

[0142] In one embodiment of the present invention, such as Figure 12 As shown, the first, second, and third fusion enhancement modules all include a fusion enhancement module. This fusion enhancement module may include a second depthwise separable convolutional layer, a second pointwise convolutional layer, a collaborative attention submodule, a gating submodule, and a batch normalization layer connected sequentially. The fusion enhancement module receives a first input feature map and a second input feature map, and outputs a fused feature map. Specifically, the fusion enhancement module is used for:

[0143] The fourth concatenated feature is extracted by using the second depthwise separable convolutional layer and the second pointwise convolutional layer connected in sequence to obtain the thirteenth feature map. The fourth concatenated feature is obtained by concatenating the first input feature map and the second input feature map along the channel dimension.

[0144] By fusing the internal features of the thirteenth feature map using the collaborative attention submodule, the fourteenth feature map is obtained.

[0145] Residual connection processing is performed on the thirteenth and fourteenth feature maps to obtain the fifteenth feature map;

[0146] The gate value is generated using the gated submodule to generate the weight values ​​of the fifteen feature maps.

[0147] Add the first product and the second product to get sixteen features. The first product is the product of the fifteen feature maps and the weight value gate, and the second product is the product of (1-gate) and the fourth concatenated feature.

[0148] The sixteen features are processed using a batch normalization layer to obtain a fused feature map.

[0149] This invention utilizes the spatial, temporal, and frequency domain features of infrared targets. However, due to the different learning paths for features from different domains, feature mismatches often occur. Therefore, a fusion enhancement module (FEM) is designed to further compensate for features, fusing and enhancing the differences and commonalities between features from different domains.

[0150] The fusion enhancement module (FEM) concatenates the two-domain features (the first and second input feature maps) along the channel dimension to obtain the fourth concatenated feature. Then, it performs preliminary processing on the fourth concatenated feature using depthwise convolution (the second depthwise separable convolutional layer) and pointwise convolution (the second pointwise convolutional layer), resulting in the thirteenth feature map. The thirteenth feature map is then fused with its internal features through the collaborative attention (Co_Attention, co_att) submodule to obtain the fourteenth feature map. Finally, the thirteenth and fourteenth feature maps are residually connected to obtain the fifteenth feature map.

[0151] The collaborative attention submodule in this embodiment of the invention: adjusts the image size of the input feature map (thirteenth feature map) through global average pooling (third average pooling layer). Where T represents the number of frames and C represents the number of channels, the channel weight information is adjusted through two (1×1) convolution layers (the twenty-second and twenty-third convolution layers). During the processing, the number of channels changes from C to C / 4, and finally to C.

[0152] Next, the fifteen feature maps are passed through a gating submodule to obtain a weight value, gate. The weight value gate is multiplied by the fifteen feature maps, and then (1-gate) is multiplied by the fourth concatenated feature to obtain the final output.

[0153] It should be noted that the processing flow of the gating submodule is the same as that of the collaborative attention module, but the number of channels in the processing flow has been changed, so that the output of the gating submodule is a weight value. , where T represents the number of frames.

[0154] In one embodiment of the present invention, the detection model loss function for:

[0155] ;

[0156] in, This represents the loss function of the detection model. This represents the boundary regression loss function. Represents the classification loss function. Represents the target probability loss function. This represents the forward optical flow loss function. This represents the backward optical flow loss function. This represents the hyperparameter of the boundary regression loss. This represents the hyperparameter of the classification loss. This represents the target probability loss hyperparameter. This represents the hyperparameter of optical flow loss;

[0157] ;

[0158] in, This represents the boundary regression loss function. This represents the intersection-union ratio hyperparameter. This represents the intersection-union ratio loss function. This represents the normalized Gaussian distance hyperparameter. This represents the normalized Gaussian distance loss function. Indicates the predicted bounding box. Represents the true bounding box. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This indicates a Gaussian distribution of the predicted bounding box. This represents the Gaussian distribution of the true bounding box. Represents a constant;

[0159] Classification loss function for:

[0160] ;

[0161] in, Let i represent the classification loss function, i represent the number of training infrared videos in the training dataset, and c represent the number of classes. For accurate classification, For predictive classification;

[0162] Target probability loss function for:

[0163] ;

[0164] in, Let i represent the target probability loss function, and let i represent the number of training infrared videos in the training dataset. For true probability, To predict probabilities;

[0165] Forward optical flow loss function for:

[0166] ;

[0167] in, This represents the forward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the forward-facing real image. Represents the forward-estimated image;

[0168] Backward optical flow loss function for:

[0169] ;

[0170] in, This represents the backward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the backward real image, This represents the backward estimated image.

[0171] In this embodiment of the invention, the three-domain deep learning network implements three-domain learning. To obtain better results, the loss function uses a modified YOLOX loss function and an optical flow loss for supervising optical flow. (Detection model loss function) as follows:

[0172] ;

[0173] in, This represents the loss function of the detection model. It is the boundary regression loss. It is classification loss. It is the target probability loss. It is the forward optical flow loss. It is a backward optical flow loss. , , and These are the four hyperparameters of the balancing loss term.

[0174] As a boundary regression loss, the IOU (Intersection over Union) loss is typically used. However, due to the small target size, the popular IOU loss may be insufficient to capture detailed information about the target distribution. Considering this, a normalized Gaussian Wasserstein distance (NWD) loss is added to minimize the differences between target regions. The IOU loss assists the network in accurately locating small targets, while the NWD loss learns their distribution characteristics. Therefore, the boundary regression loss in this embodiment of the invention... This is a regression loss that combines IOU loss and NWD loss.

[0175] ;

[0176] in, For the predicted bounding box, The bounding box is marked. and Let represent the Gaussian distributions of the predicted bounding boxes and the labeled bounding boxes, respectively. C is a constant related to the dataset. (Coefficients) and These are the two hyperparameters that balance the loss term.

[0177] For forward optical flow and backward optical flow Using forward optical flow loss and backward optical flow loss Supervise it. Decompose the input multi-frame infrared video into forward images. Backward image Forward image pair Performing a warp operation yields the backward-estimated image. Similarly, the backward image pair Performing a warp operation yields the forward-estimated image. The mean squared error (MSE) is used as a measure to quantify the similarity between the original and reconstructed images, and the final forward optical flow loss is... and backward optical flow loss They are represented as:

[0178] ;

[0179] in, This represents the forward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the forward-facing real image. This represents the forward-estimated image.

[0180] ;

[0181] in, This represents the backward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the backward real image, This represents the backward estimated image.

[0182] The proposed algorithm for detecting densely moving infrared small targets based on three-domain feature learning significantly improves the detection performance of densely moving small targets in complex scenes by fusing spatiotemporal and frequency multi-dimensional features. By using multi-domain feature complementarity enhancement, the fusion of temporal, spatial, and frequency domain features solves the problem of insufficient representation of dense small targets by single-domain features, significantly improving detection accuracy. The loss function employs a composite loss function including bounding box regression, forward and backward optical flow, and classification probability, combined with IOU and normalized Gaussian Wasserstein distance (NWD) loss, balancing target localization accuracy and distribution characteristic learning, thus optimizing the recall and localization accuracy of small target detection. Addressing the challenges of missed and false detections of densely moving targets, the method proposed in this invention can reduce the false detection rate in dense scenes and has broad application prospects.

[0183] To address the issue of weak feature correlation among densely moving small targets, this invention proposes a three-domain feature collaborative learning strategy. It models global contextual relationships through spatial domain nonlocal attention, captures motion trajectories through temporal optical flow-guided deformable convolution, and extracts multi-dimensional spectral features through frequency domain Fourier decomposition. By leveraging the complementarity of features from different domains, it achieves accurate representation of densely moving small targets in complex scenes, thus overcoming the deficiency of insufficient information from single-domain features.

[0184] To address the issue of mismatch in cross-domain feature fusion, this invention proposes a fusion enhancement module (FEM). This module uses a dual mechanism of channel attention and spatial attention to dynamically assign weights to spatial-temporal-frequency domain features. Combined with residual connections to compensate for feature differences, it effectively eliminates cross-domain feature offset and improves the robustness of multi-domain feature fusion.

[0185] To address the pain point of insufficient localization accuracy for dense small targets, this invention proposes a composite loss function that integrates normalized Gaussian Wasserstein distance (NWD) loss and IOU loss, taking into account both target distribution characteristics and bounding box regression accuracy. Simultaneously, forward and backward optical flow reconstruction losses are introduced to constrain motion feature learning, achieving a dual improvement in the detection accuracy and motion trajectory stability of dense targets.

[0186] The present invention also proposes an electronic device.

[0187] In this embodiment, such as Figure 13 As shown, the electronic device 500 includes a processor 501, a memory 503, and a computer program stored in the memory. When the processor executes the computer program, it implements the above-mentioned method for detecting densely moving infrared small targets based on three-domain feature learning.

[0188] Figure 13 This is a structural block diagram of the controller according to an embodiment of the present invention.

[0189] like Figure 13 As shown, the controller 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.

[0190] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0191] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0192] The memory 503 stores a computer program corresponding to the densely moving infrared small target detection method based on three-domain feature learning in the above embodiments of the present invention. This computer program is executed under the control of the processor 501. The processor 501 executes the computer program stored in the memory 503 to implement the content shown in the aforementioned method embodiments. Figure 13 The controller 500 shown is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.

[0193] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0194] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0195] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0196] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0197] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0198] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0199] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0200] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting densely moving infrared small targets based on three-domain feature learning, characterized in that, The method includes: Acquire the infrared video to be tested, wherein the infrared video to be tested includes N consecutive infrared video frames, where N is a positive integer; The infrared video to be tested is input into a pre-trained dense moving infrared small target detection model to obtain detection results. These results include the detection bounding box of the infrared small target, the Gaussian distribution of the bounding box, the detection classification, and the detection probability. The dense moving infrared small target detection model includes a backbone network, a spatial feature extraction module, a frequency domain feature extraction module, a temporal feature extraction module, a fusion enhancement network, and a detection head. The input of the backbone network is used to input the infrared video to be tested. The output of the backbone network is connected to the inputs of the spatial feature extraction module, the frequency domain feature extraction module, and the temporal feature extraction module, respectively. The outputs of the block and the temporal feature extraction module are both connected to the input of the fusion enhancement network. The fusion enhancement network includes a first fusion enhancement module, a second fusion enhancement module, and a third fusion enhancement module. The outputs of the spatial feature extraction module and the frequency domain feature extraction module are both connected to the input of the first fusion enhancement module. The outputs of the frequency domain feature extraction module and the temporal feature extraction module are both connected to the input of the second fusion enhancement module. The outputs of the first fusion enhancement module and the second fusion enhancement module are both connected to the input of the third fusion enhancement module. The output of the third fusion enhancement module is connected to the input of the detection head, which is used to output the detection result.

2. The method for detecting densely moving infrared small targets according to claim 1, characterized in that, The training process of the densely moving infrared small target detection model includes: Obtain a training dataset, wherein the training dataset includes training infrared videos and labeling results corresponding to each infrared video frame in the training infrared videos, the training infrared videos include N consecutive infrared video frames, and the labeling results include the true bounding box of the infrared small target, the Gaussian distribution of the true bounding box, the true classification, and the true probability; The training infrared videos in the training dataset are input into the densely moving infrared small target detection model to obtain prediction results, wherein the prediction results include the predicted bounding box of the infrared small target, the Gaussian distribution of the predicted bounding box, the predicted classification, and the predicted probability. A boundary regression loss function is constructed based on the predicted bounding box, the Gaussian distribution of the predicted bounding box, the true bounding box, and the Gaussian distribution of the true bounding box. A classification loss function is constructed based on the predicted classification and the true classification. A target probability loss function is constructed based on the true probability and the predicted probability. A forward optical flow loss function and a backward optical flow loss function are constructed based on the training infrared video of the input detection model. A detection model loss function is constructed based on the boundary regression loss function, the classification loss function, the target probability loss function, the forward optical flow loss function, and the backward optical flow loss function. The dense moving infrared small target detection model is trained according to the loss function of the detection model to obtain the trained dense moving infrared small target detection model.

3. The method for detecting densely moving infrared small targets according to claim 2, characterized in that, The step of inputting the training infrared videos from the training dataset into the densely moving infrared small target detection model includes: The backbone network is used to extract features from each infrared video frame in the training infrared video to obtain an infrared feature map group, wherein the infrared feature map group includes N infrared feature maps. The spatial feature extraction module is used to extract spatial features from the infrared feature map group to obtain a spatial feature map. The frequency feature extraction module is used to extract frequency features from each infrared feature map in the infrared feature map group to obtain a frequency feature map group, wherein the frequency feature map group includes N frequency feature maps; The time feature extraction module is used to extract time features from the infrared feature map group to obtain a time feature map; The first fusion enhancement module generates a spatial-frequency fusion feature map based on the spatial feature map and the frequency feature map group; the second fusion enhancement module generates a time-frequency fusion feature map based on the frequency feature map group and the time feature map; and the third fusion enhancement module generates a spatial-time-frequency fusion feature map based on the spatial-frequency feature map and the time-frequency feature map. The detection head is used to generate the prediction result based on the spatial-temporal-frequency fusion feature map.

4. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The backbone network includes a first branch, a second branch, and a residual connection layer. Both the first and second branches are connected to the residual connection layer. The first branch includes a first convolutional unit, a second convolutional layer, a dilated convolutional unit, a spatial attention layer, and a channel attention layer, connected in sequence. Each first convolutional unit comprises eight parallel first convolutional layers, with the convolution directions corresponding to the convolution kernels being horizontal to the right, horizontal to the left, vertically upward, vertically downward, upper right, upper left, lower right, and lower left, respectively. Each dilated convolutional unit includes four parallel dilated convolutional layers. The second branch includes a third and a fourth convolutional layer, connected in sequence. The feature extraction of each infrared video frame from the training infrared video using the backbone network includes: The grayscale difference between the center pixel and the boundary pixel in the infrared video frame in the corresponding direction is calculated using the eight parallel first convolutional layers in the first convolutional unit to obtain eight first feature maps. Multiply the two feature maps with opposite convolution directions from the eight first feature maps element by element to obtain four difference feature maps; The four difference feature maps are added element by element in a vertical reverse direction to obtain two second feature maps; The second convolutional layer is used to extract features from the first concatenated feature map to obtain a primary feature map, wherein the first concatenated feature map is obtained by concatenating two second feature maps; The primary feature map and the third feature map are subjected to residual connection processing to obtain a fourth feature map, wherein the third feature map is extracted from the infrared video frame by the third convolutional layer; The fourth feature map is processed by dilation convolution using four parallel dilation convolution layers in the dilation convolution unit to obtain four dilation feature maps. The four expanded feature maps are stitched together to obtain a secondary feature map. The spatial attention layer is used to extract features from the secondary feature map to obtain the fifth feature map; The fifth feature map of the channel attention layer is used to extract features to obtain the sixth feature map; The sixth and seventh feature maps are processed by residual connection layer to obtain infrared feature maps, wherein the seventh feature map is extracted from the third feature map by the fourth convolutional layer.

5. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The spatial feature extraction module includes a third branch, a fourth branch, and a memory enhancement submodule. The third branch and the fourth branch are both connected to the memory enhancement submodule. The third branch includes a fifth convolutional layer, a sixth convolutional layer, the spatial enhancement submodule, and a first embedding layer connected sequentially. The fourth branch includes a second embedding layer. The spatial feature extraction using the spatial feature extraction module to extract spatial features from the infrared feature map group includes: The first N-1 infrared feature images in the infrared feature image group are stitched together to obtain the second stitched feature image; The fifth convolutional layer is used to extract features from the second concatenated feature map to obtain the eighth feature map; The eighth feature map is multiplied element-wise with the Nth video feature map in the infrared feature map group, and then stitched together with the Nth infrared feature map in the infrared feature map group to obtain the third stitched feature map. The sixth convolutional layer is used to extract features from the third concatenated feature map to obtain an initial feature map; The initial feature map is spatially augmented using the spatial augmentation submodule to obtain a spatially augmented feature map. The spatial augmentation submodule includes four sub-branches and a seventh convolutional layer. The four sub-branches perform feature processing on the initial feature map. The processing results of sub-branch three and sub-branch four are residually connected. The seventh convolutional layer then extracts features from the residual connection results. The extracted results are then multiplied element-wise with the summed results to obtain the spatially augmented feature map. The summed results are the element-wise summed results of the processing results of sub-branch one and sub-branch two. Sub-branch one includes a first max-pooling layer and a first fully connected layer connected in sequence. Sub-branch two includes a first average pooling layer and a second fully connected layer connected in sequence. Sub-branch three includes a channel max-pooling layer, and sub-branch four includes a channel average pooling layer. The spatial enhancement feature map and the initial feature map are added element-wise to obtain local-global spatial features; The first embedding layer generates a first key-value pair based on the local-global spatial features, wherein the first key-value pair includes a key. Sum ; Using the memory enhancement submodule, a spatial feature map is obtained based on the first key-value pair and the second key-value pair. The second key-value pair is generated by the second embedding layer based on the Nth video feature map in the feature map group. The second key-value pair includes a key... Sum The memory enhancement submodule utilizes the softmax activation function to activate the key. s and keys The product of is normalized, and the similarity obtained after normalization is compared with the value. Multiplication and value Residual connection processing is performed, and the results of the residual connection are processed by the matching layer to obtain a spatial feature map.

6. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The step of extracting frequency features from each infrared feature map in the infrared feature map group using the frequency domain feature extraction module includes: The features of the infrared feature map are transformed to the frequency domain using Fourier transform to obtain the frequency domain information of the infrared feature map, which includes the amplitude. and phase ; Using the first convolutional group, based on the amplitude and amplitude mask The product of these factors yields the amplitude features, wherein the amplitude mask... The amplitude is affected by the sequentially connected eighth and ninth convolutional layers. and the phase The splicing result is extracted; Using the second convolution group, based on the phase and phase mask The product of these terms yields the phase feature, wherein the phase mask... The amplitude is affected by the sequentially connected eighth and tenth convolutional layers. and the phase The splicing result is extracted; An inverse Fourier transform is used to transform the infrared feature map in the frequency domain back to the original domain based on the amplitude and phase features, resulting in a frequency feature map. The first and second convolutional groups each include an eleventh convolutional layer, a second max-pooling layer, a second average-pooling layer, a first depthwise separable convolutional layer, and a first pointwise convolutional layer. The eleventh convolutional layer is used to extract features from the input product. The second max-pooling layer and the second average-pooling layer are used to pool the feature extraction results. After concatenating the residuals of the max-pooling and average-pooling results, the first depthwise separable convolutional layer and the first pointwise convolutional layer are used sequentially to process the concatenated residuals, resulting in the output features. The input product includes the amplitude... and amplitude mask product, phase and phase mask The product of the amplitude and phase features is the output feature.

7. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The temporal feature extraction module includes a coarse motion estimation branch and a feature enhancement branch connected in sequence. The coarse motion estimation branch includes a feature splicing layer, a second convolutional unit, and a twelfth convolutional layer connected in sequence. The feature enhancement branch includes a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a softmax activation function, a nineteenth convolutional layer, a twentieth convolutional layer, a flattening layer, a deformable attention submodule, a twenty-first convolutional layer, and a 3D convolutional layer. The temporal feature extraction module is used to extract temporal features from the infrared feature map set, including: The feature stitching layer is used to stitch two adjacent infrared feature maps in the infrared feature map group forward and backward respectively to obtain a first set of inter-frame relationship maps and a second set of inter-frame relationship maps. The first set of inter-frame relationship maps and the second set of inter-frame relationship maps each include four inter-frame relationship maps. The second convolutional unit is used to capture the inter-frame relationships of the first set of inter-frame relationship graphs and the second set of inter-frame relationship graphs at different motion speeds, respectively, to obtain the first inter-frame relationship and the second inter-frame relationship. The second convolutional unit includes a thirteenth convolutional layer, a first ReLU activation function, a fourteenth convolutional layer, a second ReLU activation function, a fifteenth convolutional layer and a third ReLU activation function connected in sequence. The twelfth convolutional layer is used to extract features from the first inter-frame relationship and the second inter-frame relationship, respectively, to obtain four forward optical flow images. and 4 backflow images ; The forward optical flow Towards After distortion, with the forward light flow The summation yields the forward optical flow across two frames. ; The backward optical flow Towards After distortion, with backward light flow Adding them together gives the backward optical flow across two frames. ; Each infrared feature map and forward optical flow in the infrared feature map group Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Align with the middle frame to obtain the alignment feature; The infrared feature map group, the alignment feature, and the forward optical flow are used. Forward optical flow across two frames Backward optical flow Backward optical flow across two frames Perform connection processing to obtain connection characteristics; The connection features are extracted using the sixteenth convolutional layer to obtain the ninth feature map; The deformable attention submodule is used to process the sampling point offset, standardized attention weights, and amortized features to obtain refined features. The sampling point offset is obtained based on the tenth feature map and the offset itself. The tenth feature map is extracted from the ninth feature map by the seventeenth convolutional layer, and the offset is based on the forward optical flow. Forward optical flow across two frames Backward optical flow Backward optical flow across two frames It is calculated that the standardized attention weights are obtained by processing the ninth feature map through a sequentially connected tenth convolutional layer and a softmax activation function, and the amortized features are obtained by processing the infrared feature map group through a sequentially connected nineteenth convolutional layer, the twentieth convolutional layer, and amortized layer. The refined features are extracted using the twenty-first convolutional layer to obtain the eleventh feature map; The features obtained by residual connection between the infrared feature map group and the eleventh feature map are processed using the 3D convolutional layer to obtain N twelfth feature maps; Residual concatenation is performed on N of the twelfth feature maps to obtain the time feature map.

8. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The first, second, and third fusion enhancement modules all include a fusion enhancement module. Each fusion enhancement module comprises a second depthwise separable convolutional layer, a second pointwise convolutional layer, a collaborative attention submodule, a gating submodule, and a batch normalization layer, connected sequentially. The fusion enhancement module receives a first input feature map and a second input feature map, and outputs a fused feature map. Specifically, the fusion enhancement module is used for: The fourth concatenated feature is extracted using the second depthwise separable convolutional layer and the second pointwise convolutional layer connected in sequence to obtain the thirteenth feature map, wherein the fourth concatenated feature is obtained by concatenating the first input feature map and the second input feature map along the channel dimension; The collaborative attention submodule is used to fuse the internal features of the thirteenth feature map to obtain the fourteenth feature map; Residual connection processing is performed on the thirteenth and fourteenth feature maps to obtain the fifteenth feature map; The gate submodule is used to generate the weight values ​​(gate) of the fifteen feature maps; Add the first product and the second product to obtain sixteen features, wherein the first product is the product of the fifteen feature maps and the weight value gate, and the second product is the product of (1-gate) and the fourth concatenated feature; The sixteen features are processed using the batch normalization layer to obtain a fused feature map.

9. The method for detecting densely moving infrared small targets according to claim 3, characterized in that, The detection model loss function for: ； in, This represents the loss function of the detection model. This represents the boundary regression loss function. Represents the classification loss function. Represents the target probability loss function. This represents the forward optical flow loss function. This represents the backward optical flow loss function. This represents the hyperparameter of the boundary regression loss. This represents the hyperparameter of the classification loss. This represents the target probability loss hyperparameter. This represents the hyperparameter of optical flow loss; The boundary regression loss function for: ； in, This represents the boundary regression loss function. This represents the intersection-union ratio hyperparameter. This represents the intersection-union ratio loss function. This represents the normalized Gaussian distance hyperparameter. This represents the normalized Gaussian distance loss function. Indicates the predicted bounding box. Represents the true bounding box. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This indicates a Gaussian distribution of the predicted bounding box. This represents the Gaussian distribution of the true bounding box. Represents a constant; The classification loss function for: ； in, Represents the classification loss function. This indicates the number of training infrared videos in the training dataset. For the number of categories, For accurate classification, For predictive classification; The target probability loss function for: ； in, Represents the target probability loss function. This indicates the number of training infrared videos in the training dataset. For true probability, To predict probabilities; The forward optical flow loss function for: ； in, This represents the forward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the forward-facing real image. Represents the forward-estimated image; The backward optical flow loss function for: ； in, This represents the backward optical flow loss function. Indicates the height of the image. The width of the image is represented by t, and the time frame is represented by t. Indicates the first The first frame row and number Column pixels, Represents the backward real image, This represents the backward estimated image.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the dense moving infrared small target detection method based on three-domain feature learning as described in any one of claims 1-9.

Images