Infrared target detection method and device based on dual-channel feature fusion

Abstract

The invention provides an infrared target detection method and device based on dual-channel feature fusion. The infrared target detection method comprises the following steps: acquiring an infrared image sequence; the infrared image sequence is processed, a motion feature map sequence is generated, and a motion feature map and a corresponding infrared image have the same spatial resolution and pixel alignment relation; performing multi-scale feature extraction on the infrared image by using a pre-constructed appearance feature extraction model to obtain an appearance feature set, and performing multi-scale feature extraction on the motion feature map by using a pre-constructed motion feature extraction model to obtain a motion feature set; performing bidirectional cross-modal fusion on the appearance features in the appearance feature set and the motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features; and target decoding is carried out on a fusion feature pyramid obtained based on the enhanced appearance features and the enhanced motion features through a detection decoder based on Transform, and a target detection result is obtained.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, specifically to an infrared target detection method and apparatus based on dual-path feature fusion. Background Technology

[0002] Targets in long-range infrared imaging are typically small in size, have low contrast, and weak signal-to-noise ratio. They lack clear shape, texture, or significant brightness features and are easily obscured by complex background clutter, posing a great challenge to existing infrared small target detection methods.

[0003] Existing infrared small target detection methods are mainly divided into single-frame methods and multi-frame methods. Single-frame methods utilize only the spatial information of a single image, making it difficult to effectively distinguish targets from clutter in complex backgrounds. While multi-frame methods can utilize temporal information to improve detection performance, they generally suffer from the following problems: 1) Implicit motion modeling: Existing multi-frame methods typically learn spatiotemporal features implicitly through deep neural networks, which makes the expression of motion information insufficiently explicit and clear, thus limiting the ability to discriminate motion features.

[0004] 2) Additional supervision information is required: Some methods introduce semantic motion descriptions (such as target position, velocity, direction, etc.) as additional supervision, which can improve motion representation, but significantly increases the annotation cost.

[0005] 3) Feature alignment problem: Multi-frame methods need to deal with the feature alignment problem between different frames, which usually requires the design of a special alignment module or optical flow estimation network, increasing the model complexity.

[0006] 4) Insufficient fusion of appearance and motion features: Existing methods often use simple splicing or addition to fuse appearance and motion features, lacking a deep feature interaction mechanism. Summary of the Invention

[0007] One of the purposes of this disclosure is to provide an infrared target detection method that requires no additional labeling.

[0008] One of the purposes of this disclosure is to provide an infrared target detection method that does not require an alignment module.

[0009] One of the objectives of this disclosure is to provide an infrared target detection method that can improve the discriminative ability of features.

[0010] According to a first aspect of this disclosure, an infrared target detection method includes: acquiring an infrared image sequence; processing the infrared image sequence to generate a motion feature map sequence, wherein the motion feature maps in the motion feature map sequence have the same spatial resolution and pixel alignment as the corresponding infrared images in the infrared image sequence; performing multi-scale feature extraction on the infrared images using a pre-built appearance feature extraction model to obtain an appearance feature set, and performing multi-scale feature extraction on the motion feature maps using a pre-built motion feature extraction model to obtain a motion feature set; performing bidirectional cross-modal fusion on the appearance features in the appearance feature set and the motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features; and performing target decoding on the fused feature pyramid obtained based on the enhanced appearance features and enhanced motion features using a Transformer-based detection decoder to obtain a target detection result.

[0011] Optionally, processing the infrared image sequence to generate a motion feature map sequence may include: processing the infrared image sequence using a recursive spatiotemporal difference filter model to generate a motion feature map sequence.

[0012] Optionally, processing the infrared image sequence to generate a motion feature map sequence may include: processing the infrared image sequence using a recursive spatiotemporal difference filter model to generate a motion feature map sequence, including: performing adaptive thresholding on the infrared image to obtain an input response state; processing the input response state using Gaussian kernel convolution and lateral suppression mechanisms to obtain a filtered state; calculating the center-surround contrast of the filtered state to obtain a contrast map and a contrast state; updating the temporal difference between the contrast map of the current frame and the historical state of the previous frame using exponential smoothing to obtain a temporal difference state; fusing the spatial motion component derived from the contrast state with the temporal motion component represented by the temporal difference state, and performing thresholding and nonlinear enhancement processing to generate a motion feature map aligned with the pixel level of the infrared image; and repeating the above steps until all infrared images in the infrared image sequence are traversed to obtain a motion feature map sequence.

[0013] Optionally, the appearance feature extraction model and the motion feature extraction model can use the same network architecture but with independent parameters. The network architecture is a residual neural network, and the extracted multi-scale features can include three levels: P3, P4, and P5, with the resolution decreasing sequentially and the semantic level increasing sequentially.

[0014] Optionally, bidirectional cross-modal fusion is performed on the appearance features in the appearance feature set and the motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features. This may include: at the same scale, using appearance features as queries and motion features as keys and values ​​to generate a first attention output of appearance on motion, and using motion features as queries and appearance features as keys and values ​​to generate a second attention output of motion on appearance; fusing the first attention output and appearance features to generate enhanced appearance features; and fusing the second attention output and motion features to generate enhanced motion features.

[0015] At the same scale, using appearance features as queries and motion features as keys and values, a first attention output is generated for appearance versus motion, and using motion features as queries and appearance features as keys and values, a second attention output is generated for motion versus appearance. This can include: performing adaptive spatial pooling on appearance features and motion features respectively, reducing their respective spatial resolution to a preset fixed size; adding positional encoding to the pooled appearance features and motion features respectively, and flattening them into a one-dimensional feature sequence; using the flattened appearance feature sequence as a query and the flattened motion feature sequence as a key and value, calculating the first attention output through a cross-attention mechanism; and using the flattened motion feature sequence as a query and the flattened appearance feature sequence as a key and value, calculating the second attention output through a cross-attention mechanism.

[0016] Optionally, the computation of the first and second attention can employ a multi-head attention mechanism, and layer normalization and a feedforward network can be applied after the attention computation.

[0017] Optionally, the target detection result is obtained by performing target decoding on the fused feature pyramid obtained based on enhanced appearance features and enhanced motion features through a Transformer-based detection decoder. This can include: concatenating the enhanced appearance features and enhanced motion features into a fused feature pyramid; sequentially passing the fused feature pyramid through a top-down feature pyramid network and a bottom-up path aggregation network to generate an enhanced fused feature pyramid; inputting the enhanced fused feature pyramid into the Transformer-based detection decoder, which uses a set of learnable object query vectors to perform multiple rounds of cross-attention interaction with the enhanced fused feature pyramid, updating the query representation layer by layer to obtain the final object query representation; and mapping the final object query representation through the detection head to obtain the target detection result, which includes the bounding box center coordinates, width, height, and classification confidence of each detected target.

[0018] The detection head may include two parallel fully connected sub-networks, used for bounding box regression and object classification respectively. The steps for mapping the final object query representation to obtain the object detection result through the detection head include: inputting the final object query representation into the parallel bounding box regression fully connected sub-network and the object classification fully connected sub-network; processing the final object query representation using the bounding box regression fully connected sub-network, outputting four-dimensional bounding box parameters, and activating them with a Sigmoid function to obtain normalized bounding box center x-coordinate, center y-coordinate, width, and height; processing the final object query representation using the object classification fully connected sub-network in single-class object detection mode, outputting a single-dimensional score, and activating it with a Sigmoid function to obtain the classification confidence score; denormalizing the normalized bounding box center x-coordinate, center y-coordinate, width, and height to obtain bounding box center x-coordinate, center y-coordinate, width, and height in pixels; and combining the pixel-level bounding box center x-coordinate, center y-coordinate, width, and height with the classification confidence score to form the object detection result.

[0019] According to a second aspect of this disclosure, an infrared target detection device includes: a memory storing a program or instructions, and a processor that, when the program or instructions are executed by the processor, causes the processor to perform the infrared target detection method described above.

[0020] The infrared target detection method according to the embodiments of this disclosure can improve detection performance. Attached Figure Description

[0021] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein: Figure 1 This is a flowchart illustrating an infrared target detection method according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating an infrared target detection method according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating the structure of a recursive spatiotemporal differential filter according to an embodiment of the present disclosure; Figure 4 This is a detailed structural diagram illustrating a bidirectional cross-modal attention fusion module according to an embodiment of the present disclosure; Figure 5 These are comparison images of motion feature map visualization effects according to embodiments of the present disclosure; and Figure 6 This is a schematic diagram illustrating the final target detection effect according to an embodiment of the present disclosure. Detailed Implementation

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0023] The following detailed description is provided to aid in obtaining a full understanding of the methods, apparatus, and / or systems described herein. However, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein; equivalent substitutions or changes may be made, except for operations that must occur or be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of content well-known in the art will be omitted or simplified.

[0024] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above.

[0025] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0026] The motion feature map of the infrared target detection method according to the embodiments of this disclosure is defined on the same pixel grid as the original image, and has a pixel-level alignment relationship, eliminating the need for additional optical flow estimation or deformation alignment modules.

[0027] The infrared target detection method according to the embodiments of this disclosure improves the discrimination capability through dual-path feature interaction. Specifically, it realizes deep interaction between appearance features and motion features through a bidirectional cross-modal attention mechanism. Appearance features provide contextual information for motion features, and motion features provide motion cues for appearance features. The two enhance each other, significantly improving the discrimination capability of the features.

[0028] The infrared target detection method according to embodiments of this disclosure explicitly models motion information using a recursive spatiotemporal difference filter model, generating pixel-aligned motion feature maps. Since the recursive spatiotemporal difference filter model is a parameterless deterministic algorithm, it does not require parameter learning and therefore eliminates the need for additional motion annotation data, reducing annotation costs. A detailed description is provided below with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart illustrating an infrared target detection method according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating an infrared target detection method according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating the structure of a recursive spatiotemporal differential filter according to an embodiment of the present disclosure; Figure 4 This is a detailed structural diagram illustrating a bidirectional cross-modal attention fusion module according to an embodiment of the present disclosure.

[0030] Reference Figure 1 The infrared target detection method according to the embodiments of the present disclosure may include steps S100, S200, S300, S400 and S500.

[0031] In step S100, an infrared image sequence is acquired.

[0032] As an example, an infrared image sequence can be acquired. Where t = 1, 2, ..., T represents the time step. Each frame of the original infrared image It has a size of H × W and a pixel value range of [0, 255]. For example, a single-channel infrared image can be converted into a three-channel format. This is achieved through channel replication. Additionally, preprocessing can be performed linearly, such as size normalization (e.g., adjusting to 512×512) and pixel value normalization to [0, 1]. Furthermore, image sequences can be organized chronologically to ensure correct temporal relationships.

[0033] It's important to note that for training data, the target bounding box annotations for each frame are required. For testing / inference data, only the image sequence is needed. During training, only standard bounding box annotations are required; no additional motion semantic annotations are needed.

[0034] In step S200, a sequence of motion feature maps is generated.

[0035] Specifically, the infrared image sequence can be processed to generate a motion feature map sequence, in which the motion feature map has the same spatial resolution and pixel alignment as the corresponding infrared image in the infrared image sequence.

[0036] As an example, a recursive spatiotemporal difference filter model can be used for feature extraction. The image is treated as a two-dimensional grid, with each pixel (x, y) maintaining a set of continuous state values. These states are updated using fixed local operation rules. This model does not require parameter learning and is entirely based on predefined mathematical operations, thus eliminating the need for motion-annotated data.

[0037] As an example, the steps for generating a motion feature map sequence may include: performing adaptive thresholding on the infrared image to obtain an input response state; processing the input response state through Gaussian kernel convolution and lateral suppression to obtain a filtered state; calculating the center-surround contrast of the filtered state to obtain a contrast map and a contrast state; updating the temporal difference between the contrast map of the current frame and the historical state of the previous frame through exponential smoothing to obtain a temporal difference state; fusing the spatial motion component derived from the contrast state with the temporal motion component represented by the temporal difference state, and performing thresholding and nonlinear enhancement processing to generate a motion feature map aligned with the pixel level of the infrared image; and repeating the above steps until all infrared images in the infrared image sequence are traversed to obtain a motion feature map sequence.

[0038] Specifically, the recursive spatiotemporal difference filter model may include a five-layer continuous state update mechanism, with each layer processing specific visual information, progressing layer by layer, and finally generating a motion feature map.

[0039] Reference Figure 3 The first input response layer performs adaptive thresholding on the input image (infrared image) to achieve pixel intensity response.

[0040] Specifically, the input response state can be obtained by adaptive threshold processing using the following equation (1). .

[0041] (1) Where: Threshold is an adaptive threshold function, defined as:

[0042] in, This is the gain coefficient. This is the threshold parameter.

[0043] Reference Figure 3 The second spatial filtering layer achieves spatial smoothing through Gaussian kernel convolution and introduces a lateral suppression mechanism.

[0044] The calculation formula is as follows: (2) in, In filtering state; The kernel is a Gaussian convolution, and its size can be... The standard deviation can be 1.0. Indicates the convolution operation; To suppress intensity; Used to ensure non-negative output. Spatial filtering layers enhance local contrast through a center-surround mechanism, suppressing uniform areas and highlighting edges and textures.

[0045] Reference Figure 3 The third contrast enhancement layer calculates the center-surround contrast and extracts spatial gradient information.

[0046] The center-surround contrast of the filtered state can be calculated using the following equation (3).

[0047] (3) in, This is a contrast image. The contrast threshold; is the gain coefficient; the first term in equation (3) can be used to extract positive contrast (bright target), and the second term can be used to extract negative contrast (dark target).

[0048] The contrast state S3 can be updated using the following equation (4): (4) It can calculate the ON and OFF contrast channels, simultaneously detect bright and dark targets, and enhance contrast features.

[0049] Reference Figure 3 The fourth layer is time difference layering: time difference is performed through exponential smoothing to extract motion response.

[0050] The specific calculations can be shown in equations (5) and (6) below: (5) (6) in, For time-differential states, This refers to the historical state, that is, the contrast state of the previous frame. This represents the temporal differential state of the previous frame; This is the time smoothing coefficient; This is the motion sensitivity coefficient; This represents the spatial gradient operator (used in the first frame). The first frame uses the spatial gradient to initialize the motion response, and subsequent frames detect changing regions through temporal difference, thereby exponentially smoothing the accumulation of historical motion information to form motion memory.

[0051] Reference Figure 3The fifth layer, the motion feature synthesis layer, fuses spatial motion components and temporal motion components to generate the final motion feature map.

[0052] The motion input can be constructed using the following equation (7). : (7) in, This refers to the time-series difference weights.

[0053] Then, the spatial motion components are calculated using the following equation (8). : (8) in, The center-surround difference convolution kernel can be defined as the difference of Gaussians (DoG).

[0054] (9) in, This represents the distance from the pixel to the center of the kernel.

[0055] The time motion component can be calculated using the following formula (10). : (10) in, For time motion weights.

[0056] The motion state can then be synthesized using the following equation (11). (11) in, This is the gain coefficient.

[0057] Next, a motion feature map can be generated according to the following equation (12). .

[0058] (12) in, This is the mixing coefficient.

[0059] The enhancement function can be defined as follows: (13) in, Power-law exponent is used for contrast compression; Bilateral is a two-sided filter with parameters... Normalize normalizes the results to... .

[0060] Through the above five-layer mechanism, the recursive spatiotemporal difference filter model can explicitly extract motion information from the original image sequence and generate motion feature maps. With the original image Having the same spatial resolution Furthermore, it achieves pixel-level alignment, eliminating the need for additional alignment operations.

[0061] All operations follow fixed rules and do not require learning parameters, thus eliminating the need for motion annotation data; the motion graph and the original graph are in the same coordinate system and are naturally aligned.

[0062] Reference Figure 1 In step S300, multi-scale feature extraction is performed.

[0063] As an example, a pre-built appearance feature extraction model can be used to extract features from an infrared image at multiple scales to obtain an appearance feature set, and a pre-built motion feature extraction model can be used to extract features from the motion feature map at multiple scales to obtain a motion feature set.

[0064] Reference Figure 2 The input infrared image sequence is first processed by a recursive spatiotemporal difference filter model to generate motion feature maps. These maps are then fed into two structurally identical but parameter-independent backbone networks to extract appearance and motion features, respectively. The appearance modality extracts static visual information, while the motion modality extracts dynamic change features.

[0065] Specifically, it can be based on the original infrared image. As input, multi-scale appearance features are extracted through a backbone network (e.g., ResNet 18 residual neural network, etc.): (14) in, Indicates the first The appearance characteristics of the layer Corresponding to different spatial resolutions, C l , H l , W l These represent the number of channels, height, and width of the feature map in layer l, respectively.

[0066] Reference Figure 2 The backbone network structure can be configured as follows: Input layer: Convolutional layer group 1-2: downsampling to Residual module group 3: Output P3 feature Residual module group 4: Output P4 feature Residual module group 5: Output P5 feature H and W represent the height and width of the infrared image, respectively.

[0067] Additionally, motion feature maps can be used. The input is replicated as a three-channel array, and multi-scale motion features are extracted through a backbone network with the same architecture as the external pathway but independent parameters. (15) The backbone network can have the same structure as the external pathway network, but the parameters are trained completely independently.

[0068] The two feature extraction models use the same backbone architecture, but their parameters are completely independent, optimized separately for appearance features and motion features. The three levels of the feature pyramid correspond to different downsampling ratios and semantic levels: P3 layer: higher resolution (…). P4 layer: suitable for target detection; P4 layer: medium resolution. P5 layer: suitable for detecting medium-sized targets; P5 layer: lower resolution. It has a high semantic level and is suitable for detecting large targets.

[0069] The above-described backbone network configuration is merely an example, and this disclosure is not limited thereto.

[0070] Reference Figure 1 In step S400, bidirectional cross-modal fusion is performed.

[0071] Specifically, bidirectional cross-modal fusion can be performed on appearance features in the appearance feature set and motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features.

[0072] Reference Figure 2 At multiple scale levels, enhanced features can be deeply fused through a bidirectional cross-modal attention mechanism to achieve mutual enhancement.

[0073] Specifically, at the same scale, appearance features can be used as queries and motion features as keys and values ​​to generate a first attention output of appearance on motion, and motion features can be used as queries and appearance features as keys and values ​​to generate a second attention output of motion on appearance. Then, the first attention output and appearance features are fused to generate enhanced appearance features, and the second attention output and motion features are fused to generate enhanced motion features.

[0074] For details, please refer to Figure 4It can adaptively spatially pool appearance features and motion features separately, reducing their respective spatial resolution to a preset fixed size; then, it adds positional encoding to the pooled appearance features and motion features respectively, and flattens them into a one-dimensional feature sequence; using the flattened appearance feature sequence as the query and the flattened motion feature sequence as the key and value, it calculates the first attention output through a cross-attention mechanism; using the flattened motion feature sequence as the query and the flattened appearance feature sequence as the key and value, it calculates the second attention output through a cross-attention mechanism.

[0075] Specifically, to reduce computational complexity, adaptive pooling of the P3 layer features (appearance features and motion features) can be performed first using the following equations (16) and (17): (16) (17) in: The pooled feature size is fixed; AdaptivePool is adaptive average pooling, and the pooled feature size is fixed. .

[0076] Next, using equations (18) and (19) below, the pooled features are flattened into a sequence, and positional encoding is added: (18) (19) in, , PE is a two-dimensional sinusoidal position code that preserves spatial position information.

[0077] The location encoding method is shown in equations (20) and (21) below: (20) (twenty one) in, For spatial location, For channel indexing.

[0078] The attention of appearance to motion (appearance features as the query, focusing on motion features) can be calculated using the following equations (22) and (23): (twenty two) (twenty three) The attention of motion to appearance (motion features as the query, focusing on appearance features) can be calculated using the following equations (24) and (25): (twenty four) (25) In equations (22) to (25), It is a learnable linear projection matrix; For the dimension of attention head, through calculate; The number of heads in the multi-head attention mechanism; Softmax normalization along the key dimension ensures that the sum of the attention weights is 1. Additionally, The appearance features are enhanced in context based on the motion features to highlight the motion area; Motion features are enhanced in context based on appearance features to suppress background motion.

[0079] Next, the first attention output and appearance features are fused to generate enhanced appearance features, and the second attention output and motion features are fused to generate enhanced motion features. This generation may include necessary post-processing of features. For example, the computation of the first and second attention uses a multi-head attention mechanism, and layer normalization and a feedforward network are applied after the attention computation.

[0080] Specifically, the fusion can be achieved through the following equations (26) and (27).

[0081] (26) (27) Where: LayerNorm is layer normalization; FFN is a feedforward network, which can contain two fully connected layers and ReLU activation, as shown in the following equation (28).

[0082] (28) Where w1 and b1 are the weight matrix and bias vector of the first layer, respectively, and w2 and b2 are the weight matrix and bias vector of the second layer, respectively. The dimensions of the intermediate layers can typically be [dimensions missing]. .

[0083] Reference Figure 4 It can reshape sequence features For feature maps, upsample The original P3 resolution can be calculated using the following formula (29).

[0084]

[0085] (29)

[0086]

[0087] The upsampling process uses bilinear interpolation to ultimately obtain enhanced appearance features. and enhanced motion characteristics For other levels (P4, P5), the original extracted features can be used directly. The complete enhanced dual-pathway features are as follows: .

[0088] However, this disclosure is not limited to this; detection can also be performed directly in the sequence space using DETR-type models.

[0089] In step S500, the target is decoded.

[0090] The target detection result can be output by decoding the fused feature pyramid obtained from enhanced appearance features and enhanced motion features through a Transformer-based detection decoder.

[0091] Specifically, the steps for constructing the fusion feature pyramid may include: concatenating the enhanced appearance features and enhanced motion features at each pyramid level to form multi-scale fusion features; projecting the fusion features at each level onto a unified hidden dimension through a 1×1 convolution, for example, the hidden dimension may be 256; and introducing a self-attention mechanism into the fusion features at the highest semantic level to capture global contextual information.

[0092] The steps for feature enhancement using feature pyramid networks and path aggregation networks may include: achieving coarse-to-fine information fusion through a top-down feature pyramid network using nearest neighbor upsampling and reparameterized convolutional blocks; and generating an enhanced fused feature pyramid through a bottom-up path aggregation network using straddle convolution to propagate fine-to-coarse localization details.

[0093] The steps of inputting the enhanced fusion feature pyramid into a Transformer-based detection decoder for object decoding may include: performing Top-K candidate selection based on classification scores, initializing a set of object query vectors (for example, the number of object query vectors can be 100 to 900); processing the object query vectors through multiple deformable Transformer decoder layers (for example, the number of decoder layers can be 2 to 4); employing a multi-scale deformable attention mechanism in each decoder layer, where each attention head samples a preset number of reference points at each feature scale to achieve efficient interaction between object queries and multi-scale features; further processing the query representation through a feedforward network and iteratively refining the bounding boxes, updating the query representation and corresponding anchor box coordinates layer by layer to obtain the final object query representation.

[0094] As an example, the detection head may include two parallel fully connected sub-networks, one for bounding box regression and the other for object classification. The steps for mapping the final object query representation to obtain the object detection result through the detection head include: inputting the final object query representation into the parallel bounding box regression fully connected sub-network and the object classification fully connected sub-network; processing the final object query representation using the bounding box regression fully connected sub-network, outputting four-dimensional bounding box parameters, and activating them with a Sigmoid function to obtain normalized bounding box center x-coordinate, center y-coordinate, width, and height; processing the final object query representation using the object classification fully connected sub-network in single-class object detection mode, outputting a single-dimensional score, and activating it with a Sigmoid function to obtain the classification confidence score; de-normalizing the normalized bounding box center x-coordinate, center y-coordinate, width, and height to obtain bounding box center x-coordinate, center y-coordinate, width, and height in pixels; and combining the pixel-level bounding box center x-coordinate, center y-coordinate, width, and height with the classification confidence score to form the object detection result. This will be explained in detail below.

[0095] First, an enhanced appearance feature pyramid can be constructed by combining enhanced motion features.

[0096] After obtaining the multi-scale feature set in step S400, the appearance features and motion features are concatenated along the channel dimension at each pyramid level to form a unified fused feature representation. This concatenation operation preserves the explicit contributions of both modalities, enabling the detector to utilize both static appearance information and dynamic motion cues simultaneously. The fused features at each level are then projected onto a unified hidden dimension via 1×1 convolutions for subsequent processing.

[0097] Then, feature enhancement is performed using Feature Pyramid Network (FPN) and Path Aggregation Network (PAN).

[0098] For the highest semantic level fusion features, a self-attention mechanism is first applied to capture global contextual information and enhance the semantic expressiveness of the features. Subsequently, the top-down feature pyramid path, through nearest neighbor upsampling and reparameterized convolutional blocks, progressively passes high-level semantic information to lower levels, achieving coarse-to-fine feature fusion. The bottom-up path aggregation path, through straddle convolution, progressively passes fine-grained localization information from lower levels to higher levels, achieving fine-to-coarse feature enhancement. After bidirectional enhancement via FPN-PAN, an enhanced fusion feature pyramid is obtained.

[0099] Next, the enhanced fusion feature pyramid is input into the Transformer-based detection decoder for target decoding.

[0100] The detection decoder employs a query-based detection paradigm. First, based on the classification response enhanced by fused features, a set of object query vectors is initialized using a Top-K selection mechanism, with each query vector corresponding to a potential detection candidate. The object query vectors are iteratively refined through multiple layers of deformable Transformer decoders. In each decoder layer, a multi-scale deformable attention mechanism is used, allowing each query to adaptively focus on key locations at multiple feature scales, resulting in higher computational efficiency compared to global attention. The decoder also employs an iterative bounding box refinement strategy, updating the bounding box predictions after each decoding layer to progressively improve localization accuracy. After multiple decoding layers, the final object query representation is obtained.

[0101] Finally, the final object query representation can be mapped using the detection head to obtain the target detection result.

[0102] The detection head comprises a bounding box regression subnetwork and an object classification subnetwork set up in parallel. The bounding box regression subnetwork maps the query representation to four-dimensional bounding box parameters, which are then normalized to the center coordinates and width and height after sigmoid activation. The object classification subnetwork maps the query representation to a classification score, which is then normalized to the confidence score after sigmoid activation. Finally, the normalized bounding box parameters are denormalized to pixel coordinates, and combined with the classification confidence score to form a complete object detection result. Each detection result includes the bounding box's center x-coordinate, center y-coordinate, width, height, and classification confidence score.

[0103] The above detection model can be trained using an end-to-end supervised learning approach. During the training phase, denoising queries can be introduced into the object query sequence. Random perturbations are added to the ground truth bounding boxes to generate auxiliary training tasks, accelerating model convergence. Training employs a one-to-one Hungarian matching strategy, uniquely assigning each object query to a ground truth target or labeling it as background. The training loss can use zoom loss as the classification loss, combined with L1 box regression loss and generalized intersection-union (CIU) loss for bounding box optimization. The entire training process requires only standard bounding box annotations, without any additional motion annotations.

[0104] Reference Figure 5 In this diagram, the red box represents the ground truth, a magnified view of the target area. The infrared target detection method disclosed herein can effectively separate moving targets from complex backgrounds and stably detect targets (especially small targets) in consecutive frames. Here, a small target can be defined as one whose bounding box area in the input image is less than 32×32 pixels (i.e., ≤1024 pixels). 2(Targets that occupy less than 0.1% of the total image area are considered. These targets are challenging to detect, identify, and track due to their sparse pixel information and insignificant features.) The detection results of this method closely match the Ground Truth, validating its effectiveness and robustness in moving target detection tasks.

[0105] Reference Figure 6 The red box represents the model inference result. According to the infrared target detection method of this disclosure, it can accurately and stably detect small moving targets in complex backgrounds and achieve effective temporal tracking in continuous video frames. The red box, as the model inference result, basically matches the real target. The infrared target detection method of this disclosure has good detection performance and practical value, and is suitable for intelligent monitoring in nighttime, long-distance, or low-contrast scenarios.

[0106] An infrared target detection apparatus according to an embodiment of the present disclosure may include: a memory storing a program or instructions, and a processor that, when the program or instructions are executed by the processor, causes the processor to perform the infrared target detection method described above.

[0107] The above has been referred to Figures 1 to 6 An infrared target detection method and an infrared target detection apparatus according to exemplary embodiments of the present disclosure are described. However, it should be understood that the devices, units, equipment, etc., shown in the drawings can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these units and devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these systems or devices may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0108] The instructions stored in the aforementioned computer-readable storage medium can be executed in environments deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the instructions can also be used to perform additional steps beyond those described above, or to perform more specific processing while executing the aforementioned steps. The details of these additional steps and further processing are already provided in the reference... Figures 1 to 6 As mentioned in the description of the relevant systems and methods, they will not be repeated here to avoid repetition.

[0109] It should be noted that the process generation method according to the exemplary embodiments of this disclosure can rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0110] The infrared target detection method according to the embodiments of this disclosure requires no additional annotation.

[0111] The infrared target detection method according to embodiments of this disclosure does not require an alignment module.

[0112] The infrared target detection method according to embodiments of the present disclosure can improve the ability to distinguish features.

[0113] While specific terminology has been used to describe various embodiments of this disclosure, the specification and drawings are to be regarded as illustrative rather than restrictive in order to aid in understanding this disclosure. Various modifications and changes can be made by those skilled in the art (e.g., different features in different embodiments may be combined) without departing from the broader spirit and scope of this disclosure. Therefore, the scope of this disclosure is not limited by the specific embodiments and examples, but by the claims and their equivalents.

Claims

1. An infrared target detection method, characterized in that, include: Acquire infrared image sequences; The infrared image sequence is processed to generate a motion feature map sequence, wherein the motion feature maps in the motion feature map sequence have the same spatial resolution and pixel alignment relationship as the corresponding infrared images in the infrared image sequence. The infrared image is subjected to multi-scale feature extraction using a pre-built appearance feature extraction model to obtain an appearance feature set, and the motion feature map is subjected to multi-scale feature extraction using a pre-built motion feature extraction model to obtain a motion feature set. Bidirectional cross-modal fusion is performed on the appearance features in the appearance feature set and the motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features; The target detection result is obtained by decoding the fused feature pyramid obtained from enhanced appearance features and enhanced motion features using a Transformer-based detection decoder.

2. The infrared target detection method according to claim 1, characterized in that, The infrared image sequence is processed to generate a motion feature map sequence, including: processing the infrared image sequence through a recursive spatiotemporal difference filter model to generate a motion feature map sequence.

3. The infrared target detection method according to claim 2, characterized in that, Processing the infrared image sequence to generate a motion feature map sequence includes: processing the infrared image sequence using a recursive spatiotemporal difference filter model to generate a motion feature map sequence, including: Adaptive thresholding is applied to the infrared image to obtain the input response state; The input response state is processed by Gaussian kernel convolution and lateral suppression mechanism to obtain the filtered state; Calculate the center-surround contrast of the filtered state to obtain a contrast map and contrast state; The temporal difference state is obtained by updating the contrast map of the current frame with the historical state of the previous frame through exponential smoothing. By fusing the spatial motion component derived from the contrast state with the temporal motion component represented by the temporal difference state, and performing thresholding and nonlinear enhancement processing, a motion feature map aligned with the pixel level of the infrared image is generated. The above steps are repeated until all infrared images in the infrared image sequence are traversed to obtain the motion feature map sequence.

4. The infrared target detection method according to claim 1, characterized in that, The appearance feature extraction model and the motion feature extraction model use the same network architecture but have independent parameters. The network architecture is a residual neural network. The extracted multi-scale features include three levels: P3, P4, and P5, with the resolution decreasing sequentially and the semantic level increasing sequentially.

5. The infrared target detection method according to claim 1, characterized in that, Bidirectional cross-modal fusion is performed on the appearance features in the appearance feature set and the motion features in the motion feature set to obtain enhanced appearance features and enhanced motion features, including: At the same scale, appearance features are used as queries and motion features are used as keys and values ​​to generate the first attention output of appearance on motion, and motion features are used as queries and appearance features are used as keys and values ​​to generate the second attention output of motion on appearance. The first attention output and the appearance features are fused together to generate enhanced appearance features; The second attention output and the motion features are fused to generate enhanced motion features.

6. The infrared target detection method according to claim 5, characterized in that, At the same scale, using appearance features as queries and motion features as keys and values, a first attention output on appearance versus motion is generated. Conversely, using motion features as queries and appearance features as keys and values, a second attention output on motion versus appearance is generated, including: Adaptive spatial pooling is performed on the appearance features and the motion features respectively to reduce their respective spatial resolution to a preset fixed size; Positional encodings are added to the pooled appearance and motion features, and then flattened into a one-dimensional feature sequence. The flattened appearance feature sequence is used as the query, and the flattened motion feature sequence is used as the key and value. The first attention output is calculated through a cross-attention mechanism. The flattened motion feature sequence is used as the query, and the flattened appearance feature sequence is used as the key and value. The second attention output is calculated through a cross-attention mechanism.

7. The infrared target detection method according to claim 6, characterized in that, The calculation of the first and second attention uses a multi-head attention mechanism, and layer normalization and a feedforward network are applied after the attention calculation.

8. The infrared target detection method according to claim 1, characterized in that, The target detection results are obtained by decoding the fused feature pyramid obtained from enhanced appearance features and enhanced motion features using a Transformer-based detection decoder, including: The enhanced appearance features and enhanced motion features are stitched together to form a fused feature pyramid; The fused feature pyramid is sequentially processed through a top-down feature pyramid network and a bottom-up path aggregation network to generate an enhanced fused feature pyramid. The enhanced fusion feature pyramid is input into a Transformer-based detection decoder, which uses a set of learnable object query vectors to perform multiple rounds of cross-attention interaction with the enhanced fusion feature pyramid, updating the query representation layer by layer to obtain the final object query representation. The final object query representation is mapped by the detection head to obtain the target detection result, which includes the bounding box center coordinates, width, height, and classification confidence of each detected target.

9. The infrared target detection method according to claim 8, characterized in that, The detection head includes two parallel fully connected sub-networks, used for bounding box regression and object classification, respectively. The steps of mapping the final object query representation through the detection head to obtain the object detection result include: The final object query representation is input into the parallel bounding box regression fully connected subnetwork and the target classification fully connected subnetwork, respectively. The final object query representation is processed using the bounding box regression fully connected subnetwork to output four-dimensional bounding box parameters, which are then activated by the Sigmoid function to obtain the normalized bounding box center x-coordinate, center y-coordinate, width, and height. The final object query representation is processed using the target classification fully connected subnetwork in single-class target detection mode, a one-dimensional score is output, and the classification confidence is obtained by activating the Sigmoid function. The normalized bounding box center x-coordinate, center y-coordinate, width, and height are denormalized to obtain the bounding box center x-coordinate, center y-coordinate, width, and height in pixels. The target detection result is formed by combining the horizontal and vertical coordinates of the bounding box center (in pixels), the width, and the height with the classification confidence score.

10. An infrared target detection device, characterized in that, include: Memory, which stores programs or instructions. A processor, when the program or instructions are executed by the processor, causes the processor to perform the infrared target detection method according to any one of claims 1-9.

Images