Visible-thermal infrared vehicle multi-target tracking method and system based on gating mechanism and local map interaction

Abstract

The present application belongs to the technical field of computer vision, and discloses a visible light-thermal infrared vehicle multi-target tracking method and system based on a gating mechanism and local graph interaction. Through the synergistic effect of time sequence information fusion and gating modulation, selective hierarchical fusion, complementary quality evaluation, bidirectional local cross-modal graph interaction and multi-scale aggregation, stable detection and continuous tracking of vehicle targets are realized under complex lighting and severe environmental conditions, the robustness, accuracy and computational efficiency of multi-target tracking are effectively improved, and the consistency of target identity and the integrity of trajectory in the long-term tracking process are ensured.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for multi-target tracking of visible light-thermal infrared vehicles based on gating mechanisms and local graph interaction. Background Technology

[0002] Accurate vehicle tracking is a critical issue in intelligent transportation, autonomous driving, road monitoring, and all-weather security. Most existing multi-target tracking methods rely on visible light images for vehicle target detection and cross-frame association, achieving good tracking results under normal lighting conditions. However, in complex scenarios such as nighttime, low light, backlight, fog, rain, snow, and tunnel entrances / exits, visible light images are prone to problems like blurred vehicle outlines, decreased contrast, and foreground / background confusion, leading to reduced vehicle detection accuracy and further affecting the stability of trajectory association. Thermal infrared images can reflect the vehicle's thermal radiation information and have good imaging stability under low light and complex environmental interference, but they typically lack rich color and texture details, such as vehicle exterior, license plate area, and vehicle surface structure information. Therefore, combining visible light and thermal infrared images for multi-target vehicle tracking is expected to fully utilize the complementary information between the two modalities, thereby improving the robustness of vehicle tracking in complex environments.

[0003] Currently, dual-modal multi-target tracking for vehicle targets still faces many challenges in real-world complex traffic roads or monitoring environments: (1) Insufficient utilization of the temporal information contained in historical frames makes it difficult to fully model the continuity of vehicle movement across frames. In situations such as vehicles driving side by side, overtaking, following, or partial occlusion, the target identity is easily lost or switched, thus affecting the consistency of the target identity during long-term tracking. (2) Some methods perform single-stage fusion of multi-source features before the backbone network, which easily leads to the loss of fine-grained feature information and makes it difficult to achieve context-adaptive fusion of features at different levels. When vehicles exhibit significant scale changes in the image due to changes in distance, the model often struggles to effectively adapt to such scale fluctuations. (3) If a complex fusion mechanism is adopted at each level of the backbone network, redundant computation is easily introduced and network complexity is increased, which is not conducive to achieving efficient multi-target vehicle tracking and makes it difficult to meet the real-time requirements of vehicle platforms or road monitoring equipment.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] The present invention aims to solve at least one of the above technical problems, thereby providing a method and system for multi-target tracking of visible light-thermal infrared vehicles based on gating mechanism and local graph interaction.

[0006] To achieve the above objectives, the first technical solution adopted by the present invention is as follows: A visible-thermal infrared vehicle multi-target tracking method based on gating mechanisms and local graph interaction includes: Acquire visible light and thermal infrared images of the current and previous moments, as well as the target heat map of the previous moment; In the visible light branch and the thermal infrared branch respectively, the shallow features of the current frame and the shallow features of the previous frame are fused across frames to obtain temporal interaction features. The target heat map generated at the previous moment is used to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features. The visible light temporal fusion features and thermal infrared temporal fusion features are respectively input into a backbone network with shared weights to extract multi-scale features. Bimodal fusion is performed on the visible light features from level 2 to level 5 and the thermal infrared features, while the level 0 and level 1 features are directly passed backward. The bimodal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation on the two modal features; establishing a bidirectional cross-modal graph connection within a 3×3 local neighborhood; aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages; using Sigmoid gating to perform reliability modulation on the cross-modal messages and fusing them with the source modal feature residuals to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through a 1×1 convolution to obtain fused features. The fusion features from level 2 to level 5 are aligned and aggregated at multiple scales to obtain unified features; Based on the unified features, predict the target center heatmap, target width and height, center offset and cross-frame displacement, and output the vehicle multi-target tracking results in combination with the target association strategy.

[0007] Preferably, the cross-frame cross-attention fusion includes: mapping the shallow features of the current frame and the shallow features of the previous frame to a patch token sequence through the patch embedding operator, superimposing positional encoding, performing cross-attention with the current frame token as the query and the previous frame token as the key and value, and restoring the original spatial resolution through inverse rearrangement, convolution transformation and upsampling.

[0008] Preferably, the gating weights are generated in the following manner: the target heatmap of the previous time step is input into a mapping operator composed of convolution, normalization and nonlinear activation to obtain the heatmap prior features, and the prior features are then converted into gating weights through convolution mapping and Sigmoid activation function.

[0009] Preferably, the receptive fields of the level 0 and level 1 features are small, and their ability to express cross-modal semantic complementarity relationships is limited, so they are directly passed backward without performing bimodal fusion.

[0010] Preferably, establishing a bidirectional cross-modal graph connection within a 3×3 local neighborhood includes: for a feature node at any spatial location in the source modality features, taking that node as the source node; determining multiple neighboring nodes within a 3×3 local neighborhood corresponding to the spatial location in the target modality features; adding corresponding positional encoding information to the source node and each neighboring node; mapping the source node features with added positional encoding to a query vector through a query weight matrix, mapping each neighboring node feature with added positional encoding to a key vector through a key weight matrix, and mapping each neighboring node feature to a value vector through a value weight matrix; calculating attention weights based on the similarity between the query vector and each key vector, and weighting and aggregating each value vector according to the attention weights to obtain the cross-modal message.

[0011] Preferably, the reliability modulation of the cross-modal message using Sigmoid gating includes: inputting the cross-modal message into a gating mapping function, and then processing it through a Sigmoid activation function to generate reliability gating coefficients; multiplying the reliability gating coefficients element-wise with the result of mapping the cross-modal message through the output weight matrix to obtain the gated modulated cross-modal message; adding the gated modulated cross-modal message to the source modal node features and processing it through a linear rectified activation function ReLU to obtain updated node features.

[0012] Preferably, the multi-scale alignment and aggregation of the fusion features from level 2 to level 5 includes: based on a deep aggregation network decoder architecture, a step-by-step feature aggregation method from high to low layers is adopted, channel projection dimensionality reduction is performed on the high-level fusion features, and after spatial alignment upsampling, they are added element-by-element with the adjacent low-level fusion features, and then geometric deformation correction and depth purification are performed through deformable convolution nodes until a unified feature with the same spatial resolution as the level 2 fusion feature is output.

[0013] Preferably, the output of the vehicle multi-target tracking result combined with the target association strategy includes: extracting candidate targets with confidence greater than a preset threshold based on the heat map predicted in the current frame; mapping back to the estimated position of the previous frame using the cross-frame displacement vector corresponding to each candidate target; calculating the distance score with the historical trajectory using a greedy matching strategy; determining successful association and inheriting the identity number when the distance score is less than the preset association threshold; and terminating tracking when the number of consecutive unmatched frames of the historical trajectory exceeds the retention threshold.

[0014] The second technical solution adopted in this invention is: A visible light-thermal infrared vehicle multi-target tracking system based on gating mechanisms and local graph interaction includes: The acquisition module is used to acquire the visible light image and thermal infrared image of the current moment and the previous moment, as well as the target thermal map of the previous moment; The temporal information fusion module is used to perform cross-frame cross-attention fusion of the current frame shallow features and the previous frame shallow features in the visible light branch and the thermal infrared branch respectively to obtain temporal interaction features, and to use the target heat map generated in the previous moment to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features. A dual-modal fusion module is used to input the visible light temporal fusion features and the thermal infrared temporal fusion features into a shared-weight backbone network to extract multi-scale features. Dual-modal fusion is performed on the visible light features from level 2 to level 5 and the thermal infrared features, while the level 0 and level 1 features are directly passed backward. The dual-modal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation on the two modal features; establishing a bidirectional cross-modal graph connection within a 3×3 local neighborhood; aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages; using Sigmoid gating to perform reliability modulation on the cross-modal messages and fusing them with the source modal feature residuals to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through a 1×1 convolution to obtain fused features. A multi-scale aggregation module is used to perform multi-scale alignment and aggregation of the fused features from level 2 to level 5 to obtain unified features; The output module is used to predict the target center heatmap, target width and height, center offset and cross-frame displacement based on the unified features, and output the vehicle multi-target tracking results in combination with the target association strategy.

[0015] Compared with the prior art, the present invention has the following technical effects: This invention achieves stable detection and continuous tracking of vehicle targets under complex lighting and harsh environmental conditions through the synergistic effects of temporal information fusion and gating modulation, selective hierarchical fusion, complementary quality assessment, bidirectional local cross-modal graph interaction, and multi-scale aggregation. It effectively improves the robustness, accuracy, and computational efficiency of multi-target tracking and ensures the consistency of target identity and the integrity of trajectory during long-term tracking.

[0016] This invention achieves adaptive filtering of historical position priors by using heatmap-gated modulation of temporal interaction features, enhancing trajectory continuity. It employs a selective hierarchical fusion strategy to ensure effective fusion of deep semantics while avoiding low-level redundant computation and noise amplification. A complementary quality assessment mechanism adaptively suppresses low-quality modal interference, highlighting more discriminative modal information in the current scene. Through bidirectional local cross-modal graph interaction and reliability gating, fine-grained modal complementarity propagation is achieved while controlling computational complexity, effectively suppressing cross-modal noise. Multi-scale alignment and aggregation solve the problem of cross-scale feature space misalignment, enabling full integration of semantic information at different levels and improving the accuracy of target detection and displacement prediction. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the visible light-thermal infrared vehicle multi-target tracking method based on gating mechanism and local graph interaction provided for embodiments of the present invention; Figure 2 This is a schematic diagram of the overall structure of the visible light-thermal infrared vehicle multi-target tracking network based on gating mechanism and local graph interaction of the present invention. Figure 3 This is a schematic diagram of the time-series information fusion module of the present invention; Figure 4 This is a schematic diagram of the dual-modal fusion module of the present invention; Figure 5 This is a diagramming example of the bidirectional local cross-modal graph of the present invention; Figure 6 This is a schematic diagram illustrating the details of the local cross-modal graph interaction implementation of the present invention; Figure 7 A schematic diagram of the visible light-thermal infrared vehicle multi-target tracking system based on gating mechanism and local graph interaction provided for an embodiment of the present invention. Detailed Implementation

[0018] To make the technical solution of the present invention clearer, a preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. Those skilled in the art can also make formal modifications or equivalent substitutions to the present invention without departing from its spirit.

[0019] refer to Figure 1 The first embodiment of the present invention provides a visible light-thermal infrared vehicle multi-target tracking method based on gating mechanism and local graph interaction, including S101-S105. (See reference...) Figure 2 In a specific embodiment of the present invention, the overall processing flow of the visible light-thermal infrared vehicle multi-target tracking network includes five stages: multi-source input, temporal information fusion, feature extraction and bimodal fusion, multi-scale aggregation, and output. The input end receives the visible light image, thermal infrared image, and target heatmap from the previous moment; after processing by the temporal information fusion module, visible light temporal fusion features and thermal infrared temporal fusion features are obtained respectively; subsequently, these are input into the shared-weight backbone network for multi-scale feature extraction, and selective bimodal fusion is performed at levels 2 to 5; the fused multi-scale features are aligned and aggregated by the multi-scale aggregation module, and finally, the output head predicts the target center heatmap, target width and height, center offset, and cross-frame displacement, and outputs the tracking result in conjunction with the target association strategy. The following describes... Figure 1 Each step is explained in detail.

[0020] S101, acquire the visible light image and thermal infrared image of the current moment and the previous moment, as well as the target thermal map of the previous moment.

[0021] This invention uses the visible light-thermal infrared dual-modal images of two adjacent frames in a video sequence as the basic input. Let the current processing time be t, then first, the visible light image at that time is acquired. and thermal infrared images And the visible light image of the previous time step (t-1) in the same video sequence. and thermal infrared images .

[0022] To guide the network to focus on the spatial distribution of targets at historical moments, the input also includes the target heatmap HM from the previous moment. t-1 As a priori information about spatial location, the above five sets of data together constitute the complete input set of the network at the current moment.

[0023] During model training, to improve the model's generalization ability and prevent overfitting, data augmentation can be performed on the current bimodal input data. This data augmentation can employ commonly used geometric and color transformation strategies, such as random scaling and cropping, random translation, random rotation, and color perturbation. It is important to emphasize that for visible light and thermal infrared images, these spatial geometric transformations must be performed synchronously and consistently on both modal images and their corresponding bounding boxes to ensure strict spatial semantic alignment of the bimodal data. Color perturbation is performed independently and randomly frame by frame. During online inference, the model directly reads the original images of the current and historical frames without requiring additional data augmentation.

[0024] Previous moment target heat map HM t-1 The generation method is as follows: During the training phase, the target's real-time annotation information (such as the target's center point coordinates) is used to render Gaussian response peaks on the image feature map using a two-dimensional Gaussian kernel function, representing the spatial position probability distribution of the target at historical moments; during the online inference phase, a corresponding heatmap is generated based on the tracking results of the previous moment (i.e., the predicted target center point position). The decay radius of the two-dimensional Gaussian kernel function can be controlled using a standard deviation adaptive to the target size, which is a conventional technique for generating Gaussian heatmaps.

[0025] S102, in the visible light branch and the thermal infrared branch respectively, the shallow features of the current frame and the shallow features of the previous frame are fused across frames to obtain temporal interaction features, and the target heat map generated at the previous moment is used to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features.

[0026] After constructing the input data, to enhance the current frame features' ability to perceive the target's historical motion state, this invention performs cross-frame temporal information fusion in both the visible light and thermal infrared branches. The purpose of this step is to establish an explicit feature interaction relationship between the current frame and the previous frame, enabling the current frame features to perceive historical motion information before entering subsequent deep feature extraction. Simultaneously, it utilizes the prior historical position represented by the target heatmap of the previous frame to adaptively filter the temporal interaction features, thereby improving target detection capability and trajectory continuity.

[0027] refer to Figure 2 , Figure 3 For two consecutive frames of visible light and thermal infrared images, the corresponding shallow features are first obtained using visible light feature extractors and thermal infrared feature extractors, respectively. In this embodiment, the feature extractor can be implemented using a 7×7 convolutional layer to extract initial low-level visual features from the original image. Its output channel number can be set to 16, resulting in shallow features with a spatial resolution of H×W and 16 channels, where H represents the height of the feature map and W represents the width of the feature map. It should be noted that using a 7×7 convolution to extract shallow features is a conventional method in convolutional neural network design. Those skilled in the art can also choose other convolutional kernel sizes (such as 5×5) or different numbers of output channels depending on actual computational resources and accuracy requirements.

[0028] Obtain the shallow features X of the current frame t and the shallow features X of the previous frame t-1 Then, mapping is performed using the patch embedding operator. (See reference) Figure 3 The patch embedding operator can be implemented using convolutional mapping with a kernel size and stride of 16. This divides the shallow features into 16×16 image patches and maps them to high-dimensional features, resulting in the mapped features. At this point, the spatial resolution is reduced to... , Where H′ represents the height of the mapped feature map and W′ represents the width of the mapped feature map; the number of channels is expanded to C′=4C (where C is the number of shallow feature channels), and C′ represents the number of mapped feature channels.

[0029] Subsequently, the mapped features are flattened into a sequence of patch tokens (i.e. Figure 3 In the F2T (Feature to Token) operation, let the total number of spatial locations be L = H′W′, where L represents the total number of spatial locations after flattening (i.e., the length of the patch token sequence). Then we can obtain: the query token (Q) corresponding to the current frame token. t ), the key token corresponding to the token in the previous frame (K) t-1 ) and value token (Vt-1 To preserve spatial location information, a learnable location code E is introduced and superimposed on the query token and key token to obtain a query representation with location information. and key representation .

[0030] To ensure that the current frame can fully utilize the encoded information of the previous frame, As a query and V t-1 Using these as key and value respectively, a cross-frame attention mechanism is executed to obtain the current frame token after initial interaction of temporal information, denoted as Z. t : The cross-attention mechanism can be implemented using single-head or multi-head attention, and its internal calculation follows the scaling dot product attention mechanism commonly used in this field.

[0031] After calculating the current frame token after the initial interaction of timing information, the reverse rearrangement operation is performed (i.e. Figure 3 The T2F (Token to Feature) operation shown in the diagram restores the token to a two-dimensional spatial feature, which is then subjected to convolution and upsampling operations to restore it to the same spatial resolution as the shallow features of the current frame, thus obtaining the temporal interaction feature. .

[0032] This enables cross-frame temporal information fusion between features of the previous frame and features of the current frame, allowing features of the current frame to have the ability to perceive historical motion information before entering the extraction of subsequent main features.

[0033] While the aforementioned cross-attention-based temporal fusion establishes cross-frame feature interaction, the temporal interaction features may contain background information or noise unrelated to the target motion. Therefore, this invention further utilizes the target heatmap from the previous time step to perform gated modulation on the temporal interaction features, thereby enhancing the representation ability of the target's historical position and motion continuity.

[0034] refer to Figure 3 On the right, let the target heatmap of the previous moment be HM. t-1 First, the heatmap prior features are input into a heatmap mapping operator to obtain the heatmap prior features. The heatmap mapping operator can be composed of convolutional layers, normalization layers, and nonlinear activation layers in sequence, used to map a single-channel heatmap into a feature representation with the same number of channels as the shallow features. Subsequently, the heatmap prior features are convolved and then processed by a sigmoid activation function to generate heatmap gating weights G with values ​​ranging from 0 to 1. t-1 .

[0035] Finally, using the gate weight G t-1 Temporal interaction features Perform element-wise modulation and combine it with the shallow features X of the current frame. t Perform residual connections to obtain the output features after time-series information fusion. The specific calculation formula is as follows: ; where ⊙ represents the element-wise multiplication symbol.

[0036] Through the aforementioned gating modulation method, the historical location information represented by the target heatmap of the previous moment can adaptively filter the temporal interaction features: near the target's historical location, the gating weight approaches 1, allowing for sufficient transmission of temporal information; in the background region, the gating weight approaches 0, effectively suppressing the propagation of irrelevant temporal noise. Compared to the unconditional injection method that simply adds the features of the previous frame's heatmap to the fused features, the gating modulation mechanism of this invention achieves conditional utilization of historical priors, significantly improving the robustness of the model in temporal modeling under complex scenarios.

[0037] By fusing the aforementioned temporal information, visible light temporal fusion features and thermal infrared temporal fusion features are obtained respectively, which serve as inputs for subsequent backbone network feature extraction.

[0038] S103, the visible light temporal fusion features and thermal infrared temporal fusion features are respectively input into the backbone network with shared weights to extract multi-scale features. The visible light features and thermal infrared features at levels 2 to 5 are subjected to bimodal fusion, and the features at levels 0 and 1 are directly passed backward. The bimodal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation on the two modal features; establishing bidirectional cross-modal graph connections in a 3×3 local neighborhood, aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages, using Sigmoid gating to perform reliability modulation on the cross-modal messages and fusing them with the source modal feature residuals to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through 1×1 convolution to obtain fused features.

[0039] In dual-modal multi-scale feature extraction, the backbone network can adopt a multi-scale feature extraction architecture commonly used in the field, such as a deep convergent network (DLA). The visible light branch and the thermal infrared branch share the backbone network parameters. During the forward propagation process of stepwise downsampling, the backbone network sequentially extracts multi-scale features from level 0 to level 5. Among them, the receptive fields of level 0 and level 1 features are small, mainly representing local small-scale spatial information; the features from level 2 to level 5 have larger receptive fields and stronger semantic expressive power.

[0040] refer to Figure 2To balance the dual-modal fusion effect and computational efficiency, this invention adopts a selective dual-modal fusion strategy: the features of level 0 and level 1 are not subjected to dual-modal fusion, but are only passed on to the next level as the previous level features; the visible light features and thermal infrared features of levels 2 to 5 are input into the dual-modal fusion module to generate the fused features of the corresponding level.

[0041] The technical basis of the selective strategy lies in the fact that features at levels 0 and 1 have small receptive fields and limited ability to express cross-modal semantic complementarity relationships. Introducing complex cross-modal graph interaction mechanisms onto these features can easily increase redundant computation and amplify low-level noise. Features at levels 2 to 5, however, already possess sufficient semantic abstraction capabilities to effectively carry the interaction and fusion of cross-modal complementary information. Therefore, skipping the bimodal fusion at levels 0 and 1 significantly reduces network computational complexity while maintaining accuracy.

[0042] For the visible light features of the i-th level and thermal infrared characteristics Where i∈{2,3,4,5}, the bimodal fusion module performs the following three-level cascaded processing: Level 1: Complementary Quality Assessment. See also... Figure 4 On the left, global average pooling (i.e., the average pooling operation shown by symbol A in the figure) is performed on the two modal features respectively to obtain the corresponding global statistical descriptors. and The global average pooling method averages the spatial dimensions of the feature map, compressing the spatial information of each channel into a single value and generating a global statistical vector for each channel. The specific calculation formula is as follows: ; ; where GAP(⋅) represents the global average pooling operation.

[0043] Subsequently, the global statistical descriptors of the two modalities are concatenated (i.e., the concatenation operation shown by symbol C in the figure) and input into the fully connected mapping function. The complementary quality weights are obtained after Softmax normalization. , where w represents the weight. In this embodiment, the fully connected mapping function can be implemented by inserting a ReLU activation function between two fully connected layers; it should be noted that the specific number of fully connected layers and the type of intermediate activation function are conventional design choices in this field. The formula for calculating the complementary quality weight is: .

[0044] The complementary mass weights are used to perform weighted modulation of the two modal features (i.e., element-wise multiplication as indicated by the symbol ⊗ in the figure) to obtain the weighted visible light features. and thermal infrared characteristics The horizontal lines above the variables represent the weighted modulated features. This step allows the two modal features to undergo quality-reliability-based adaptive modulation before entering subsequent cross-modal interactions, thereby reducing the negative impact of low-quality modes on the fusion results.

[0045] Level 2: Bidirectional local cross-modal graph interaction. (Reference) Figure 4 Middle and Figure 5 , Figure 6 For the weighted bimodal features, this invention establishes a cross-modal graph connection between visible light and thermal infrared within a 3×3 local neighborhood, thereby achieving fine-grained complementary information propagation.

[0046] Specifically, for a feature node (source node) at any spatial location p in the source modality features, multiple neighboring nodes are determined within a 3×3 local neighborhood corresponding to the spatial location p in the target modality features. This local neighborhood is denoted as N(p), representing a neighborhood window centered at location p. See also... Figure 5 Taking the direction from thermal infrared mode to visible light mode as an example, the orange visible light node z at position p in the source mode feature p The 3×3 local neighborhood N(p) around position p in the target modal feature (i.e. Figure 5 Cross-modal connections are established within the set of thermal infrared nodes {q1,q2,…,q9} within the green box.

[0047] To preserve spatial correspondence, corresponding location encoding information e is added to the source node and each neighboring node. p and e q The source node features with added location encoding are then queried through the weight matrix W. q Mapped to query vector q p The features of each neighboring node after adding position encoding are then weighted by the key weight matrix W. k Mapped to key vector k q The features of each neighboring node are weighted by the value weight matrix W. v Mapped to value vector v q .

[0048] Within the 3×3 local neighborhood N(p), according to the query vector q p With each key vector k q Similarity calculation attention weight a p,q And based on the attention weights, apply them to each value vector v q Perform weighted aggregation to obtain cross-modal message m p In this embodiment, the attention weights are calculated using a scaled dot product attention mechanism: ; Where d represents the feature dimension in the attention calculation, used for scaling to prevent the dot product result from being too large; q' represents the position index of any neighboring node in the local neighborhood N(p).

[0049] See Figure 6 Taking the thermal infrared to visible light direction (T→RGB) as an example, according to the above method, the source mode node features Through linear mapping (i.e., through the weight matrix W) q A query vector q is generated by performing a linear transformation. The node features in the target modality N(p) are linearly mapped (i.e., through the weight matrix W). k W v A linear transformation is performed to generate a key vector k and a value vector v. Then, attention weights a are applied. p,q The calculation and weighted aggregation of value vectors yield the cross-modal message m. p .

[0050] To reduce the introduction of noise and unreliable information during cross-modal propagation, this invention further incorporates a reliability gating mechanism. The cross-modal message m... p The input gating mapping function G(⋅) is then processed by the Sigmoid activation function σ(⋅) to generate the reliability gating coefficients g. p The reliability gating coefficient g p With the cross-modal message m p Output weight matrix W o The mapped results are multiplied element-wise to obtain the gated modulated cross-modal message. Finally, the gated modulated cross-modal message is combined with the source modal node features. The features are summed and then processed using the Rectified Linear Activation Function (ReLU) to obtain the updated node features. The specific calculation formula is as follows: ; ; where ⊙ represents element-wise multiplication.

[0051] The aforementioned local cross-modal graph interaction process performs the same operation in the visible light to thermal infrared direction (RGB→T), thereby obtaining the enhanced visible light and thermal infrared features after bidirectional graph interaction. Compared to the global interaction method that establishes cross-modal connections globally, the local graph interaction of this invention restricts computation to a 3×3 spatial neighborhood, significantly reducing computational redundancy; at the same time, by introducing reliability gating, it can effectively suppress the propagation of unreliable cross-modal messages while accurately modeling the local correspondence of bimodalities.

[0052] Level 3: Channel attention modulation. See also Figure 4 On the right, the visible light features of the i-th level after bidirectional graph interaction enhancement. and thermal infrared characteristics Perform global average pooling (as shown by symbol A in the diagram) to obtain the corresponding channel descriptors. and The specific calculation formula is as follows: , .

[0053] The channel descriptors are concatenated (as shown by symbol C in the figure) and then input into the channel mapping function Γ(⋅), and processed by the hyperbolic tangent activation function tanh(⋅) to generate the channel modulation coefficients. In this embodiment, the channel mapping function can be implemented by inserting a ReLU activation function between two convolutional layers; it should be noted that the specific number of convolutional layers and the type of intermediate activation function are conventional design choices in this field. The formula for calculating the channel modulation coefficient is: .

[0054] The hyperbolic tangent activation function has an output range of -1 to 1, enabling bidirectional modulation (enhancement or suppression) of channel features. The enhanced features of both modes are then subjected to residual self-modulation using the channel modulation coefficients, i.e., by multiplying them element-wise and adding the result to the original features, resulting in the modulated visible light and thermal infrared features. , ;in, and These are the modulated visible light and thermal infrared characteristics. and This represents the visible light enhancement feature and thermal infrared enhancement feature of the i-th level after bidirectional local cross-modal graph interaction. Finally, the modulated two-modal features are concatenated and mapped using a 1×1 convolution to obtain the final fused feature.

[0055] The aforementioned complementary quality assessment, bidirectional local cross-modal graph interaction, and channel attention modulation constitute a progressive cascaded structure: the quality assessment first filters reliable modal inputs, the local graph interaction propagates complementary information and performs secondary noise suppression on the basis of reliability, and the channel attention further refines the fusion representation. The three-level structure is coupled with each other, jointly improving the robustness and discriminative power of dual-modal fusion.

[0056] S104, perform multi-scale alignment and aggregation on the fusion features from level 2 to level 5 to obtain unified features.

[0057] After selective bimodal fusion, the network acquires a multi-scale fused feature set from level 2 to level 5. (Reference) Figure 2 To fully integrate semantic information at different scales and resolve the misalignment of cross-scale features in spatial dimensions, the aforementioned fused features are input into a multi-scale aggregation module for alignment and aggregation.

[0058] The multi-scale aggregation module can employ a deep aggregation network decoder (DLA-Decoder) architecture, using a hierarchical feature aggregation approach from high to low layers. Since the number of channels in the fused features differs across layers, projection nodes (such as 1×1 convolutional layers) are first used to reduce the dimensionality and map the channels of the high-level fused features, ensuring that their channel count is strictly aligned with adjacent low-level features. Subsequently, for the high-level features after channel projection, conventional upsampling operations such as transposed convolution or bilinear interpolation are used to amplify the spatial dimension, ensuring that their spatial resolution accurately matches that of adjacent low-level fused features.

[0059] The high-level features, after channel and spatial alignment, are added element-wise with the adjacent low-level fused features. Considering the differences in imaging mechanisms between bimodal images and the spatial misalignment and feature shift that may be introduced during upsampling, the added features are further input into deformable convolution nodes. The deformable convolution is a conventional spatial feature correction method in this field, which can adaptively learn the spatial sampling offset to perform geometric deformation correction and depth refinement on the aggregated features.

[0060] Starting from the fifth-level fusion feature, the network performs the aforementioned "channel projection-spatial upsampling-deformable fusion" operation level by level, from high to low layers, and aligns and fuses it with the adjacent low-level fusion features in turn, until the output is a unified feature with the same spatial resolution as the second-level fusion feature. This achieves full fusion of semantic features at different scales, providing a unified and information-rich feature representation for subsequent target detection and displacement prediction.

[0061] S105, based on the unified features, predict the target center heatmap, target width and height, center offset and cross-frame displacement, and output the vehicle multi-target tracking result in combination with the target association strategy.

[0062] The unified features output by the multi-scale aggregation module are input into the prediction output head of the network to complete multi-target detection and cross-frame displacement prediction. The output head can employ a conventional convolutional branch structure, such as a parallel prediction branch consisting of 3×3 and 1×1 convolutions, generating the following prediction results: a target center heatmap for the current frame, target width and height dimensions, center point offset, and cross-frame displacement vector. Specifically, the center heatmap is used to locate the target center position, the size branch is used to regress the width and height of the target bounding box, the offset branch is used to compensate for center point pixel deviations caused by network downsampling discretization, and the displacement branch is used to predict the two-dimensional offset vector required to map the target center point in the current frame to the same target center point in the previous frame.

[0063] During the training phase, a multi-task joint loss function can be constructed for each of the aforementioned prediction branches. In this implementation, the center point heatmap can employ an improved focal loss to address the imbalance between positive and negative samples, while size regression, center shift, and cross-frame displacement can be optimized using L1 loss. Each loss term can be weighted and summed using preset weight coefficients to balance the magnitude differences between different tasks. The construction and optimization strategies for the loss function are conventional techniques in the field of multi-task learning.

[0064] After constructing the network's forward structure and multi-task joint loss function, the bimodal multi-target tracking network is jointly trained end-to-end based on the preprocessed training dataset. End-to-end training refers to optimizing each module of the network (including the backbone network, temporal information fusion module, bimodal fusion module, and multi-scale aggregation module) as a whole, simultaneously updating the weight parameters of each module through backpropagation—a common technique in deep learning network training.

[0065] During the backpropagation and parameter update phases, the adaptive moment estimator optimizer (Adam optimizer), commonly used in this field, can be employed to perform global iterative updates on each weight parameter. The Adam optimizer adaptively adjusts the learning rate of each parameter by calculating the first and second moment estimates of the gradient, a standard optimization technique in deep learning. As a specific implementation, the initial learning rate of the Adam optimizer can be set to 1.25 × 10⁻⁶. −4 The learning rate can be dynamically adjusted according to the actual training situation, for example, by using a learning rate decay strategy or a cosine annealing strategy.

[0066] The total number of training epochs can be dynamically determined based on the training data size and the changes in validation set loss. As a specific implementation, if a validation set exists in the dataset, the training cycle can be controlled by monitoring changes in the joint loss value of the validation set: when the validation set loss tends to level off and does not decrease significantly over several consecutive epochs, an early stopping mechanism is triggered to end the training. This early stopping mechanism aims to ensure sufficient model convergence while effectively preventing overfitting. It should be noted that the initial learning rate, early stopping trigger conditions, and upper limit of the number of training epochs mentioned above are specific implementation examples. Those skilled in the art can use other optimizers (such as stochastic gradient descent SGD, RMSprop, etc.) and hyperparameter configurations that can achieve equivalent training results, depending on the training data size, hardware computing power, and actual convergence performance.

[0067] Through the aforementioned end-to-end joint training mechanism, the weight parameters of each module of the network are updated collaboratively under a unified optimization objective, enabling temporal information fusion, dual-modal feature fusion, and multi-scale aggregation to be jointly optimized for multi-target tracking tasks, thereby achieving deep complementarity of visible light and thermal infrared dual-modal information.

[0068] During the online inference phase, local extrema are extracted from the heatmap predicted in the current frame as candidate targets. Only predicted center points with a confidence score greater than a preset detection threshold (preferably 0.5 in this invention) are retained as valid candidate targets. For each valid candidate target, its center position is mapped back to the estimated position of the previous frame using its corresponding cross-frame displacement vector. Subsequently, a greedy matching strategy is used to calculate the distance score between the estimated position and the center points of all surviving historical trajectories in the previous frame, which is a conventional method in the field of multi-target tracking data association.

[0069] The trajectory status is determined and updated based on the distance score: when the distance score between the current valid candidate target and a certain historical trajectory is less than a preset association threshold (preferably 0.4 in this invention), the association is considered successful, and the target inherits the identity number of the corresponding historical trajectory; for valid candidate targets in the current frame that fail to match any historical trajectory, they are initialized as a completely new tracking trajectory; if a living historical trajectory fails to match any valid candidate target in the current frame, it is marked as unmatched and retained. When the number of consecutive unmatched frames of the historical trajectory exceeds a preset retention threshold, it is determined that the corresponding target has left the field of view or has been occluded for a long time, and the tracking process of the trajectory is terminated. The preset retention threshold is preferably 30 frames. Through the above target association and matching method, the model ultimately continuously outputs a continuous and stable tracking trajectory.

[0070] refer to Figure 7 The second embodiment of the present invention provides a visible light-thermal infrared vehicle multi-target tracking system 700 based on gating mechanism and local graph interaction, including an acquisition module 701, a temporal information fusion module 702, a dual-modal fusion module 703, a multi-scale aggregation module 704, and an output module 705. The functions of each module are described in detail below.

[0071] The acquisition module 701 is used to acquire the visible light image and thermal infrared image of the current moment and the previous moment, as well as the target thermal map of the previous moment. The temporal information fusion module 702 is used to perform cross-frame cross-attention fusion of the current frame shallow features and the previous frame shallow features in the visible light branch and the thermal infrared branch respectively to obtain temporal interaction features, and to use the target heat map generated by the previous moment to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features. The dual-modal fusion module 703 is used to input the visible light temporal fusion features and the thermal infrared temporal fusion features into a shared-weight backbone network to extract multi-scale features. Dual-modal fusion is performed on the visible light features from level 2 to level 5 and the thermal infrared features, while the level 0 and level 1 features are directly passed backward. The dual-modal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation on the two modal features; establishing a bidirectional cross-modal graph connection within a 3×3 local neighborhood; aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages; using Sigmoid gating to perform reliability modulation on the cross-modal messages and fusing them with the source modal feature residuals to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through a 1×1 convolution to obtain fused features. The multi-scale aggregation module 704 is used to perform multi-scale alignment and aggregation of the fused features from level 2 to level 5 to obtain unified features; The output module 705 is used to predict the target center heatmap, target width and height, center offset and cross-frame displacement based on the unified features, and output the vehicle multi-target tracking results in combination with the target association strategy.

[0072] It should be noted that although the above preferred embodiments have systematically described the present invention, for those skilled in the art, any adjustments, modifications or optimizations made to the above methods without departing from the principles and scope of the claims should also be considered within the protection scope of the present invention.

Claims

1. A visible light-thermal infrared vehicle multi-target tracking method based on gating mechanism and local graph interaction, characterized in that, include: Acquire visible light and thermal infrared images of the current and previous moments, as well as the target heat map of the previous moment; In the visible light branch and the thermal infrared branch respectively, the shallow features of the current frame and the shallow features of the previous frame are fused across frames to obtain temporal interaction features. The target heat map generated at the previous moment is used to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features. The visible light temporal fusion features and thermal infrared temporal fusion features are respectively input into the backbone network with shared weights to extract multi-scale features. The visible light features and thermal infrared features of levels 2 to 5 are fused in a dual-modal manner, and the level 0 and level 1 features are directly passed on to the next level. The bimodal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation of the two modal features; establishing a bidirectional cross-modal graph connection in a 3×3 local neighborhood, aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages, performing reliability modulation on the cross-modal messages using Sigmoid gating and fusing them with the residuals of the source modal features to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through a 1×1 convolution to obtain fused features; The fusion features from level 2 to level 5 are aligned and aggregated at multiple scales to obtain unified features; Based on the unified features, predict the target center heatmap, target width and height, center offset and cross-frame displacement, and output the vehicle multi-target tracking results in combination with the target association strategy.

2. The method as described in claim 1, characterized in that, The cross-frame attention fusion includes: mapping the shallow features of the current frame and the shallow features of the previous frame to a sequence of patch tokens through the patch embedding operator, superimposing positional encoding, performing cross-attention with the current frame token as the query and the previous frame token as the key and value, and restoring the original spatial resolution through inverse rearrangement, convolution transformation and upsampling.

3. The method as described in claim 1, characterized in that, The gating weights are generated as follows: the target heatmap from the previous time step is input into a mapping operator consisting of convolution, normalization, and nonlinear activation to obtain the heatmap prior features, and the prior features are then processed by convolution mapping and the Sigmoid activation function to generate gating weights.

4. The method as described in claim 1, characterized in that, The receptive fields of the Level 0 and Level 1 features are small, and their ability to express cross-modal semantic complementarity is limited. They are directly passed backward without performing bimodal fusion.

5. The method as described in claim 1, characterized in that, Establishing a bidirectional cross-modal graph connection within a 3×3 local neighborhood includes: for a feature node at any spatial location in the source modality features, taking that node as the source node; determining multiple neighboring nodes within the 3×3 local neighborhood corresponding to the spatial location in the target modality features; adding corresponding positional encoding information to the source node and each neighboring node; mapping the source node features with added positional encoding to a query vector through a query weight matrix, mapping each neighboring node feature with added positional encoding to a key vector through a key weight matrix, and mapping each neighboring node feature to a value vector through a value weight matrix; calculating attention weights based on the similarity between the query vector and each key vector, and weighting and aggregating each value vector according to the attention weights to obtain the cross-modal message.

6. The method as described in claim 1, characterized in that, The reliability modulation of the cross-modal message using Sigmoid gating includes: inputting the cross-modal message into a gating mapping function, and then processing it through a Sigmoid activation function to generate reliability gating coefficients; multiplying the reliability gating coefficients element-wise with the result of mapping the cross-modal message through the output weight matrix to obtain the gated modulated cross-modal message; adding the gated modulated cross-modal message to the source modal node features and processing it through a linear rectified activation function ReLU to obtain updated node features.

7. The method as described in claim 1, characterized in that, The multi-scale alignment and aggregation of the fused features from level 2 to level 5 includes: based on a deep aggregation network decoder architecture, a step-by-step feature aggregation method from high to low layers is adopted. Channel projection dimensionality reduction is performed on the high-level fused features. After spatial alignment and upsampling, the features are added element-wise with the adjacent low-level fused features. Then, geometric deformation correction and depth purification are performed through deformable convolution nodes until a unified feature with the same spatial resolution as the level 2 fused feature is output.

8. The method as described in claim 1, characterized in that, The multi-target vehicle tracking results output by combining the target association strategy include: extracting candidate targets with confidence scores greater than a preset threshold based on the heat map predicted in the current frame; mapping back to the estimated position of the previous frame using the cross-frame displacement vector corresponding to each candidate target; calculating the distance score with the historical trajectory using a greedy matching strategy; determining successful association and inheriting the identity number when the distance score is less than the preset association threshold; and terminating tracking when the number of consecutive unmatched frames of the historical trajectory exceeds the retention threshold.

9. A visible light-thermal infrared vehicle multi-target tracking system based on gating mechanism and local graph interaction, characterized in that, include: The acquisition module is used to acquire the visible light image and thermal infrared image of the current moment and the previous moment, as well as the target thermal map of the previous moment; The temporal information fusion module is used to perform cross-frame cross-attention fusion of the current frame shallow features and the previous frame shallow features in the visible light branch and the thermal infrared branch respectively to obtain temporal interaction features, and to use the target heat map generated in the previous moment to generate gating weights to perform element-wise gating modulation on the temporal interaction features to obtain visible light temporal fusion features and thermal infrared temporal fusion features. The dual-modal fusion module is used to input the visible light temporal fusion features and the thermal infrared temporal fusion features into the backbone network with shared weights to extract multi-scale features, perform dual-modal fusion on the visible light features and thermal infrared features from level 2 to level 5, and directly pass the level 0 and level 1 features to the back. The bimodal fusion includes: calculating complementary quality weights based on global statistical information to perform weighted modulation of the two modal features; establishing a bidirectional cross-modal graph connection in a 3×3 local neighborhood, aggregating neighborhood information through a graph attention mechanism to obtain cross-modal messages, performing reliability modulation on the cross-modal messages using Sigmoid gating and fusing them with the residuals of the source modal features to obtain bidirectional enhanced features; performing channel attention residual self-modulation on the bidirectional enhanced features and mapping them through a 1×1 convolution to obtain fused features; A multi-scale aggregation module is used to perform multi-scale alignment and aggregation of the fused features from level 2 to level 5 to obtain unified features; The output module is used to predict the target center heatmap, target width and height, center offset and cross-frame displacement based on the unified features, and output the vehicle multi-target tracking results in combination with the target association strategy.

Images