Target tracking system based on multi-modal deep feature proxy cross attention guidance

Abstract

The invention relates to a target tracking system based on multi-modal deep feature proxy cross attention guidance, and the method combines and uses a visible light image and a thermal infrared image, and improves the performance of a tracking algorithm under a complex illumination condition through an innovative proxy cross attention mechanism. The system comprises a multi-modal detector and a data associator, wherein the detector consists of a double-branch feature extraction network, a proxy cross attention feature enhancement module and a feature pyramid sharing convolution module; the data correlator comprises a neural Kalman filter and a low-confidence-coefficient-based detection box reuse strategy, the challenge of a complex motion scene is solved by dynamically adjusting process noise and observation noise parameters, and the long-term tracking performance is improved. According to the method, multi-modal data fusion is carried out on a deep feature level, so that target detection and tracking are optimized, and the tracking precision and stability in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically to a target tracking system based on multimodal deep feature agent cross-attention guidance. Background Art

[0002] Multimodal multi-object tracking (MMOT) fuses complementary information from visible light, thermal infrared, and low-light modalities to construct multi-target correlation trajectories across video frames, demonstrating significant advantages in robust perception in complex environments. Existing research primarily focuses on a hierarchical architecture for multimodal feature fusion: at the pixel level, a spatially spaced cross-sampling strategy is used to separate high-frequency details from low-frequency semantic features, combined with a dual-stream codec network to suppress modal noise and achieve cross-modal feature complementarity. At the shallow feature level, a multi-task unified correlation matching mechanism is proposed, integrating appearance similarity metrics, motion consistency constraints, and long-term memory reassociation modules to construct an end-to-end tracking and inference framework.

[0003] However, existing methods still face key challenges in the fusion of multimodal deep features: the differentiated distribution of multimodal deep semantics leads to insufficient mining of feature complementarity, the computational redundancy of traditional attention mechanisms restricts model performance, and fixed-parameter motion modeling methods are difficult to adapt to complex motion patterns in dynamic scenes.

[0004] Existing multimodal detectors focus on feature-level fusion, effectively combining the differential information of the two modalities. Transformer-based dual-cross attention models have become the mainstream approach for deep feature aggregation. While this approach achieves significant performance improvements, it also inevitably increases computational cost.

[0005] At the data association level, the Kalman filter has long been a core tool for motion modeling. The settings for its process noise and observation noise parameters directly impact prediction and update accuracy. However, the traditional method's reliance on preset parameters limits its adaptability to dynamic motion states.

[0006] Therefore, a new tracking method is needed that can more effectively fuse multimodal deep features, improve the computational efficiency of the attention mechanism, and achieve adaptive modeling of complex motion states. Summary of the Invention

[0007] To address the above problems, the present invention proposes a target tracking system and method based on agent cross-attention guidance with multimodal deep features. Based on the Agent Cross-attention guided multimodal multi-object tracking with Neural Kalman filter (ACNTrack) framework, multimodal data fusion is performed at the deep feature level, which optimizes target detection and tracking, and improves tracking accuracy and stability in complex environments.

[0008] The technical solution of the present invention is as follows: a target tracking system based on multimodal deep feature proxy cross-attention guidance, comprising a multimodal detector and a data associator, wherein the multimodal detector comprises a dual-branch feature extraction network, a proxy cross-attention feature enhancement module, a feature pyramid shared convolution module, and a prediction head module; and the data associator comprises a neural Kalman filter and a low-confidence detection box reuse module, wherein:

[0009] (1) The dual-branch feature extraction network processes visible light and thermal infrared images respectively to extract their respective feature representations;

[0010] (2) The proxy cross-attention feature enhancement module interactively fuses the dual-branch features and captures the complementary relationship between modalities through the proxy attention mechanism;

[0011] Given input First, each feature map is converted into a feature sequence by flattening operation and a learnable position code is added. The position code is a trainable parameter of dimension H×W×C, which is used to encode the spatial information between feature sequences. The feature sequence T after position coding is tir and As the input of the ACFE module; the two agent cross attention modules use independent parameter sets and do not share weights. The mathematical expression is as follows:

[0012]

[0013] Where T tir With T rgb represent the input thermal infrared and visible light features, To output thermal infrared features, Characterize the ACFE module function of the thermal infrared branch;

[0014] The specific operation process of the ACFE module is as follows: First, the thermal infrared modal feature sequence T tir Projection is the value matrix V tir With the key matrix At the same time, the visible light modal feature sequence Trgb Convert to query matrix

[0015] Q rgb =T rgb W Q ,V tir =T tir W V ,K tir =T tir W K

[0016] in is the weight matrix;

[0017] The proxy attention mechanism can be expressed as:

[0018] O A =Attn S (Q rgb ,A,Attn s (A,K tir ,V tir ))

[0019] Its equivalent form is:

[0020]

[0021] In the formula is the newly defined agent feature sequence, σ(·) represents the softmax function, T represents the matrix transpose, and the function Attn s represents the attention similarity mechanism function, It represents the attention similarity mechanism function based on Q and K;

[0022] The proxy attention mechanism includes two softmax-based attention calculation processes: proxy aggregation and proxy broadcast. First, the proxy word A is specified as the query, and A, K are executed. tir With V tir The attention information is weighted to aggregate the interaction features V A ; then V A is the value, A is the key, and the query matrix Q is used rgb Perform a secondary attention weighted calculation to broadcast the interactive agent feature information to all query tokens and generate output O; introduce agent bias into the attention mechanism:

[0023] O A =σ(Q rgb A T +B2)σ(AK tir T +B1)V tir

[0024] In the formula and is the proxy bias term;

[0025] The deep convolution module is introduced to enhance features. The ACFE module consists of three parts: DWConv, proxy bias and proxy attention. The calculation formula is:

[0026]

[0027] in

[0028] Adopt hybrid pooling to adaptively aggregate features:

[0029] A a =AvgPooling(Q rgb ),A m =MaxPooling(Q rgb )

[0030] A=αA a +(1-α)·A m

[0031] Where A a With A m is the pooled compression feature, and α is a learnable weight parameter;

[0032] The second ACFE module enhances the visible branch characteristics:

[0033]

[0034] Finally, the feature fusion function Ψ fusion (·) The output is:

[0035]

[0036] (3) The feature pyramid shared convolution module extracts multi-scale features through convolution kernels with different expansion rates to enhance the contextual information of a single modality;

[0037] (4) The prediction head module is responsible for outputting the location, category and size information of the target;

[0038] (5) The neural Kalman filter dynamically adjusts the process noise and observation noise parameters through the neural network to improve the modeling ability of complex motion states;

[0039] (6) The low-confidence detection box reuse module improves detection efficiency through a three-stage matching strategy. The specific process is as follows:

[0040] 1) First, the detection box D k According to the confidence threshold τ, it is divided into high-score detection boxes D highand low-score detection box D low ;

[0041] 2) First stage high confidence matching: calculating trajectory With high score detection box D high The IoU distance and feature distance between them generate the comprehensive cost matrix C high , matching is completed through the Hungarian algorithm;

[0042] 3) Second stage low confidence matching: the remaining trajectory T remain Compared with the low-score detection box D low Perform matching and generate the cost matrix C through IoU distance low , again using the Hungarian algorithm for matching;

[0043] 4) The third stage of uncertain trajectory matching: Match the unconfirmed trajectory T unconfirmed Unmatched detection D remain Perform matching and update the status of unmatched trajectories;

[0044] 5) Reuse of low-confidence detection frames: Select detection frames with confidence exceeding the threshold η from the remaining detection frames to initialize new trajectories to improve detection utilization efficiency.

[0045] Furthermore: In the dual-branch feature extraction network, the thermal infrared and visible light images are first processed independently, as shown below:

[0046]

[0047] In the formula They represent the feature maps output by the i-th layer (i=1, 2, 3, 4, 5) of the thermal infrared and visible light branches respectively. The input thermal infrared and visible light images are Φ backbone is the feature extraction function; the backbone network extracts feature clustering information under different receptive fields through multi-scale features; then, the features of different receptive fields are and Input feature fusion module as follows:

[0048]

[0049] In the formula Represents the fusion features of the i-th layer (i=3,4,5), the feature fusion function Ψ fusion (·) Integrate multimodal information through convolutional layers P3-P5;

[0050] The final stage is to collect feature maps Passed to the neck network, and then input into the prediction head to complete the regression and classification tasks:

[0051]

[0052] D cls Represents the classification branch, responsible for the category prediction of the target; D bbox Represents the bounding box regression branch, which is responsible for predicting the precise location and size of the object.

[0053] Furthermore, the feature pyramid shared convolution module implements multi-scale feature extraction through multi-expansion rate convolution with shared weights. The process is as follows:

[0054] First, dimensionality reduction is performed through 1×1 convolution:

[0055]

[0056] Where, Represents the convolution output of the thermal infrared feature branch with an expansion rate of 1, Indicates the thermal infrared branch feature at the P5 stage of the network layer, c1 represents the input feature dimension, c hidden Represents the output feature dimension;

[0057] Then, a series of dilated convolutions are performed using a shared weight matrix W and different dilation rates d for feature extraction:

[0058]

[0059] Where, The pyramid feature representing the expansion rate d, Represents the feature map of the previous level of the pyramid, where the filling P calculation formula is:

[0060]

[0061] Where K represents the convolution kernel size;

[0062] Output features of different expansion rates are concatenated along the channel dimension:

[0063]

[0064] In the formula Represent the pyramid feature maps with expansion rates of 1, d1, d3, and d5 respectively;

[0065] Finally, the concatenated features are reduced in dimension through 1×1 convolution:

[0066]

[0067] In the formula represents the output feature map of the feature pyramid shared convolution module, and c2 represents the output dimension of the entire module;

[0068] The final FPS-Conv operation is defined as:

[0069]

[0070] where d k is the expansion rate of the kth layer.

[0071] Further: The neural Kalman filter learns the noise factor parameters by constructing two fully connected neural networks, where:

[0072] Process noise covariance matrix Q k Depend on and Determine the observation noise covariance matrix R k Depend on Decide;

[0073] The neural network dynamically adjusts these parameters based on the current state of the target:

[0074]

[0075] in and is a neural network composed of fully connected layers; and is the noise factor representing position and acceleration, x k-1 represents the state vector at time k-1, z k Then it represents the observation vector at time k;

[0076] To ensure the positive definiteness of the covariance matrix, add a stability constraint:

[0077]

[0078] Where ∈ is the preset lower bound, σ pi ,σ vi ,σ mi is the noise factor representing position and acceleration, where i∈[1,2,3,4];

[0079] Further: The motion compensation mechanism in the data association process is as follows:

[0080] First calculate the motion compensation matrix between adjacent frames This is achieved through the function findMotion(·);

[0081] Trajectory Set Each trajectory t in t is predicted by applying a neural Kalman filter, and then a motion compensation operation MotionCompensation(·) is performed;

[0082] Motion compensation is used to enhance cross-frame correlation consistency and improve adaptability in dynamic scenes.

[0083] The contributions of this invention are as follows:

[0084] 1. Proxy Cross-Attention Mechanism: This paper proposes an agent cross-attention guided feature enhancement module (ACFE) to efficiently aggregate multi-level feature information of RGB and TIR modalities. The agent cross-attention feature enhancement module captures the complementary relationship between thermal infrared and visible light modalities, and improves the efficiency of long-distance association of deep cross-modal features. The module consists of three parts: agent attention, agent bias, and deep convolution. Through two softmax-based attention calculation processes, agent aggregation and agent broadcast, it avoids the pairwise similarity calculation between the query matrix and the key matrix, reduces the computational complexity, and enhances the diversity of features through the deep convolution module.

[0085] For the first time, the proxy attention mechanism is proposed to be applied to multimodal target tracking. Through the two steps of proxy aggregation and proxy broadcasting, the deep features of visible light and thermal infrared modalities are efficiently integrated, the computational complexity is reduced, and the feature complementarity is improved.

[0086] 2. Feature Pyramid Shared Convolution: An innovative multi-scale feature extraction module is proposed: Feature Pyramid Shared Convolution (FPS-Conv) module to replace the traditional fast spatial pyramid pooling (Spatial Pyramid Pooling Fast, SPPF): By introducing a convolution layer with a variable expansion rate, this module can capture multi-scale features and enhance the ability to extract multi-scale contextual information of a single modality. Traditional SPPF pooling realizes multi-scale feature extraction, but its single feature aggregation method limits the representation of complex scenes. The FPS-Conv operation proposed in the present invention extracts multi-scale features through convolution kernels with different expansion rates, effectively expands the receptive field and retains information continuity, and aggregates multi-granularity features through a parallel multi-branch structure, so that the model can further capture local details and multi-scale contextual information of a single modality. Capturing multi-level feature information through convolution kernels with different expansion rates enhances the representation ability of a single modality in complex scenes.

[0087] 3. Neural Kalman Filter (NKF): A neural network-based adaptive Kalman filter that uses the neural network to dynamically adjust process noise and observation noise parameters. The process noise and observation noise parameters in traditional Kalman filters are typically determined by preset constants, making them difficult to adapt to complex dynamic scenarios. This invention constructs two fully connected neural networks to dynamically learn these parameters, adaptively adjusting the Kalman filter parameters based on the current motion state. By learning and dynamically adjusting the process noise and observation noise parameters, the invention improves the ability to model complex motion patterns and enhances the accuracy and stability of data association.

[0088] Fourth, the data associator uses a three-stage progressive data association strategy, combined with motion compensation and feature fusion techniques to achieve robust target tracking. Furthermore, through a low-confidence detection box reuse strategy, detection efficiency is improved, enabling robust and efficient multi-target tracking in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A block diagram of a neural Kalman tracking algorithm according to an embodiment of the present invention;

[0090] Figure 2 This is a diagram of the neural network structure of a detector based on visible light and thermal infrared modal fusion according to an embodiment of the present invention;

[0091] Figure 3 This is a structural diagram of a feature fusion module according to an embodiment of the present invention;

[0092] Figure 4 This is a visual display of the multi-target tracking results of an embodiment of the present invention. DETAILED DESCRIPTION

[0093] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments. Obviously, the embodiments described are only some examples of the present invention, not all examples. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without expending creative work are within the scope of protection of the present invention.

[0094] The present embodiment provides a target tracking system and method based on multimodal deep feature agent cross attention guidance, including a multimodal detector and a data associator, such as Figure 1 The multimodal detector includes a dual-branch feature extraction network, a proxy cross-attention feature enhancement module, a feature pyramid shared convolution module, and a prediction head module; the data associator includes a neural Kalman filter and a low-confidence detection box reuse module.

[0095] The first step in the algorithm of this embodiment is to build a suitable hardware platform. This embodiment uses an Intel Core i9-13900K processor (main frequency 3.0GHz) and dual NVIDIA GeForce RTX 3090 graphics cards to implement the algorithm based on PyTorch 2.0.0 and Python 3.10.15.

[0096] The construction and experimental verification of the algorithm of this embodiment include the following aspects: 1 Multimodal multi-target tracking framework with deep feature fusion

[0097] The overall framework of ACNTrack tracker is as follows Figure 1 As shown in the figure, the system follows the detection-tracking paradigm and consists of two main components: a multimodal detector and a data associator. The multimodal detector uses a dual-branch backbone network to extract features from visible light and thermal infrared images respectively. Its processing pipeline mainly includes the following stages: Feature Pyramid Shared Convolution (FPS-Conv) to extract multi-scale features from a single modality, proxy cross-attention to aggregate cross-modal features, and finally outputs the detection results through the network neck and prediction head for subsequent association.

[0098] The ACNTrack tracking algorithm uses a three-stage progressive data association strategy, combined with motion compensation and feature fusion technology to achieve robust target tracking. The core process of the data association process is as follows:

[0099] (1) Initialization and input configuration: The algorithm input includes multimodal video sequences RGB&TIR(&L 2 ), multimodal detector Det, neural Kalman filter NKF, appearance feature extractor Enc, as well as high confidence threshold τ and new trajectory threshold η. Initialization stage creates an empty trajectory set And process the input video stream frame by frame.

[0100] (2) Multimodal detection and feature extraction: For the current frame Call the detector Det to obtain the detection set According to the threshold τ, it is divided into high-score detection and low score detection At the same time, for each detection frame d, the appearance features in RGB mode are extracted by the feature extractor Enc and stored as F high .

[0101] (3) Motion prediction and compensation: Calculate the motion compensation matrix between adjacent frames through the function findMotion(·) Used to correct cross-frame motion deviation. For each trajectory t in , the neural Kalman filter NKF is first applied to perform state prediction, and then the motion compensation operation MotionCompensation(·) is performed to improve the adaptability of the motion model in dynamic scenes.

[0102] (4) High confidence detection association: Calculate the trajectory prediction box and The IoU distance cost matrix C iou =IOUDist(·), while integrating trajectory history features With the current frame feature F high Generate feature distance cost matrix C emb =FusionDist(·). The comprehensive cost matrix C is constructed by taking the minimum value operation high =min(C iou ,C emb ), and the Hungarian algorithm is used to complete the first matching. The unmatched tracks and detections are recorded as and

[0103] (5) Low confidence detection association: for the remaining tracks Calculate its difference with low score detection The IoU distance cost matrix C low , and the Hungarian algorithm is used again for secondary matching. This stage focuses on recovering missed targets caused by occlusion or deformation, and the unmatched trajectory is obtained after updating. and remaining detection

[0104] (6) Uncertain trajectory matching and trajectory maintenance: For unconfirmed trajectories Calculate the number of unmatched detections with high confidence The IoU distance C iou , and integrate global trajectory features With F high Generate feature distance C emb , construct the final cost matrix C high =min(C iou ,C emb ) for a third match. Tracks that continue to be unmatched are maintained using NKF prediction. Tracking is terminated if no match occurs for more than 40 frames. Successfully matched tracks have their motion state and appearance features updated to maintain the model's robustness to changes in target appearance.

[0105] (7) New trajectory initialization and reuse of low confidence detection: and low confidence remaining detections Filter the detection boxes whose confidence exceeds the threshold τ, initialize the new trajectory and add it to the set This mechanism effectively suppresses false alarms caused by detection noise while ensuring the timely capture of newly emerging targets.

[0106] (8) Output and iteration: Output the updated trajectory set The algorithm iteratively processes subsequent video frames. It uses motion compensation to enhance cross-frame association consistency, feature fusion to reduce the probability of identity jumps, and a three-stage matching strategy to improve detection utilization efficiency, ultimately achieving stable tracking in complex multimodal scenarios.

[0107] 2 Multimodal Object Detector

[0108] The multimodal detector adopts a dual-branch backbone network structure, such as Figure 2 First, the thermal infrared and visible light images are processed independently as shown below:

[0109]

[0110] In the formula They represent the feature maps output by the i-th layer of the thermal infrared and visible light branches (i = 1, 2, 3, 4, 5 represent different layers of the network), and the input thermal infrared and visible light images are Feature extraction function Φ backbone The backbone network is widely used in standard target detectors, such as CSPDarkNet. The backbone network extracts feature clustering information under different receptive fields through multi-scale features. Subsequently, the features of different receptive fields are combined. and Input feature fusion module as follows:

[0111]

[0112] In the formula Represents the fusion features of the i-th layer (i=3,4,5), the feature fusion function Ψ fusion (·) Integrate multimodal information through convolutional layers P3 to P5 (e.g. Figure 2 shown).

[0113] The final stage is to collect feature maps Passed to the neck network, and then input into the prediction head to complete the regression and classification tasks:

[0114]

[0115] 3-Agent Cross-Attention Feature Enhancement Module

[0116] The agent cross attention feature enhancement module in this embodiment is as follows Figure 3 As shown in the figure, this module captures the complementary relationship between thermal infrared and visible light modalities, improving the efficiency of long-distance correlation of deep cross-modal features.

[0117] Given input F tir and First, each feature map is converted into a feature sequence by flattening and a learnable position encoding is added. The position encoding is a trainable parameter of dimension H×W×C, which is used to encode the spatial information between feature sequences. The position-encoded feature sequence T tir and As the input of the ACFE module. The two agent cross attention modules use independent parameter sets and do not share weights. For clarity, Figure 3 Only the calculation process of the ACFE module of the thermal infrared branch is shown. Its mathematical expression is as follows:

[0118]

[0119] Where T tir With T rgb represent the input thermal infrared and visible light features, To output thermal infrared features, Characterize the ACFE module function of the thermal infrared branch.

[0120] The specific operation process of the ACFE module is as follows: First, the thermal infrared modal feature sequence T tir Projection is the value matrix V tir With the key matrix At the same time, the visible light modal feature sequence T rgb Convert to query matrix

[0121] Q rgb =T rgb W Q ,V tir =T tir W V ,K tir =T tir W K

[0122] in is the weight matrix.

[0123] The proxy attention mechanism can be expressed as:

[0124] O A =Attn S (Q rgb ,A,Attn S (A,K tir ,V tir ))

[0125] Its equivalent form is:

[0126]

[0127] In the formula is the newly defined proxy feature sequence, and σ(·) represents the softmax function.

[0128] like Figure 3 As shown in the figure, the proposed proxy attention mechanism includes two softmax-based attention calculation processes: proxy aggregation and proxy broadcast. Specifically, first specify the proxy word A as the query, execute A, K tir With V tir The attention information is weighted to aggregate the interaction features V A ; then V A is the value, A is the key, and the query matrix Q is used rgb Perform a secondary attention weighted calculation to broadcast the interactive agent feature information to all query words and generate output O. In this way, Q rgb and K tir The pairwise similarity between the two is calculated, while the information exchange between each query-key pair is still achieved through the proxy token. Proxy attention combines the advantages of softmax linear attention to achieve performance improvement. To optimize the use of position information, a proxy bias is introduced in the attention mechanism:

[0129] O A =σ(Q rgb A T +B2)σ(AK tir T +B1)V tir

[0130] In the formula and is the proxy bias term.

[0131] Although proxy attention has the advantages of low computational complexity and high expressiveness of linear attention, it may lose feature diversity. Therefore, the depthwise convolution (DWConv) module is introduced to enhance features. Figure 3 As shown in Figure 2, the ACFE module consists of three parts: DWConv, proxy bias, and proxy attention. The calculation formula is:

[0132]

[0133] in Under the default settings, proxy token A is obtained through average pooling. Average pooling effectively preserves image background information, while maximum pooling is better at extracting salient features such as texture. To effectively combine the advantages of both and extract more information, hybrid pooling is used to adaptively aggregate features:

[0134] A a =AvgPooling(Qrgb ),A m =MaxPooling(Q rgb )

[0135] A=αA a +(1-α)·A m

[0136] Where A a With A m is the pooled compression feature, and α is a learnable weight parameter.

[0137] Similarly, the second ACFE module enhances the visible light branch characteristics:

[0138]

[0139] Finally, the feature fusion function Ψ fusion (·) The output is:

[0140]

[0141] 4 Feature Pyramid Shared Convolution

[0142] The feature pyramid shared convolution (FPS-Conv) operation is as follows Figure 2 As shown in Figure 2, this module extracts multi-scale features through convolution kernels with different expansion rates, effectively expanding the receptive field and retaining information continuity. It aggregates multi-granularity features through a parallel multi-branch structure, enabling the model to further capture single modality local details and multi-scale context information. Taking the TIR modality as an example, the input feature map is input feature map After 1×1 convolution, the dimension is reduced to aisle

[0143]

[0144] A series of dilated convolutions are performed using a shared weight matrix W and different dilation rates d. The dilation rate controls the sampling interval of the convolution kernel, capturing more multi-scale information while maintaining the receptive field:

[0145]

[0146] The calculation formula for filling P is:

[0147]

[0148] Output features of different expansion rates are concatenated along the channel dimension:

[0149]

[0150] Finally, the concatenated features are reduced in dimension through 1×1 convolution:

[0151]

[0152] The final FPS-Conv operation is defined as:

[0153]

[0154] where d k is the k-th layer dilation rate. Through convolution with multiple dilation rates, FPS-Conv further extracts multi-scale contextual information. Compared to traditional pooling, which aggregates single features, FPS-Conv achieves multi-scale feature fusion, enhancing the network's ability to represent multiple scenes in a single modality.

[0155] 5. Neural Kalman Filter

[0156] Neural Kalman filter such as Figure 1 As shown in the figure, it dynamically adjusts the process noise and observation noise parameters through a neural network to improve the performance of the traditional Kalman filter in complex motion scenes.

[0157] In the field of multi-target tracking, discrete Kalman filters are often used to model target motion in the image plane, and their true state vectors are usually defined as seven-tuples or eight-tuples. This embodiment uses eight-tuples to represent the target state:

[0158]

[0159] The corresponding observation vector is defined as:

[0160]

[0161] By building two fully connected neural networks to learn 12 parameters σ, the dynamic update is based on the current motion state. and

[0162]

[0163] in and It is a neural network composed of fully connected layers. and Dynamically adjust Q k With R k To enhance motion modeling capabilities. and It can be expressed as:

[0164]

[0165] In order to ensure that the covariance matrix remains positive during Cholesky decomposition, a stability constraint is added to the output weights:

[0166]

[0167] Where ∈ is the preset lower bound (position σ p and σ m is 0.01, speed σ v is 0.001), this constraint ensures the matrix is ​​positive definite.

[0168] 6 Experimental platform and parameter settings

[0169] The algorithm in this example is evaluated on the RGBT multi-object tracking task. The VT-MOT and UniRTL datasets are used for this task. The HOTA, DetA, AssA, MOTA, and IDF1 metrics are used for the multimodal visual multi-object tracking task.

[0170] The algorithm multimodal target detection task in this embodiment is set to train 60 rounds and batch size 24, and adopts stochastic gradient descent optimizer (momentum 0.9, weight decay 5×10 -4 ); In the multi-target tracking task, the NKF network was trained for 300 rounds with an initial learning rate of 1×10 -4 , Adam optimizer is selected, batch size is 24. The high confidence threshold τ is 0.5, and the new trajectory threshold η is 0.4.

[0171] 5 Experimental results

[0172] By comparing with the latest single-target tracking algorithms, the proposed algorithm in this embodiment is further evaluated for its multi-target tracking accuracy and robustness on the VT-MOT and UniRTL datasets. The performance of various multi-target tracking algorithms is shown in Table 1:

[0173] Table 1

[0174]

[0175]

[0176] It can be seen that the method of this embodiment has competitive performance. Similarly, it is compared with the currently popular multi-target tracking methods on the UniRTL dataset, as shown in Table 2.

[0177] Table 2

[0178]

[0179] Table 2 shows the advantages of the method of this embodiment in multi-target tracking. It is competitive with the latest methods and surpasses single-modality tracking algorithms.

[0180] Table 3 shows the different modal fusion tracking results of the ACNTrack method of this embodiment on the UniRTL dataset. The results show that the method of this embodiment can perform well in all modal fusions.

[0181] Table 3

[0182]

[0183] like Figure 4 As shown in the figure, the ACNTrack method of this embodiment performs well in tracking visualization for multi-target tracking tasks in challenging scenarios such as occlusion and low illumination. This demonstrates the effectiveness of the proposed proxy cross-attention mechanism, feature pyramid shared convolution, and neural Kalman filter, enabling high-precision and high-stability multi-target tracking in complex environments.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target tracking system based on multimodal deep feature proxy cross-attention guidance, characterized by: It includes a multimodal detector and a data associator. The multimodal detector includes a dual-branch feature extraction network, a proxy cross-attention feature enhancement module, a feature pyramid shared convolution module, and a prediction head module. The data associator includes a neural Kalman filter and a low-confidence detection box reuse module, wherein: (1) The dual-branch feature extraction network processes visible light and thermal infrared images respectively to extract their respective feature representations; (2) The proxy cross-attention feature enhancement module interactively fuses the dual-branch features and captures the complementary relationship between modalities through the proxy attention mechanism; Given input First, each feature map is converted into a feature sequence by flattening operation and a learnable position code is added. The position code is a trainable parameter of dimension H×W×C, which is used to encode the spatial information between feature sequences. The feature sequence T after position coding is tir and As the input of the ACFE module; the two agent cross attention modules use independent parameter sets and do not share weights. The mathematical expression is as follows: Where T tir With T rgb represent the input thermal infrared and visible light features respectively, To output thermal infrared features, Characterize the ACFE module function of the thermal infrared branch; The specific operation process of the ACFE module is as follows: First, the thermal infrared modal feature sequence T tir Projection is the value matrix V tir With the key matrix At the same time, the visible light modal feature sequence T rgb Convert to query matrix Q rgb =T rgb W Q ,V tir =T tir W V ,K tir =T tir W K in is the weight matrix; The proxy attention mechanism can be expressed as: O A =Attn S (Q rgb ,A,Attn S (A,K tir ,V tir )) Its equivalent form is: In the formula is the newly defined agent feature sequence, σ(·) represents the softmax function, T represents the matrix transpose, and the function Attn S represents the attention similarity mechanism function, It represents the attention similarity mechanism function based on Q and K; The proxy attention mechanism includes two softmax-based attention calculation processes: proxy aggregation and proxy broadcast. First, the proxy word A is specified as the query, and A, K are executed. tir With V tir The attention information is weighted to aggregate the interaction features V A ; then V A is the value, A is the key, and the query matrix Q is used rgb Perform a secondary attention weighted calculation to broadcast the interactive agent feature information to all query tokens and generate output O; introduce agent bias into the attention mechanism: About A =σ(Q rgb And T +B2)σ(AK tir T +B1)V tir In the formula and is the proxy bias term; The deep convolution module is introduced to enhance features. The ACFE module consists of three parts: DWConv, proxy bias and proxy attention. The calculation formula is: in Adopt hybrid pooling to adaptively aggregate features: A a =AvgPooling(Q rgb ),A m =MaxPooling(Q rgb ) A=αA a +(1-a)·A m Where A a With A m is the pooled compression feature, and α is a learnable weight parameter; The second ACFE module enhances the visible branch characteristics: Finally, the feature fusion function Ψ fusion (·) The output is: (3) The feature pyramid shared convolution module extracts multi-scale features through convolution kernels with different expansion rates to enhance the contextual information of a single modality; (4) The prediction head module is responsible for outputting the location, category and size information of the target; (5) The neural Kalman filter dynamically adjusts the process noise and observation noise parameters through the neural network to improve the modeling ability of complex motion states; (6) The low-confidence detection box reuse module improves detection efficiency through a three-stage matching strategy. The specific process is as follows: 1) First, the detection box D k According to the confidence threshold τ, it is divided into high-score detection boxes D high and low-score detection box D low ; 2) First stage high confidence matching: calculating trajectory With high score detection box D high The IoU distance and feature distance between them generate the comprehensive cost matrix C high , matching is completed through the Hungarian algorithm; 3) Second stage low confidence matching: the remaining trajectory T remain Compared with the low-score detection box D low Perform matching and generate the cost matrix C through IoU distance low , again using the Hungarian algorithm for matching; 4) The third stage of uncertain trajectory matching: Match the unconfirmed trajectory T unconfirmed Unmatched detection D remain Perform matching and update the status of unmatched trajectories; 5) Reuse of low-confidence detection frames: Select detection frames with confidence exceeding the threshold η from the remaining detection frames to initialize new trajectories to improve detection utilization efficiency.

2. The target tracking system based on multimodal deep feature proxy cross-attention guidance according to claim 1 is characterized by: In the dual-branch feature extraction network, the thermal infrared and visible light images are first processed independently, as shown below: In the formula They represent the feature maps output by the i-th layer (i=1, 2, 3, 4, 5) of the thermal infrared and visible light branches respectively. The input thermal infrared and visible light images are Φ backbone is the feature extraction function; the backbone network extracts feature clustering information under different receptive fields through multi-scale features; then, the features of different receptive fields are and Input feature fusion module as follows: In the formula Represents the fusion features of the i-th layer (i=3,4,5), the feature fusion function Ψ fusion (·) Integrate multimodal information through convolutional layers P3-P5; The final stage is to collect feature maps Passed to the neck network, and then input into the prediction head to complete the regression and classification tasks: D cls Represents the classification branch, responsible for the category prediction of the target; D bbox Represents the bounding box regression branch, which is responsible for predicting the precise location and size of the object.

3. The target tracking system based on multimodal deep feature proxy cross attention guidance according to claim 1, characterized in that: The feature pyramid shared convolution module implements multi-scale feature extraction through weight-sharing multi-expansion rate convolution. Taking the thermal infrared feature branch as an example, the process is as follows: First, dimensionality reduction is performed through 1×1 convolution: Where, Represents the convolution output of the thermal infrared feature branch with an expansion rate of 1, Indicates the thermal infrared branch feature at the P5 stage of the network layer, c1 represents the input feature dimension, c hidden Represents the output feature dimension; Then, a series of dilated convolutions are performed using a shared weight matrix W and different dilation rates d for feature extraction: Where, The pyramid feature representing the expansion rate d, Represents the feature map of the previous level of the pyramid, where the filling P calculation formula is: Where K represents the convolution kernel size; Output features of different expansion rates are concatenated along the channel dimension: In the formula Represent the pyramid feature maps with expansion rates of 1, d1, d3, and d5 respectively; Finally, the concatenated features are reduced in dimension through 1×1 convolution: In the formula represents the output feature map of the feature pyramid shared convolution module, and c2 represents the output dimension of the entire module; The final FPS-Conv operation is defined as: where d k is the expansion rate of the kth layer.

4. The target tracking system based on multimodal deep feature proxy cross attention guidance according to claim 1, characterized in that: The Neural Kalman Filter learns the noise factor parameters by constructing two fully connected neural networks, where: Process noise covariance matrix Q k Depend on and Determine the observation noise covariance matrix R k Depend on Decide; The neural network dynamically adjusts these parameters based on the current state of the target: in and A neural network consisting of fully connected layers; To ensure the positive definiteness of the covariance matrix, add a stability constraint: Where ∈ is the preset lower bound.

5. The target tracking system based on multimodal deep feature proxy cross attention guidance according to claim 1, characterized in that: The motion compensation mechanism in the data association process is as follows: First calculate the motion compensation matrix between adjacent frames This is achieved through the function findMotion(·); Trajectory Set Each trajectory t in t is predicted by applying a neural Kalman filter, and then a motion compensation operation MotionCompensation(·) is performed; Motion compensation is used to enhance cross-frame correlation consistency and improve adaptability in dynamic scenes.

Images