RGBT target tracking method based on multilayer global feature fusion and mapping template updating

Abstract

The invention discloses an RGBT target tracking method based on multilayer global feature fusion and mapping template updating, and aims to solve the problem that a target is easy to lose in tracking under the conditions of poor illumination and target deformation. A model proposed by the method comprises four main parts: a twin structure-based feature extraction network, a Transform feature fusion network, a tracking result prediction network and a template updating network. According to the method, a constructed network is pre-trained by using a visible light data set and then fine-tuned by using an RGBT data set, and finally a to-be-tracked target in a video sequence is obtained and tracked by using a trained model. Through the effective feature fusion algorithm and the template updating method, the target can be effectively tracked under the complex background condition, and the tracking robustness and accuracy of the target are ensured.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to an RGBT target tracking method based on multi-layer global feature fusion and mapping template updating. Background Art

[0002] In the field of computer vision, target tracking technology has been widely used in traffic management, human-computer interaction, national defense and military. However, most existing trackers rely on single-modal data, which often leads to insufficient robustness and tracking performance in complex environments. Recent studies have shown that the use of multimodal information can significantly improve the performance of trackers. RGBT target tracking can solve the problem that traditional visible light target tracking methods are prone to losing targets under poor lighting conditions by integrating the complementary advantages of visible light data and infrared data. Therefore, the complementarity of visible light and infrared features plays a key role in improving the performance of target trackers. In addition, the target template with only the initial frame is difficult to cope with the complex tracking of target deformation. Therefore, the update of the template enables the target tracker to cope with various complex environments more effectively. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an RGBT target tracking method based on multi-layer global feature fusion and mapping template updating, wherein the RGBT target tracking method can achieve robust tracking of targets in complex backgrounds.

[0004] The technical solution of the present invention to solve the above technical problems is:

[0005] The RGBT target tracking method based on multi-layer global feature fusion and mapping template update includes the following steps:

[0006] (S1) Build a target tracking network: The model includes a twin structure feature extraction network, a Transformer feature fusion network, and a tracking result prediction network;

[0007] (S2) Model training and fine-tuning: Use the pre-training data set to pre-train the feature extraction network based on the twin structure, and use the stochastic gradient descent (SGD) method to train until the loss value converges to obtain the pre-trained model; use the training data set to fine-tune the pre-trained model, and use SGD to train until the loss value converges to obtain the trained network;

[0008] (S3) Target tracking stage: obtain the first frame of the visible light and infrared image sequence of the RGBT test dataset, select the target to be tracked, and input the trained RGBT target tracking model based on the twin structure and Transformer feature fusion to generate the target tracking result; according to the information of the tracking result, directly map it back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

[0009] Preferably, in step (S1), the construction of the twin structure feature extraction network, Transformer feature fusion network and tracking result prediction network includes the following steps:

[0010] (S1-1) Constructing a twin structure feature extraction network: The twin structure-based feature extraction network adopts a deep symmetrical structure, which is composed of 4 modified ResNet50s;

[0011] (S1-2) Construct Transformer feature fusion network: Based on the modification of Transformer network, realize the fusion of multi-modal and multi-level features; finally, obtain the response map of tracking results through cross-correlation operation;

[0012] (S1-3) Construct a tracking result prediction network: The tracking result prediction network contains three branches: a classification branch that predicts the category of each position in the response map; a prediction box regression branch that predicts the target position; and a centrality branch that predicts the centrality score of each position.

[0013] (S1-4) Constructing the template update network: This includes the mapping calculation of the coordinate positions in the tracking results. Through the characteristics of convolution and Transformer, the tracking result coordinates are mapped back to the corresponding positions of the fused multi-level templates.

[0014] Preferably, in step (S2), pre-training and fine-tuning of the pre-trained model are performed based on the feature extraction network of the twin structure, including the following steps:

[0015] (S2-1), the feature extraction network includes 4 ResNet50s with the same structure, and every 2 ResNet50s form a pair of twin structure feature extraction networks. Use the visible light data structure as a pre-training data set to pre-train one pair of twin structure feature extraction networks, use SGD to optimize the tracking model until the model loss value converges, and finally save the pre-trained model;

[0016] (S2-2) Use the pre-trained model parameters saved in step (S2-1) to initialize the feature extraction network, freeze the first two layers of the ResNet50 network, use the RGBT training dataset to fine-tune the tracking model, and at the same time reduce the learning rate of the network. Use SGD optimization until the model loss value converges to obtain a trained network.

[0017] Preferably, in step (S3), obtaining a target template to be tracked in the visible light and infrared images of the RGBT test data set and tracking the target template comprises the following steps:

[0018] (S3-1), the template image is centered on the target selected in the first frame of the visible light and infrared image sequence, and 127×127 pixels are cropped. The subsequent frames are search images, and the search image is 255×255 pixels;

[0019] (S3-2), inputting the visible light and infrared template graphs and search graphs obtained in step (S3-1) into the feature extraction network of the twin structure respectively, to obtain multi-layer visible light and infrared template features and search features;

[0020] (S3-3) Using the Transformer feature fusion network, the multi-layer visible light and infrared template features and the visible light and infrared search features obtained in step (S3-2) are fused to generate multi-layer fused template features and fused search features.

[0021] (S3-4), performing cross-correlation operations on the fused template features and fused search features of the same layer in pairs to obtain a multi-layer response map for tracking prediction;

[0022] (S3-5) Concatenate the multiple layers of response maps and input them into the tracking result prediction network to generate a 6-dimensional vector f = (cls, cen, l, t, r, b) to predict the target position in the visible light and infrared images, where cls is the classification score, cen is the center score, l+r and t+b are the width and height of the predicted target in the current frame.

[0023] (S3-6) Mapping the coordinate information of the predicted tracking result back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

[0024] Preferably, in step (S1-1), the downsampling operations in the last two convolution blocks (Conv4 and Conv5) in the original ResNet50 are deleted to provide richer spatial information.

[0025] Preferably, in step (S1-2), the Transformer feature fusion module (TFF) has multiple encoding layers, each of which is composed of a multi-head attention module and a multi-layer perceptron module. TFF removes the position encoding and performs pooling preprocessing on Q, K, and V with a pooling step of 2 to increase the speed of multimodal feature fusion.

[0026] The pooling attention function can be expressed as:

[0027]

[0028] Among them, when Q is a visible light feature, K and V are infrared features; when Q is an infrared feature, K and V are visible light features.

[0029] The multi-head attention module can be expressed as:

[0030] MultiHead(Q,K,V)=concat(H i )W o ,(i=1,...,n) (2)

[0031] H i =PoolAttention(QW i Q ,KW i K ,VW i V ) (3)

[0032] Among them, W i Q ,W i K ,W i V and W o is the parameter of the linear mapping, and n is the number of heads of multi-head attention.

[0033] The Transformer feature fusion module can be expressed as:

[0034] TFF(Q,K,V)=Norm(F+MLP(F)) (4)

[0035] F=Norm(Q+MultiHead(Q,K,V)) (5)

[0036] The visible light feature is used as Q, and the infrared feature is used as K, V, and input into a TFF to obtain the response feature of the visible light feature on the infrared feature. Similarly, the response feature of the infrared light feature on the visible light feature is obtained. Then these two response features are added and input into TFF for self-attention representation to obtain the fused template feature and the fused search feature.

[0037] The cross-correlation operation can be defined as:

[0038] F fi =X fi *Z fi ', (i=2,3) (6)

[0039] F = concat(F f2 ,F f3 ) (7)

[0040] Where, is the cross-correlation operator; Z fi ' is the template feature after fusion of the i-th (i=2,3) layer. When it is the first frame, Z fi ' is the initial template feature. When the number of tracking frames is greater than 1, Z fi ' is the updated template feature; X fi is the search feature after fusion at the i-th (i=2,3) layer; F fi is the response map of the i-th layer (i=2,3); F is the multi-level fusion feature that integrates the features of the 2nd and 3rd layers.

[0041] Preferably, in step (S2-1), the total loss function of the network can be expressed as:

[0042] L=L cls +λ1L reg +λ2L cen (8)

[0043] Among them, λ1 and λ2 are hyperparameters; L cls is the classification loss, which is implemented by the cross entropy function;

[0044] L reg L is the regression loss, used to predict the location of the target box. reg It can be defined as:

[0045]

[0046] Where L iou is the intersection-over-union ratio of the real box and the predicted box, which can be obtained by g (i,j) Calculate. reg (i,j) is the distance from the image coordinate (i, j) to the four edges of the real box.

[0047] L cen L is the center loss, which is used to prevent the predicted target from deviating from the center. cen It can be defined as:

[0048]

[0049] in, is the center score of the image coordinate (i, j).

[0050] Preferably, in step (S3-5), the classification score map uses a ratio and scale penalty mechanism to suppress abnormal deformation of the predicted target box:

[0051]

[0052] Among them, k is a hyperparameter, r is the aspect ratio, r' is the aspect ratio of the last frame, s is the target size of the previous frame, and s' is the target size of the current frame.

[0053] The cosine window penalty (H) mechanism is used to suppress the displacement of target anomalies. The final classification score map can be expressed as:

[0054] cls=(1-α)cls*penalty+αH (12)

[0055] Among them, α is a hyperparameter. In the final classification score graph, the position with the largest score is the predicted target position.

[0056] Preferably, in step (S3-6), according to the information of the tracking result, the corresponding position of the fused multi-level template is directly mapped back to obtain the updated template features. In order not to lose the initial template features, the template update adopts the weighted addition method:

[0057] Z fi '=0.3Z fi +0.7Z fin (13)

[0058] Where Z fin The tracking result of the nth frame is directly mapped back to the features of the corresponding position of the i-th layer search feature map, Z fi is the initial template feature.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The RGBT target tracking method based on multi-layer global feature fusion and mapping template update of the present invention uses the pruned ResNet50 as the feature extraction network of the twin structure to provide richer spatial information and improve network efficiency.

[0061] 2. The RGBT target tracking method based on multi-layer global feature fusion and mapping template update of the present invention designs a Transformer feature fusion network, deletes the position encoding of the Transformer basic module, and performs pooling preprocessing on Q, K, and V with a pooling step of 2, thereby improving the multimodal feature fusion speed and the fusion capability of multimodal information.

[0062] 3. The RGBT target tracking method based on multi-layer global feature fusion and mapping template update of the present invention utilizes multi-level visible light and infrared fusion feature information. The multi-level fusion information can predict the target in different receptive fields, making the target tracker performance more robust.

[0063] 4. The mapping template update method of the present invention uses the tracking result of the previous frame to directly map back to the corresponding position of the fused multi-level search feature for updating the template of the next frame tracking. The template update network does not need to be trained, so it can make the target tracker performance more robust while adding almost no additional computing power and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of the RGBT target tracking method based on multi-layer global feature fusion and mapping template updating of the present invention.

[0065] Figure 2 It is a schematic diagram of the Transformer feature fusion network of the RGBT target tracking method based on multi-layer global feature fusion and mapping template update of the present invention, including the overall Transformer feature fusion network and the local Transformer feature fusion module.

[0066] Figure 3 It is a schematic diagram of a template updating network of the RGBT target tracking method based on multi-layer global feature fusion and mapping template updating of the present invention.

[0067] Figure 4 It is a schematic diagram of the overall model of the RGBT target tracking method based on multi-layer global feature fusion and mapping template updating of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0069] See also Figure 1 The RGBT target tracking method based on multi-layer global feature fusion and mapping template update of the present invention comprises the following steps:

[0070] (S1) Build a target tracking network: The model includes a twin structure feature extraction network, a Transformer feature fusion network, and a tracking result prediction network;

[0071] (S2) Model training and fine-tuning: Use the pre-training data set to pre-train the feature extraction network based on the twin structure, and use the stochastic gradient descent (SGD) method to train until the loss value converges to obtain the pre-trained model; use the training data set to fine-tune the pre-trained model, and use SGD to train until the loss value converges to obtain the trained network;

[0072] (S3) Target tracking stage: obtain the first frame of the visible light and infrared image sequence of the RGBT test dataset, select the target to be tracked, and input the trained RGBT target tracking model based on the twin structure and Transformer feature fusion to generate the target tracking result; according to the information of the tracking result, directly map it back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

[0073] See also Figure 1 In step (S1), the construction of the twin structure feature extraction network, Transformer feature fusion network and tracking result prediction network includes the following steps:

[0074] (S1-1) Constructing a twin structure feature extraction network: The twin structure-based feature extraction network adopts a deep symmetrical structure, which is composed of 4 modified ResNet50s;

[0075] (S1-2) Construct Transformer feature fusion network: Based on the modification of Transformer network, realize the fusion of multi-modal and multi-level features; finally, obtain the response map of tracking results through cross-correlation operation;

[0076] (S1-3) Construct a tracking result prediction network: The tracking result prediction network contains three branches: a classification branch that predicts the category of each position in the response map; a prediction box regression branch that predicts the target position; and a centrality branch that predicts the centrality score of each position.

[0077] (S1-4) Constructing the template update network: This includes the mapping calculation of the coordinate positions in the tracking results. Through the characteristics of convolution and Transformer, the tracking result coordinates are mapped back to the corresponding positions of the fused multi-level templates.

[0078] See also Figure 1 In step (S2), the feature extraction network based on the twin structure is pre-trained and the pre-trained model is fine-tuned, including the following steps:

[0079] (S2-1), the feature extraction network includes 4 ResNet50s with the same structure, and every 2 ResNet50s form a pair of twin structure feature extraction networks. Use the visible light data structure as a pre-training data set to pre-train one pair of twin structure feature extraction networks, use SGD to optimize the tracking model until the model loss value converges, and finally save the pre-trained model;

[0080] (S2-2) Use the pre-trained model parameters saved in step (S2-1) to initialize the feature extraction network, freeze the first two layers of the ResNet50 network, use the RGBT training dataset to fine-tune the tracking model, and at the same time reduce the learning rate of the network. Use SGD optimization until the model loss value converges to obtain a trained network.

[0081] See also Figure 1 In step (S3), the target template to be tracked in the visible light and infrared images of the RGBT test data set is obtained and tracked, including the following steps:

[0082] (S3-1), the template image is centered on the target selected in the first frame of the visible light and infrared image sequence, and 127×127 pixels are cropped. The subsequent frames are search images, and the search image is 255×255 pixels;

[0083] (S3-2), inputting the visible light and infrared template graphs and search graphs obtained in step (S3-1) into the feature extraction network of the twin structure respectively, to obtain multi-layer visible light and infrared template features and search features;

[0084] (S3-3) Using the Transformer feature fusion network, the multi-layer visible light and infrared template features and the visible light and infrared search features obtained in step (S3-2) are fused to generate multi-layer fused template features and fused search features.

[0085] (S3-4), performing cross-correlation operations on the fused template features and fused search features of the same layer in pairs to obtain a multi-layer response map for tracking prediction;

[0086] (S3-5) Concatenate the multiple layers of response maps and input them into the tracking result prediction network to generate a 6-dimensional vector f = (cls, cen, l, t, r, b) to predict the target position in the visible light and infrared images, where cls is the classification score, cen is the center score, l+r and t+b are the width and height of the predicted target in the current frame.

[0087] (S3-6) Mapping the coordinate information of the predicted tracking result back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

[0088] See also Figure 2 The Transformer feature fusion module (TFF) of the present invention has multiple encoding layers, each of which is composed of a multi-head attention module and a multi-layer perceptron module. TFF removes the position encoding and performs pooling preprocessing on Q, K, and V with a pooling step of 2 to improve the speed of multimodal feature fusion.

[0089] The pooling attention function can be expressed as:

[0090]

[0091] Among them, when Q is a visible light feature, K and V are infrared features; when Q is an infrared feature, K and V are visible light features.

[0092] The multi-head attention module can be expressed as:

[0093] MultiHead(Q,K,V)=concat(H i )W o ,(i=1,...,n) (2)

[0094] H i =PoolAttention(QW i Q ,KW i K ,VW i V ) (3)

[0095] Among them, W i Q ,W i K ,W i V and W o is the parameter of the linear mapping, and n is the number of heads of multi-head attention.

[0096] The Transformer feature fusion module can be expressed as:

[0097] TFF(Q,K,V)=Norm(F+MLP(F)) (4)

[0098] F=Norm(Q+MultiHead(Q,K,V)) (5)

[0099] The visible light feature is used as Q, and the infrared feature is used as K, V, and input into a TFF to obtain the response feature of the visible light feature on the infrared feature. Similarly, the response feature of the infrared light feature on the visible light feature is obtained. Then these two response features are added and input into TFF for self-attention representation to obtain the fused template feature and the fused search feature.

[0100] See also Figure 3 , the cross-correlation operation can be defined as:

[0101] F fi =X fi ★Z fi ', (i=2,3) (6)

[0102] F = concat(Ff2 ,F f3 ) (7)

[0103] Where, is the cross-correlation operator; Z fi ' is the template feature after fusion of the i-th (i=2,3) layer. When it is the first frame, Z fi ' is the initial template feature. When the number of tracking frames is greater than 1, Z fi ' is the updated template feature; X fi is the search feature after fusion at the i-th (i=2,3) layer;; F fi is the response map of the i-th layer (i=2,3); F is the multi-level fusion feature that integrates the features of the 2nd and 3rd layers.

[0104] The tracking result prediction network consists of three branches: a classification branch that predicts the category of each position in the response map; a prediction box regression branch that predicts the target position; and a centrality branch that predicts the centrality score of each position. Each branch participates in the calculation of the loss. The total loss function of the network can be expressed as:

[0105] L=L cls +λ1L reg +λ2L cen (8)

[0106] Among them, λ1 and λ2 are hyperparameters; L cls is the classification loss, which is implemented by the cross entropy function;

[0107] L reg L is the regression loss, used to predict the location of the target box. reg It can be defined as:

[0108]

[0109] Where L iou is the intersection-over-union ratio of the real box and the predicted box, which can be obtained by g (i,j) Calculate. reg (i,j) is the distance from the image coordinate (i, j) to the four edges of the real box.

[0110] L cen L is the center loss, which is used to prevent the predicted target from deviating from the center. cen It can be defined as:

[0111]

[0112] in, is the center score of the image coordinate (i, j).

[0113] Preferably, in step (S3-5), the classification score map uses a ratio and scale penalty mechanism to suppress abnormal deformation of the predicted target box:

[0114]

[0115] Among them, k is a hyperparameter, r is the aspect ratio, r' is the aspect ratio of the last frame, s is the target size of the previous frame, and s' is the target size of the current frame.

[0116] The cosine window penalty (H) mechanism is used to suppress the displacement of target anomalies. The final classification score map can be expressed as:

[0117] cls=(1-α)cls*penalty+αH (12)

[0118] Among them, α is a hyperparameter. In the final classification score graph, the position with the largest score is the predicted target position.

[0119] Preferably, in step (S3-6), according to the information of the tracking result, the corresponding position of the fused multi-level template is directly mapped back to obtain the updated template features. In order not to lose the initial template features, the template update adopts the weighted addition method:

[0120] Z fi '=0.3Z fi +0.7Z fin (13)

[0121] Where Z fin The tracking result of the nth frame is directly mapped back to the features of the corresponding position of the i-th layer search feature map, Z fi is the initial template feature.

[0122] The above is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited to the above content. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A RGBT target tracking method based on multi-layer global feature fusion and mapping template update, It is characterized in that The following steps are involved: (S1) Build a target tracking model: The model includes a twin structure feature extraction network, a Transformer feature fusion network, and a tracking result prediction network; (S2) Model training and fine-tuning: Use the visible light pre-training dataset to pre-train the feature extraction network based on the twin structure, and use the stochastic gradient descent (SGD) method to train until the loss value converges to obtain the pre-trained model; use the RGBT training dataset to fine-tune the pre-trained model, and use SGD to train until the loss value converges to obtain the trained network; (S3) Target tracking stage: obtain the first frame of the visible light and infrared image sequence of the RGBT test dataset, select the target to be tracked, and input the trained RGBT target tracking model based on the twin structure and Transformer feature fusion to generate the target tracking result; according to the information of the tracking result, directly map it back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

2. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update according to claim 1, step (S1) specifically The following steps are involved: (S1-1) Constructing a twin structure feature extraction network: The twin structure-based feature extraction network adopts a deep symmetrical structure, which is composed of 4 modified ResNet50s; (S1-2) Construct Transformer feature fusion network: Based on the modification of Transformer network, realize the fusion of multi-modal and multi-level features; finally, obtain the response map of tracking results through cross-correlation operation; (S1-3), Construct the tracking result prediction network: The tracking result prediction network contains three branches: a classification branch that predicts each position category in the response map; The prediction box regression branch predicts the target location; the centrality branch predicts the centrality score for each location. (S1-4) Constructing the template update network: This includes the mapping calculation of the coordinate positions in the tracking results. Through the characteristics of convolution and Transformer, the tracking result coordinates are mapped back to the corresponding positions of the fused multi-level templates.

3. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update in claim 1, in step (S2), the pre-training of the feature extraction network based on the twin structure and the fine-tuning of the model are performed. The following steps are involved: (S2-1), the feature extraction network includes 4 ResNet50s with the same structure, and every 2 ResNet50s form a pair of twin structure feature extraction networks. The visible light data structure is used as a pre-training data set to pre-train one pair of twin structure feature extraction networks, and the SGD optimization tracking model is used until the model loss value converges, and finally the pre-training model is saved; (S2-2) Use the pre-trained model parameters saved in step (S2-1) to initialize the feature extraction network, freeze the first two layers of the ResNet50 network, use the RGBT training dataset to fine-tune the tracking model, and at the same time reduce the learning rate of the network. Use SGD optimization until the model loss value converges to obtain a trained network.

4. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update according to claim 1, in step (S3) specifically The following steps are involved: (S3-1), the template image is centered on the target selected in the first frame of the visible light and infrared image sequence, and 127×127 pixels are cropped. The subsequent frames are search images, and the search image is 255×255 pixels; (S3-2), inputting the visible light and infrared template graphs and search graphs obtained in step (S3-1) into the feature extraction network of the twin structure respectively, to obtain multi-layer visible light and infrared template features and search features; (S3-3), using the Transformer feature fusion network to fuse the multi-layer visible light and infrared template features and the visible light and infrared search features obtained in step (S3-2) to generate multi-layer fused template features and fused search features; (S3-4), performing cross-correlation operations on the fusion template features and fusion search features of the same layer in pairs to obtain a multi-layer response map for tracking prediction; (S3-5) Concatenate the multiple layers of response maps and input them into the tracking result prediction network to generate a 6-dimensional vector f = (cls, cen, l, t, r, b) to predict the target position in the visible light and infrared images, where cls is the classification score, cen is the center score, l+r and t+b are the width and height of the predicted target in the current frame. (S3-6) Mapping the coordinate information of the predicted tracking result back to the corresponding position of the fused multi-level search feature for updating the template for the next frame tracking.

5. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template updating according to claim 2, in step (S1-1), the downsampling operations in the last two convolution blocks (Conv4 and Conv5) in the original ResNet50 are deleted.

6. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update according to claim 2, in step (S1-2), the Transformer feature fusion module (TFF) has multiple encoding layers, each encoding layer is composed of a multi-head attention module and a multi-layer perceptron module, TFF deletes the position encoding, and performs pooling preprocessing on Q, K, and V, and the pooling step size is 2; The pooling attention function can be expressed as: in, When Q is a visible light feature, K and V are infrared features. When Q is an infrared feature, K and V are visible light features. The multi-head attention module can be expressed as: MultiHead(Q,K,V)=concat(H i )W o ,(i=1,...,n) (2) H i =PoolAttention(QW i Q ,KW i K ,VW i V ) (3) Among them, W i Q ,W i K ,W i V and W o is the parameter of linear mapping, n is the number of heads of multi-head attention; The Transformer feature fusion module can be expressed as; TFF(Q,K,V)=Norm(F+MLP(F)) (4) F=Norm(Q+MultiHead(Q,K,V)) (5) The visible light feature is used as Q, and the infrared feature is used as K, V. They are input into a TFF to obtain the response feature of the visible light feature on the infrared feature. Similarly, the response feature of the infrared feature on the visible light feature is obtained. Then, these two response features are added and input into TFF for self-attention representation to obtain the fused template feature and the fused search feature. The cross-correlation operation can be defined as: F fi =X fi ★Z fi ',(i=2,3) (6) F=concat(F f2 ,F f3 ) (7) Where, is the cross-correlation operator; Z fi ' is the template feature after fusion of the i-th (i=2,3) layer. When it is the first frame, Z fi ' is the initial template feature. When the number of tracking frames is greater than 1, Z fi ' is the updated template feature; X fi is the search feature after fusion at the i-th (i=2,3) layer; F fi is the response map of the i-th layer (i=2,3); F is the multi-level fusion feature that integrates the features of the 2nd and 3rd layers.

7. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template updating according to claim 3, in step (S2-1), the total loss function of the network can be expressed as: L=L cls +λ 1 L reg +λ 2 L cen (8) in, λ 1 , 2 is a hyperparameter; L cls is the classification loss, which is implemented by the cross entropy function; L reg is the regression loss, used to predict the location of the target box, L reg It can be defined as: Where L iou is the intersection-over-union ratio of the real box and the predicted box, which can be obtained by g (i,j) Calculate, v reg (i,j) is the distance from the image coordinate (i, j) to the four edges of the real box. L cen is the center loss, which is used to prevent the predicted target from deviating from the center. cen It can be defined as: in, is the center score of the image coordinate (i, j).

8. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update according to claim 4, in step (S3-5), the classification score map adopts a ratio and scale penalty mechanism to suppress abnormal deformation of the predicted target box: in, k is a hyperparameter, r is the aspect ratio, r' is the aspect ratio of the last frame, s is the target size of the previous frame, and s' is the target size of the current frame; The cosine window penalty (H) mechanism is used to suppress the displacement of target anomalies. The final classification score map can be expressed as: cls=(1-α)cls*penalty+αH (12) Among them, α is a hyperparameter. In the final classification score graph, the position with the largest score is the predicted target position.

9. According to the RGBT target tracking method based on multi-layer global feature fusion and mapping template update of claim 4, in step (S3-6), the information of the tracking result is directly mapped back to the corresponding position of the fused multi-level template to obtain the updated template feature. In order not to lose the initial template feature, the template update adopts the weighted addition method: <h2 style=";text-align:left;direction:ltr">Z<h2 style=";text-align:left;direction:ltr"> fi <h2 style=";text-align:left;direction:ltr"> '=0.3Z<h2 style=";text-align:left;direction:ltr"> fi <h2 style=";text-align:left;direction:ltr"> +0.7Z<h2 style=";text-align:left;direction:ltr"> fin <h2 style=";text-align:left;direction:ltr"> (13) Where Z fin The tracking result of the nth frame is directly mapped back to the features of the corresponding position of the i-th layer search feature map, Z fi is the initial template feature.

Images