Visible light-infrared target tracking method and system based on deep network

Abstract

The present invention discloses a visible-light-infrared target tracking method and system based on a deep network, comprising: constructing a symmetrical two-stream twin network, including a visible light sub-network and an infrared sub-network; inputting a visible light sample-candidate image and an infrared sample-candidate image into the visible light sub-network and the infrared sub-network, respectively, to extract templates and candidate features; enhancing the templates and candidate features using a channel attention module and a channel-spatial joint attention module; classifying and branching the original templates and candidate features, as well as the enhanced templates and candidate features, through classification and regression branches to obtain corresponding classification response maps and regression response maps; fusing the classification and regression response maps; and processing the classification and regression response maps through an adaptive peak selection module to obtain target location. The present invention maintains high-speed operation while ensuring tracking robustness, and has excellent performance when applied to visible-light-infrared target tracking tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of visible light-infrared target tracking, and in particular relates to a visible light-infrared target tracking method and system based on a deep network. Background Art

[0002] In recent years, thanks to the solid foundation laid by the rapid development of computer hardware, deep learning has continued to gain popularity, garnering widespread attention in both industry and academia. Deep learning models often have deeper network structures and more parameters, enabling the extraction of deeper and more discernible features, but also requiring larger amounts of data. Computer vision, leveraging the widespread application of deep learning, has also achieved exciting progress in a wider range of fields.

[0003] Target tracking is an important branch and fundamental problem in computer vision, and has been a hot topic and a difficult topic in the field of vision in recent years. However, target tracking currently faces many challenging issues, especially in complex environmental conditions (such as low light, rain, and fog), where the quality of visible light images is significantly affected, making tracking the target object difficult. Research has found that thermal infrared sensors provide more stable features for these situations. The features of the two modalities can complement each other in various challenging scenarios: when visible light is interfered with by low illumination, high illumination, or background occlusion, thermal information can effectively overcome these interferences; when the thermal spectrum is affected by thermal crosstalk, glass, and other factors, visible light information can effectively process them.

[0004] Over the past decade, researchers have proposed numerous methods for visible-infrared (VIS) target tracking. From traditional VIS target tracking methods based on sparse representation and correlation filtering to VIS target tracking based on deep learning, VIS target tracking has become a targeted research topic in the tracking field. Traditional RGB-T target tracking methods are mostly online, aiming to utilize simple and effective hand-crafted visual features, combined with shallow appearance models, and matching or classification algorithms for fast and efficient target tracking. For example, sparse representation-based methods combine visible and infrared information to sparsely represent each sample in the target template space. To suppress noise and reduce error, researchers have proposed a VIS target tracking method using low-rank-constrained correlation filtering [Zhai S, Shao P, Liang X, et al. Fast RGB-T tracking via cross-modal correlation filters [J]. Neurocomputing, 2019, 334:172-181.]. Cross-modal correlation filters are used to obtain the interdependence between the visible light and thermal infrared modalities, realizing the collaborative fusion of multiple modalities so that the learned filters can contain useful information from different data sources, thereby obtaining robust tracking results. In recent years, with the continuous development of deep networks and the release of large-scale visible light-infrared target tracking datasets, visible light-infrared target tracking methods based on deep learning have gradually become mainstream, such as using dual convolutional neural networks to extract visible light and infrared features separately for cascade fusion; however, this method will introduce redundant noise. In order to improve performance, researchers have proposed a visible light-infrared target tracking method based on dense feature aggregation and pruning network [Zhu Y, Li C, Luo B, et al. Dense feature aggregation and pruning for rgbt tracking [C] / / Proceedings of the 27th ACM International Conference on Multimedia. 2019: 465-472.], which provides richer feature representation while eliminating redundant noise; in addition, based on multi-adapter and challenge perception [Long Li C, Lu A, Hua Zheng A, et al. Multi-adapter rgbt tracking [C] / / Proceedings of the IEEE International Conference on Computer Vision Workshops.2019:0-0.], deep methods based on global attention and local attention have also been proposed to further improve the tracking accuracy.However, since these deep models all use online training to improve model performance during the tracking process, their speed often becomes a weakness and they cannot run in real time. Summary of the Invention

[0005] The present invention discloses a visible light-infrared target tracking method and system based on a deep network, which can resist the decrease of tracking speed and improve the tracking accuracy and speed.

[0006] The technical solution to realize the present invention is: a visible light-infrared target tracking method based on deep network, comprising the steps of:

[0007] In the first step, the input video includes visible light video sequence and infrared video sequence, that is, the network input includes infrared sample image sequence and infrared candidate image sequence, visible light sample image sequence and visible light candidate image sequence;

[0008] The second step is to establish a symmetrical two-stream twin network structure, where the backbone networks of the visible light branch and the infrared branch share weights respectively;

[0009] The third step is to adopt a channel attention module and a channel-spatial joint attention module, which are trainable and embedded in the network as feature enhancement units.

[0010] In the fourth step, the original features and the enhanced features are cross-correlated through the classification branch and the regression branch to obtain the classification response map and the regression response map and fuse them; the classification branch uses the cross entropy as the loss function, and the regression branch uses the smoothed L1 norm as the loss function to jointly train the network;

[0011] In the fifth step, during the tracking and inference phase, the classification response map and the regression response map are processed by the adaptive peak selection module to complete the target positioning.

[0012] A visible light-infrared target tracking system based on a deep network includes a data acquisition and processing module and a target tracking network, wherein:

[0013] The acquisition and processing module is used to obtain infrared sample images and infrared candidate images, visible light sample images and visible light candidate images;

[0014] The target tracking network is a symmetrical two-stream twin network structure, including a visible light sub-network and an infrared sub-network for feature extraction, a channel attention module and a channel-space joint attention module embedded in the visible light sub-network and the infrared sub-network, a classification branch and a regression branch, an image fusion module and an adaptive peak selection module; the channel attention module and the channel-space joint attention module are used to extract feature enhancement, the classification branch and the regression branch are used for the classification and branching of the original template and candidate features and the enhanced template and candidate features, respectively, and output the corresponding classification response map and regression response map; the image fusion module is used to fuse the classification response map and the regression response map, and the adaptive peak selection module is used to process the classification response map and the regression response map to obtain target positioning.

[0015] Compared with the existing technology, the present invention has the following significant features: (1) establishing a symmetrical deep two-stream twin network structure to effectively extract the semantic features of the visible light modality and the infrared modality; (2) using the attention mechanism to adaptively enhance the features of the visible light modality and the infrared modality to enhance their discriminative power; (3) through response-level fusion and the use of the maximum peak selection method to obtain higher accuracy in target positioning; (4) adopting an end-to-end network structure with a simple, concise and orderly structure, abandoning the complex pre-processing and post-processing processes, and low computational complexity; it has the advantages of strong generalization, high tracking accuracy and fast tracking speed, with an accuracy of 88.2% on the benchmark dataset and a speed of up to 140 frames per second. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0017] Figure 2 It is a network structure diagram of the method of the present invention.

[0018] Figure 3 This is a structural diagram of the channel attention module of the present invention.

[0019] Figure 4 This is a structural diagram of the spatial attention module of the present invention. DETAILED DESCRIPTION

[0020] Combine Figure 1 The following describes the implementation process of the present invention in detail. A visible light-infrared target tracking method using a dual-modal response fusion deep network has the following steps:

[0021] In the first step, the input sample-candidate image pair contains visible light modality and infrared modality, that is, the network input contains infrared sample image and infrared candidate image, denoted as Visible light sample image and visible light candidate image, denoted as

[0022] The second step is to establish a symmetrical two-stream twin network structure, where the backbone networks of the visible light branch and the infrared branch share weights. The backbone network is composed of multiple two-dimensional convolution units connected in series, and the output of each two-dimensional convolution unit serves as the input of the next two-dimensional convolution unit. rgb ∈R h×w×c is the visible light image data, h, w, and c are the three dimensions of the visible light image data, namely height, width, and number of channels. tir ∈R H×W×C ,. H, W, C are the three dimensions of infrared image data, namely height, width, and number of channels. The input of the i-th (1≤i≤n) two-dimensional convolution unit is X i , the output is Then we have:

[0023]

[0024]

[0025]

[0026]

[0027] Where n represents the number of two-dimensional convolutional units in the backbone network, Relu(·) represents the activation function, BN(·) represents the batch normalization function, and the operator represents the convolution operation, and denote the convolution kernel of the i-th two-dimensional convolution unit of the visible light and infrared backbone networks respectively (k×k is the convolution kernel size), and are the bias terms corresponding to the visible light and infrared convolution kernels respectively.

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] and Represents the i-th visible light and infrared two-dimensional convolution unit respectively. rgb and Feat tir are the outputs of visible light data and infrared data after passing through the backbone network, Frgb (·) and F tir (·) denote the visible light and infrared backbone networks, respectively.

[0035] The third step is to use the channel attention module and the channel-space joint attention module, which are trainable and embedded in the network as feature enhancement units. Figure 3 , the input Feat of the channel attention module rgb ∈R h×w×c , Feat tir ∈R h×w×c :

[0036] Feat rgb ∈R h×w×c ,Feat tir ∈R h×w×c

[0037] Feat union =Cat(Feat rgb ,Feat tir )

[0038] Weight=G(Resize(FC(GP(Feat union ))))

[0039]

[0040]

[0041] Among them, Feat rgb 、Feat rgb Respectively represent the visible light features and infrared features extracted by the backbone network, Cat(·,·) represents the cascade operation according to the channel dimension, and Feat union ∈R h×w×2c Represents the concatenated features; GP(·) represents the global pooling operation, FC(·) represents the fully connected layer, Resize(·) represents the resizing operation, G(·) represents the activation function, and Weight∈R h×w×2c Indicates the calculated weight; * indicates the dot product of the matrix, Represents the weighted (enhanced) features; Apart(·) represents the splitting of features by channel dimension. They represent the weighted (enhanced) visible light features and infrared features respectively.

[0042] Combine Figure 4 , the input Feat of the spatial attention module rgb ∈R h×w×c , Feat tir ∈R h×w×c :

[0043] Feat rgb ∈R h×w×c ,Feat tir ∈R h×w×c

[0044] Feat union =Cat(Feat rgb ,Feat tir )

[0045]

[0046]

[0047]

[0048] Among them, Feat rgb 、Feat rgb Represent visible light features and infrared features respectively, Cat(·,·) represents the cascade operation according to the channel dimension, Feat union ∈R h×w×2c Represents the features after cascading; Avg(·) represents the average pooling operation, Max(·) represents the maximum pooling operation, is a two-dimensional convolution, S spatial ∈R k×k×c represents the two-dimensional convolution kernel of the spatial attention module, G(·) represents the activation function, and Weight∈R h×w×2c is the calculated weight; × is the broadcast multiplication, Represents the weighted (enhanced) features; Apart(·) represents the splitting of features by channel dimension. They are the weighted (enhanced) visible light features and infrared features respectively.

[0049] The channel-spatial attention module consists of a series of channel attention modules and spatial attention modules. The visible light sample features and infrared sample features extracted by the backbone network are fed into the channel attention module for enhancement, and the visible light candidate features and infrared candidate features extracted by the backbone network are then enhanced by the channel-spatial attention module.

[0050] In the fourth step, the original features and enhanced features are cross-correlated through the classification branch and regression branch to obtain the classification response map and regression response map and then fused. The network contains 4 classification branches and 2 regression branches, namely: original visible light classification branch, enhanced visible light classification branch, original infrared classification branch, enhanced infrared classification branch, enhanced visible light regression branch, and enhanced infrared regression branch. The input of the classification branch is Respectively represent sample features and candidate features with the same attributes (such as infrared enhanced sample features and infrared enhanced candidate features):

[0051]

[0052]

[0053] in, is the classification sample convolution kernel, is the classification candidate convolution kernel, represents a two-dimensional convolution, Represents the cross-correlation operation, ClsMap∈R n×n×2 is a categorical response plot.

[0054] The input of the regression branch is Respectively represent sample features and candidate features of the same attribute (such as visible light enhanced sample features and visible light enhanced candidate features):

[0055]

[0056]

[0057] in, is the regression sample convolution kernel, is the regression candidate convolution kernel, represents a two-dimensional convolution, Represents the cross-correlation operation, RegMap∈R n×n×4 is the regression response graph. The original features and enhanced features pass through 4 classification branches and 2 regression branches, generating a total of 6 different response graphs:

[0058]

[0059] ClSMap rgb ∈R n×n×l×2 , ClSMap tir ∈R n×n×l×2 ,

[0060]

[0061] They represent respectively: enhanced visible light classification response map, enhanced infrared classification response map, original visible light classification response map, original infrared classification response map, enhanced visible light regression response map, and enhanced infrared regression response map.

[0062]

[0063]

[0064]

[0065] Where + represents the point-to-point addition of matrix elements. ClsMap1∈Rn×n×l×2 , ClsMap2∈R n×n×l×2 , RegMap∈R n×n×l×4 These are the three final response images, where n represents the side length of the response image and l represents the number of anchor boxes at different scales.

[0066] The fifth step is to process the classification response map and the regression response map through the adaptive peak selection module in the tracking and inference stage to complete the target positioning. In the tracking and inference stage, by comparing the maximum response values ​​of the two response maps ClsMap1 and ClsMap2, the coordinates corresponding to the maximum response value anchor box are taken as the index, and the boundary offset of the prediction box relative to the previous frame is located in RegMap. Let the index of the maximum response value point of ClsMap1 be Let the maximum response value point index of ClsMap2 be

[0067] like

[0068]

[0069] but

[0070]

[0071] like

[0072]

[0073] but

[0074]

[0075] I max Indicates the coordinates of the anchor box with the maximum response value. max After that, the boundary offset can be located in the regression response map RegMap:

[0076] (Δcx,Δcy,Δw,Δh)=RegMap2[I max ]

[0077] Δcx, Δcy, Δw, Δh represent the horizontal offset, vertical offset, width offset, and height offset of the prediction box center, respectively. Based on (Δcx, Δcy, Δw, Δh), the prediction box of the current frame can be calculated:

[0078] (cx,cy,w,h)=(cx0+Δcx,cy0+Δcy,w0+Δw,h0+Δh)

[0079] Among them, cx, cy, w, h represent the center horizontal coordinate, center vertical coordinate, width, and height of the current frame prediction box; cx0, cy0, w0, and h0 represent the center horizontal coordinate, center vertical coordinate, width, and height of the previous frame prediction box.

[0080] In the sixth step, the classification branch uses cross entropy as the loss function, and the regression branch uses the coordinate normalized smoothed L1 norm as the loss function to jointly train the network. The classification branch completes the binary classification task of distinguishing foreground and background. The response map output by the classification branch during the training phase is ClsMap∈R n×n×l×2 , after size transformation to ClsMap∈R M×2 , M = n × n × l, its corresponding label L cls ∈R M ; from L cls Extract p positive samples from the dataset to generate a positive sample set And record its index From L cls Extract q negative samples from the negative sample set to generate a negative sample set And record its index Extract the index from the ClsMap Generate positive prediction set for corresponding samples Extract Index Generate negative prediction set for corresponding samples Then we have:

[0081]

[0082] The regression branch completes the task of bounding box regression. The response map output by the regression branch during the training phase is RegMap∈R n ×n×l×4 , after size transformation, it is RegMap∈R M×4 , M = n × n × l, its corresponding label L reg ∈R M×4 ; from L reg Extract p positive samples from the dataset to generate a positive sample set And record its index Extract index from RegMap Generate positive prediction set for corresponding samples Then we have:

[0083]

[0084] Where SmoothL1(·) represents the smoothed L1 norm.

[0085] The final Loss is defined as:

[0086]

[0087] Among them, γ∈[0,10) is a hyperparameter used to balance the two losses.

[0088] The method of the present invention was simulated using three real-world visible-light-infrared datasets: the GTOT dataset, the RGBT234 dataset, and the LasHeR dataset. The GTOT dataset contains 50 sequences, totaling 7,800 manually annotated visible-light-infrared image pairs, with each sequence containing an average of 157 frames. The RGBT234 dataset contains 234 sequences, totaling 116,700 manually annotated visible-light-infrared image pairs, with each sequence containing an average of 498 frames. The LasHeR dataset contains 1,224 sequences, totaling 734,800 manually annotated visible-light-infrared image pairs, with each sequence containing an average of 600 frames. Because GTOT and RGBT234 do not have training and test sets, two training-testing methods were used in the experiment: 1. The network was trained using the RGBT234 and LasHeR datasets as the training sets, and tested using the GTOT dataset as the test set; 2. The network was trained using the LasHeR dataset as the training set, and tested using the RGBT234 dataset as the test set. During each forward propagation of training, a sequence is randomly extracted from the training set, and the visible light-infrared template image pair and visible light-infrared candidate image pair corresponding to the sequence are generated and sent to the network. The network optimizer algorithm is selected as stochastic gradient descent (SGD), the momentum is set to 0.9, the weight decay is set to 0.0005, the initial learning rate is set to 0.01, the termination learning rate is set to 0.00001, the batch size is set to 28, and the data set does not require any preprocessing. The two groups of experiments use PR (PrecisionRate), SR (SuccessRate) and FPS (FramesPerSecond) as evaluation indicators. Simulation experiment software environment: Ubuntu16.04 operating system, Python3.7, Pytorch1.7.1; simulation experiment hardware environment: i7-9700k processor, 16gb memory, NVIDIA 2080ti graphics card; the network architecture used in the experiment is as follows Figure 2 shown.

[0089] Tables 1 and 2 show the performance metrics of our proposed method in simulation experiments on the GTOT and RGBT234 datasets, respectively. The experimental results demonstrate that our method is highly effective on real-world datasets, achieving a tracking accuracy of 88.2% on the GTOT dataset and 75.0% on the RGBT234 dataset, while maintaining a speed of 140 frames per second. Our proposed method significantly outperforms leading methods including ECO, SGT, DuSiamRT, and SiamFT in terms of PR, SR, and FPS. However, methods such as DAPNet and MANet, which employ online training strategies, are significantly limited in speed, achieving only approximately 1 frame per second on the GTOT and RGBT234 datasets. Our proposed method, employing an offline training and online tracking strategy, achieves comparable PR and SR metrics to DAPNet and MANet, while outperforming them by over a hundred times. These results demonstrate that our proposed method effectively leverages the discriminative semantic features of both visible and infrared modalities, maintaining high tracking speed while maintaining both accuracy and robustness.

[0090] Table 1 Tracking results of different methods on the GTOT dataset

[0091] ECO SGT DAPNet MANet SiamFT DuSiamRT Ours PR (%) 77.0 85.1 88.2 88.9 75.8 76.6 88.2 SR (%) 63.1 62.8 70.7 71.9 62.3 62.8 70.3 FPS 11.2 5.0 1.5 1.2 32.0 117.0 140.0

[0092] Table 2 Tracking results of different methods on the RGBT234 dataset

[0093]

[0094]

[0095] This method uses end-to-end offline training and online tracking to combat tracking speed degradation. A two-stream twin network structure is also established to extract deep semantic information from both modalities. Channel attention and spatial attention modules are used to generate adaptive weights to enhance feature discrimination and reduce noise, enabling better fusion of features from both modalities. This method achieves tracking accuracy of 88.2% on the GTOT dataset and 75.0% on the RGBT234 dataset, while running at 140 frames per second, six times the real-time standard. This demonstrates the superiority of this method over other approaches.

Claims

1. A visible light-infrared target tracking method based on deep network, characterized in that: Including steps: Construct a symmetrical two-stream twin network, including a visible light sub-network and an infrared sub-network; Input the visible light sample-candidate image and infrared sample-candidate image into the visible light sub-network and infrared sub-network respectively to extract templates and candidate features; Enhance templates and candidate features through channel attention modules and channel-spatial joint attention modules; The original template and candidate features as well as the enhanced template and candidate features are classified and branched through the classification branch and the regression branch to obtain the corresponding classification response graph and regression response graph; Fusion of classification response graph and regression response graph; The classification response map and regression response map are processed by the adaptive peak selection module to obtain target positioning; The channel attention module and the channel-space joint attention module are embedded in the visible light sub-network and the infrared sub-network respectively; the template and candidate features are enhanced by the channel attention module and the channel-space joint attention module specifically as follows: The input of the channel attention module is Feat rgb ∈R h×w×c , Feat tir ∈R h×w×c , the output enhanced features are: Feat rgb ∈R h×w×c ,Feat tir ∈R h×w×c Feat union =Cat(Feat rgb ,Feat tir ) Weight=G(Resize(FC(GP(Feat union )))) Among them, Cat(·,·) represents the cascade operation according to the channel dimension, Feat union ∈R h×w×2c Represents the concatenated features; GP(·) represents the global pooling operation, FC(·) represents the fully connected layer, Resize(·) represents the resizing operation, G(·) represents the activation function, and Weight∈R h×w×2c Indicates the calculated weight; * indicates the dot product of the matrix, Indicates the enhanced features; Apart(·) indicates the splitting of features by channel dimension. They represent the enhanced visible light features and infrared features respectively, and h, w, and c are the three dimensions of the visible light image data, namely height, width, and number of channels; Input Feat of the spatial attention module rgb ∈R h×w×c , Feat tir ∈R h×w×c , the output enhanced features are: Feat rgb ∈R h×w×c ,Feat tir ∈R h×w×c Feat union =Cat(Feat rgb ,Feat tir ) Among them, Cat(·,·) represents the cascade operation according to the channel dimension, Feat union ∈R h×w×2c Represents the features after cascading; Avg(·) represents the average pooling operation, Max(·) represents the maximum pooling operation, is a two-dimensional convolution, S spatial ∈R k×k×c represents the two-dimensional convolution kernel of the spatial attention module, G(·) represents the activation function, and Weight∈R h×w×2c is the calculated weight; × is the broadcast multiplication, Indicates the enhanced features; Apart(·) indicates the splitting of features by channel dimension. They are the enhanced visible light features and infrared features; The channel-spatial attention module is composed of a channel attention module and a spatial attention module connected in series, which enhances the visible light candidate features and infrared candidate features extracted by the backbone network of the visible light sub-network and the infrared sub-network.

2. The method for tracking visible light and infrared targets based on a deep network according to claim 1, wherein: The backbone network of the visible light sub-network and infrared sub-network is composed of multiple two-dimensional convolution units connected in series, and the output of each two-dimensional convolution unit is used as the input of the next two-dimensional convolution unit. rgb ∈R h×w×c is the visible light image data, h, w, c are the three dimensions of the visible light image data, namely height, width, and number of channels; tir ∈R H×W×C , H, W, and C are the three dimensions of infrared image data, namely height, width, and number of channels; the input of the i-th two-dimensional convolution unit is X i , the output is Then we have: Where n represents the number of two-dimensional convolutional units in the backbone network, Relu(·) represents the activation function, BN(·) represents the batch normalization function, and the operator represents the convolution operation, and They represent the convolution kernel of the i-th two-dimensional convolution unit of the visible light and infrared backbone networks, respectively, k×k is the convolution kernel size, and are the bias terms corresponding to the visible light and infrared convolution kernels respectively; the output is: and Represents the i-th visible light and infrared two-dimensional convolution unit, Feat rgb and Feat tir are the outputs of visible light data and infrared data after passing through the backbone network, F rgb (·) and F tir (·) denotes the backbone networks of the visible light sub-network and infrared sub-network, respectively.

3. The visible light-infrared target tracking method based on deep network according to claim 1, characterized in that: The classification branch includes 4 classification branches, and the regression branch includes 2 regression branches, namely the original visible light classification branch, the enhanced visible light classification branch, the original infrared classification branch, the enhanced infrared classification branch, the enhanced visible light regression branch, and the enhanced infrared regression branch.

4. The method for tracking visible light and infrared targets based on a deep network according to claim 3, wherein: The input of the classification branch is Represent the sample features and candidate features of the same attribute respectively, and the output classification response graph is: in, is the classification sample convolution kernel, is the classification candidate convolution kernel, represents a two-dimensional convolution, Represents the cross-correlation operation, ClsMap∈R n×n×2 is a categorical response plot; The input of the regression branch is Represent the sample features and candidate features of the same attribute respectively, and the output regression response graph is: in, is the regression sample convolution kernel, is the regression candidate convolution kernel, represents a two-dimensional convolution, Represents the cross-correlation operation, RegMap∈R n×n×4 is the regression response plot; The original features and enhanced features pass through 4 classification branches and 2 regression branches to generate the enhanced visible light classification response map Enhanced infrared classification response map Original visible light classification response map ClsMap rgb , original infrared classification response map ClsMap tir , Enhanced visible light regression response map Enhanced infrared regression response map They are: 。 5. The method for tracking visible light and infrared targets based on a deep network according to claim 4, characterized in that: The specific steps for fusing the classification response map and the regression response map are as follows: Among them, + represents the point-to-point addition of matrix elements, ClsMap1∈R n×n×l×2 , ClsMap2∈R n×n×l×2 , RegMap∈R n ×n×l×4 The three response maps are fused, n represents the side length of the response map, and l represents the number of anchor boxes of different scales.

6. The method for tracking visible light and infrared targets based on a deep network according to claim 4, characterized in that: The classification branch uses cross entropy as the loss function, and the regression branch uses the coordinate normalized smoothed L1 norm as the loss function to jointly train the network; The classification branch completes the binary classification task of distinguishing foreground from background. The response map output by the classification branch during the training phase is ClsMap∈R n×n×l×2 , after size transformation to ClsMap∈R M×2 , M = n × n × l, its corresponding label L cls ∈R M ; from L cls Extract p positive samples from the dataset to generate a positive sample set And record its index From L cls Extract q negative samples from the negative sample set to generate a negative sample set And record its index Extract the index from the ClsMap Generate positive prediction set for corresponding samples Extract Index Generate negative prediction set for corresponding samples Then the loss function is: The regression branch completes the task of bounding box regression. The response map output by the regression branch during the training phase is RegMap∈R n ×n×k×4 , after size transformation, it is RegMap∈R M×4 , M = n × n × l, its corresponding label L reg ∈R M×4 ; from L reg Extract p positive samples from the dataset to generate a positive sample set And record its index Extract index from RegMap Generate positive prediction set for corresponding samples Then the loss function is: Where SmoothL1(·) represents the smoothed L1 norm; The final Loss is defined as: Among them, γ∈[0,10) is a hyperparameter used to balance the two losses.

7. The method for tracking visible light and infrared targets based on a deep network according to claim 4, wherein: The classification response map and regression response map are processed by the adaptive peak selection module to obtain the target positioning as follows: By comparing the maximum response values ​​of the two response maps ClsMap1 and ClsMap2, taking the coordinates corresponding to the maximum response value anchor box as the index, locating the boundary offset of the prediction box relative to the previous frame in the RegMap response map, and setting the maximum response value point index of ClsMap1 to Let the maximum response value point index of ClsMap2 be like but like but I max Indicates the coordinates corresponding to the maximum response value anchor box, and the boundary offset in the positioning regression response map RegMap is: (Δcx,Δcy,Δw,Δh)=RegMap2[Imax] Δcx, Δcy, Δw, and Δh represent the horizontal coordinate offset, vertical coordinate offset, width offset, and height offset of the prediction box center, respectively. The current frame prediction box is calculated based on (Δcx, Δcy, Δw, and Δh): (cx,cy,w,h)=(cx0+Δcx,cy0+Δcy,w0+Δw,h0+Δh) Among them, cx, cy, w, h represent the center horizontal coordinate, center vertical coordinate, width, and height of the current frame prediction box; cx0, cy0, w0, and h0 represent the center horizontal coordinate, center vertical coordinate, width, and height of the previous frame prediction box.

8. A deep network-based visible light-infrared target tracking system implementing the method according to any one of claims 1 to 6, characterized in that: It includes data acquisition and processing modules and target tracking network, including: The acquisition and processing module is used to obtain infrared sample images and infrared candidate images, visible light sample images and visible light candidate images; The target tracking network is a symmetrical two-stream twin network structure, including a visible light sub-network and an infrared sub-network for feature extraction, a channel attention module and a channel-space joint attention module embedded in the visible light sub-network and the infrared sub-network, a classification branch and a regression branch, an image fusion module and an adaptive peak selection module; the channel attention module and the channel-space joint attention module are used to extract feature enhancement, the classification branch and the regression branch are used for the classification and branching of the original template and candidate features and the enhanced template and candidate features, respectively, and output the corresponding classification response map and regression response map; the image fusion module is used to fuse the classification response map and the regression response map, and the adaptive peak selection module is used to process the classification response map and the regression response map to obtain target positioning.

Images