Cross-modal target tracking method and system based on fully convolutional twin network

Abstract

The invention discloses a cross-modal target tracking method based on a fully convolutional twin network. The method comprises the following steps: inputting a visible light image and a thermal infrared image into a cross-modal target tracking network model to obtain a tracking structure; the cross-modal target tracking network model comprises a feature extraction backbone network, a self-adaptive complementary fusion module and a target classification regression network, and the feature extraction backbone network comprises a visible light backbone network and a thermal infrared backbone network which are the same in structure; the visible light backbone network and the thermal infrared backbone network respectively extract features of a template and a search area through convolution and perform adaptive convolution to extract time sequence information; and the self-adaptive complementary fusion module performs channel attention enhancement and space attention enhancement on the visible light features and the thermal infrared features extracted by the feature extraction backbone network and fuses the visible light features and the thermal infrared features. The invention also discloses a cross-modal target tracking system based on the fully convolutional twin network for realizing the method. The method can improve the stability and robustness of tracking in different environments.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a cross-modal target tracking method and system. Background Art

[0002] At present, most target tracking methods mainly use visible light data for target tracking, but in some cases, such as complex lighting environments and smoke, the target is difficult to locate and track. During the tracking process, the target may be lost, thereby reducing the tracking accuracy.

[0003] Since visible light images and thermal infrared images have complementary advantages, cross-modal target tracking that combines visible light images with thermal infrared images has been proven to be an efficient way to improve tracking performance. Among them, effectively integrating multimodal information is a key task in cross-modal target tracking. Cross-modal image fusion mainly covers three fusion strategies: pixel level, feature level, and decision level. Pixel-level fusion can enrich the details of the image, such as edges, colors, and textures, which is very helpful for image analysis and processing, but lacks high-level semantic information. Feature-level fusion can extract semantic data of the target or area of ​​interest, but the spatial detail information is slightly insufficient. Decision-level fusion cannot fully utilize the cross-modal correlation and cannot use the model to jointly model the target or image.

[0004] In addition, there is a lot of temporal information in the target tracking process, which can effectively solve the variability of the target appearance. The usual practice is to introduce a dynamic template and an update controller to capture the appearance changes of the target, but this method will store a large amount of temporal information, which will lead to increased memory usage and computational complexity, which is a huge challenge for the target tracking algorithm. Summary of the invention

[0005] In view of the defects of the above-mentioned prior art, the present invention provides a cross-modal target tracking method based on a fully convolutional twin network, the purpose of which is to improve the stability and robustness of tracking in lighting changes, darkness, and low-visibility environments. The present invention also provides a cross-modal target tracking system based on a fully convolutional twin network for implementing this method.

[0006] The technical solution of the present invention is as follows:

[0007] A cross-modal target tracking method based on a fully convolutional twin network includes the following steps:

[0008] Step 1: Construct a dataset containing visible light and thermal infrared video pairs as a training set for the network model;

[0009] Step 2, constructing a cross-modal target tracking network model; the cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module and a target classification regression network, the feature extraction backbone network includes a visible light backbone network and a thermal infrared backbone network with the same structure, the visible light backbone network and the thermal infrared backbone network respectively extract the features of the template and the search area through convolution and perform adaptive convolution to extract timing information, and the adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network;

[0010] Step 3: Initialize the parameters of the cross-modal target tracking network model;

[0011] Step 4: Preprocess the training set data and perform model training. During training, select a video sequence, randomly extract two frames of visible light images and thermal infrared images at the same time, use the visible light and thermal infrared images as templates, and the corresponding thermal infrared images and visible light images as search areas;

[0012] Step 5: Input the data to be tracked into the trained cross-modal target tracking network model to obtain the tracking result.

[0013] Furthermore, the visible light backbone network and the thermal infrared backbone network both include four convolutional layers, and an adaptive convolutional layer is arranged after the last two convolutional layers to extract timing information.

[0014] Furthermore, the structure of the adaptive convolution layer is: for any input X t , you can get the output

[0015]

[0016] Among them, the operator * represents the convolution operation, W b and b b are learnable parameters, Where f represents the convolution operation,

[0017]

[0018] Represents the input of the adaptive convolutional layer of L frames after pooling.

[0019] Furthermore, when t≤L-1, the first frame is used to make up for it until t>L-1.

[0020] Furthermore, the adaptive complementary fusion module in step 2 performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network as follows:

[0021] The first set of features is obtained by element-wise adding the visible light features with the thermal infrared features after channel attention enhancement and spatial attention enhancement, and the second set of features is obtained by element-wise adding the thermal infrared features with the visible light features after channel attention enhancement and spatial attention enhancement. The first and second sets of features are then concatenated according to the channels and convolved for dimensionality reduction.

[0022] Furthermore, in step 2, the channel attention enhancement is performed by splicing the visible light feature and the thermal infrared feature to obtain the connection feature, and then performing the channel attention enhancement. The visible light feature after the channel attention enhancement is Thermal infrared characteristics are

[0023]

[0024] is the connection feature, where δ is the global pooling, ω is the fully connected layer, ε is Sigmoid, is the channel product, and Split is the operation of splitting features along the channel dimension.

[0025] Furthermore, the spatial attention enhancement in step 2 is to splice the visible light features and thermal infrared features after channel attention enhancement along the channel dimension to obtain the union feature U s , the visible light feature after spatial attention enhancement is Thermal infrared characteristics are

[0026]

[0027]

[0028] Among them, ρ is the average pooling operation, φ is the maximum pooling operation, and Cat is the operation of splicing features along the channel dimension. is a two-dimensional convolution operation, and W is the calculated spatial attention weight.

[0029] Furthermore, the preprocessing in step 4 is to perform image rotation, image flipping, image scaling, image translation and grayscale operations.

[0030] Furthermore, the loss function during model training is

[0031]

[0032] Lcls represents the weighted focal loss, L iou represents IoU loss, L1 represents L1 loss, λ iou is the hyperparameter representing the IoU loss weight, is the hyperparameter representing the L1 loss weight.

[0033] Another technical solution of the present invention is a cross-modal target tracking system based on a fully convolutional twin network, comprising the following modules:

[0034] A training data construction module is used to construct a data set containing visible light and thermal infrared video pairs as a training set for the network model;

[0035] A network model construction module, used to construct a cross-modal target tracking network model; the cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module and a target classification regression network, the feature extraction backbone network includes a visible light backbone network and a thermal infrared backbone network with the same structure, the visible light backbone network and the thermal infrared backbone network respectively extract features of the template and the search area through convolution and perform adaptive convolution to extract timing information, and the adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network;

[0036] An initialization module is used to initialize the parameters of the cross-modal target tracking network model;

[0037] The training module is used to preprocess the training set data and perform model training. During training, a video sequence is selected, and two frames of visible light images and thermal infrared images are randomly taken out at the same time. The visible light and thermal infrared images are used as templates respectively, and the corresponding thermal infrared images and visible light images are used as search areas respectively.

[0038] And, an output module is used to input the data to be tracked into the trained cross-modal target tracking network model to obtain the tracking result.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] The present invention adopts online time-series adaptive convolution in the feature extraction network, and uses an adaptive method to update the weights of the last three layers of convolution of the backbone network, thereby utilizing time-series information to enhance the extracted features to adapt to different targets and environments, thereby improving tracking accuracy while reducing the amount of calculation and reducing memory usage.

[0041] The method of the present invention introduces an adaptive combination of space and channel attention on the basis of feature-level fusion, and uses dual attention to solve the problem of lack of spatial detail information, so that the model can simultaneously pay attention to the information of different spatial positions and different channels in the image, so that the obtained fused features have more robust feature representation capabilities and can maximize the use of complementary information between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of the cross-modal target tracking method based on a fully convolutional twin network of the present invention.

[0043] Figure 2 It is a structural diagram of the cross-modal target tracking network model of the present invention.

[0044] Figure 3 Schematic diagram of the structure of the feature extraction backbone network.

[0045] Figure 4 Schematic diagram of the structure of time-adaptive convolution in the feature extraction backbone network.

[0046] Figure 5 Schematic diagram of the structure of the adaptive complementary fusion module.

[0047] Figure 6 Schematic diagram of the structure of adaptive channel attention.

[0048] Figure 7 Schematic diagram of the structure of adaptive spatial attention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the embodiments, but are not intended to limit the present invention.

[0050] The cross-modal target tracking system based on the fully convolutional twin network of the embodiment of the present invention includes the following modules:

[0051] A training data construction module is used to construct a data set containing visible light and thermal infrared video pairs as a training set for the network model;

[0052] A network model building module is used to build a cross-modal target tracking network model; the cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module and a target classification and regression network. The feature extraction backbone network includes a visible light backbone network and a thermal infrared backbone network with the same structure. The visible light backbone network and the thermal infrared backbone network respectively extract features of the template and the search area through convolution and perform adaptive convolution to extract timing information. The adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network;

[0053] An initialization module is used to initialize the parameters of the cross-modal target tracking network model;

[0054] The training module is used to preprocess the training set data and perform model training. During training, a video sequence is selected, and two frames of visible light images and thermal infrared images are randomly taken out at the same time. The visible light and thermal infrared images are used as templates respectively, and the corresponding thermal infrared images and visible light images are used as search areas respectively.

[0055] And, an output module is used to input the data to be tracked into the trained cross-modal target tracking network model to obtain the tracking result.

[0056] Please combine Figure 1 and Figure 2 As shown in the figure, the cross-modal target tracking method based on the fully convolutional twin network implemented by the system includes the following steps:

[0057] Step 1: Construct cross-modal training and test sets. Merge the training sets of RGBT234 and LasHeR as the training set of the network model, and use the same operation method to obtain the test set.

[0058] Step 2: Construct a cross-modal target tracking network model. The cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module, and a target classification regression network.

[0059] This step specifically includes: Step 2.1, build a feature extraction backbone network, please combine Figure 3 As shown in Figure 1, the feature extraction backbone network includes a visible light (RGB) backbone network and a thermal infrared (TIR) ​​backbone network. The two have exactly the same structure and both use four-layer full convolution. The convolution kernel size of the first layer is 31×31×64, the convolution kernel size of the second layer is 31×31×256, the convolution kernel size of the third layer is 15×15×512, and the convolution kernel size of the fourth layer is 7×7×1024. In order to better integrate temporal information into the features, adaptive convolution is added after the third and fourth convolution layers, as shown in Figure 1. Figure 4 For any input X of adaptive convolution t , you can get the output

[0060]

[0061] Among them, the operator * represents the convolution operation, W t and b t are the weight and bias of the convolution respectively. and W b and b bis a learnable parameter. Since the temporal adaptive convolution processes one frame at a time, it only considers the temporal context of the previous frame. You can get:

[0062]

[0063] Among them, Cat represents the splicing operation, and the output features are pooled. It can be obtained by performing global average pooling (GAP) on the features of each frame, that is, For the calibration factor and Use two convolutions Perform convolution with a kernel size of L, that is Where f represents the convolution operation, and the initial weight of f is set to 0, so at initialization, W t =W b , b t =b b When t≤L-1, use the first frame to make up for it until t>L-1.

[0064] Step 2.2, design an adaptive complementary fusion module, such as Figure 5 As shown in the figure, the specific processing process of the feature through this module is as follows:

[0065] (1) After the visible light and thermal infrared images pass through the feature extraction backbone network, the visible light features of size H×W×C can be obtained respectively. and an infrared feature of size H×W×C

[0066] (2) Visible light features and thermal infrared characteristics By splicing according to channel C, we can get a connection feature of size H×W×2C

[0067] (3) Use dual attention mechanism to further extract The salient information in Input such as Figure 6 In the channel adaptive attention module shown, the visible light features after channel attention enhancement can be obtained and thermal infrared characteristics

[0068]

[0069] Among them, δ is the global pooling, ω is the fully connected layer, ε is Sigmoid, is the channel product, and Split is the operation of splitting features along the channel dimension.

[0070] (4) The output of adaptive channel attention Splicing along the channel dimension, the union feature is U s , then enter Figure 7 In the adaptive spatial attention module shown, the weight vector is represented as V s , we can get:

[0071]

[0072] Among them, ρ is the average pooling operation, φ is the maximum pooling operation, and Cat is the operation of splicing features along the channel dimension. is a two-dimensional convolution operation, W is the calculated spatial attention weight, ε is the Sigmoid function, and the output is expressed as Then we can get:

[0073]

[0074] Outputting the visible light features of spatial attention and thermal infrared characteristics Concatenate along the channel dimension to obtain the H×W×2C attention feature

[0075] (5) Connect the features Thermal infrared feature part and attention feature in The first set of features after fusion can be obtained by adding the visible light feature part element by element. Connect features The visible light feature part and attention feature in The thermal infrared feature parts in the fusion can be added element by element to obtain the second set of features.

[0076] (6) The two groups and After the features are concatenated according to the channels, the size is H×W×2C. At this time, a set of 1×1 convolutions is performed to reduce the channel dimension, thereby achieving feature fusion.

[0077] Step 2.3, build a target classification regression network: The target classification regression network consists of 12 Conv-bn-relu structures, which predict the target category, local offset and normalized bounding box size, that is, the position and size of the target.

[0078] Step 3. Initialize the weights and thresholds of all networks in the model, and set the network hyperparameters, including the network learning rate, number of iterations, and batch size. Specifically, the feature extraction backbone network used is ResNet-50, the last layer of Conv5 is removed, and the fourth layer Conv4 is used as the final feature output, where adaptive convolution is used in the 3×3 convolutions in the last two layers of Conv3 and Conv4 respectively. The input template image size is 128, and the search area image size is 256. In the temporal adaptive convolution, the value of L is set to 3, that is, three frames will be saved. The experiment uses stochastic gradient descent as the optimizer, and the initial learning rate is 10 -2 , the later learning rate is 10 -5 The batch size is set to 16, a total of 50 epochs are trained, the momentum is set to 0.9, and the quality decay is 5×10 -4 .

[0079] Step 4: preprocess the training set data. Select a video sequence in the training set, randomly take out two frames of visible light images and thermal infrared images at the same time, use the visible light and thermal infrared images as templates respectively, and then use the corresponding thermal infrared images and visible light images as search areas respectively.

[0080] The specific steps are:

[0081] Step 4.1, according to the category and bounding box of the target, perform data enhancement preprocessing on the video frames in the training set, including image rotation, image flipping, image scaling, image translation and grayscale operations;

[0082] Step 4.2: After data enhancement, the template image is resized to 128×128 pixels, and the search area image is resized to 256×256 pixels.

[0083] Step 4.3: Perform the preprocessing corresponding to steps 4.1 to 4.2 on all infrared images and visible light images.

[0084] Step 5: Use classification loss and regression loss to construct a loss function for training the network model.

[0085] Loss Function for:

[0086]

[0087] Among them, L cls represents the weighted focal loss, used for classification; L iou represents IoU loss for bounding box regression; L1 represents L1 loss for bounding box regression, λ iou and Represents hyperparameters, which represent the weights corresponding to IoU loss and L1 loss respectively.

[0088] Step 6: Use the gradient descent method to perform end-to-end model training on the entire tracking model until the loss value converges.

[0089] Step 7: Input the test set into the trained tracking model and use the output of the model as the tracking result. This includes the following steps:

[0090] Step 7.1. Randomly select a video sequence in the test set, use the first frame as the template, and the subsequent frames as the search area.

[0091] Step 7.2, crop the first frame according to the target's true bounding box position, that is, crop it with the target as the center and twice the size of the target box to obtain a template, and convert its size to 128×128 pixels; the subsequent frames are centered on the target obtained in the previous frame and cropped with 4 times the size of the target box, and the cropped image is converted to a picture of size 256×256 pixels.

[0092] Step 7.3: Input the cropped data into the network model, first pass through the feature extraction B backbone network, and then pass through the adaptive complementary fusion module to obtain the fusion features.

[0093] Step 7.4: Reshape the obtained fusion features into a two-dimensional spatial feature map and input it into the target classification and regression network. After passing through the fully convolutional neural network, the target classification map, local offset and normalized bounding box size are obtained. The position with the highest score in the target classification map is the target position. The local offset is used to correct the discretization error caused by the reduction in resolution. The bounding box is mapped to describe the size of the target.

[0094] The following is a test comparison of the method of this embodiment and various existing tracking methods on the data sets LasHeR and RGBT234.

[0095] Table 1 Accuracy and success rate of cross-modal target tracking on the LasHeR dataset

[0096] method Accuracy Success rate Method of the present invention 54.4 43.2 APFNet 50.0 36.2 MACNet 48.2 35.0 MANet 45.5 32.6 CAT 45.0 31.4 DAFNet 44.8 31.1 FANet 44.1 30.9 CMR 35.2 25.7 SGT 32.7 23.2

[0097] Table 2 Accuracy and success rate of cross-modal target tracking on the RGBT234 test set

[0098] Dataset Accuracy Success rate Method of the present invention 80.4 56.8 DAFNet 79.6 54.4 HMFT 78.8 56.9 MANet 77.7 53.9 DAPNet 76.6 53.7 SiamCDA 76.0 56.9 MDNet 71.0 49.0

[0099] As can be seen from Table 1, the cross-modal target tracking method based on the fully convolutional twin network proposed in the present invention has achieved satisfactory accuracy and success rate on the LasHeR test set. As can be seen from Table 2, the accuracy of the present invention on the RGBT234 test set is the highest, and the success rate is only 0.01% lower than the highest method. Therefore, the method of the present invention has obvious superiority.

Claims

1. A cross-modal target tracking method based on a fully convolutional twin network, characterized in that: The following steps are involved: Step 1: Construct a dataset containing visible light and thermal infrared video pairs as a training set for the network model; Step 2, constructing a cross-modal target tracking network model; the cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module and a target classification regression network, the feature extraction backbone network includes a visible light backbone network and a thermal infrared backbone network with the same structure, the visible light backbone network and the thermal infrared backbone network respectively extract the features of the template and the search area through convolution and perform adaptive convolution to extract timing information, and the adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network; Step 3: Initialize the parameters of the cross-modal target tracking network model; Step 4: Preprocess the training set data and perform model training. During training, select a video sequence, randomly extract two frames of visible light images and thermal infrared images at the same time, use the visible light and thermal infrared images as templates, and the corresponding thermal infrared images and visible light images as search areas; Step 5: Input the data to be tracked into the trained cross-modal target tracking network model to obtain the tracking result.

2. The cross-modal target tracking method based on a fully convolutional twin network according to claim 1, characterized in that: The visible light backbone network and the thermal infrared backbone network both include four convolutional layers, and an adaptive convolutional layer is arranged after the last two convolutional layers to extract timing information.

3. The cross-modal target tracking method based on a fully convolutional twin network according to claim 2, characterized in that: The structure of the adaptive convolution layer is: t , you can get the output Among them, the operator * represents the convolution operation, W b and b b are learnable parameters, Where f represents the convolution operation, Represents the input of the adaptive convolutional layer of L frames after pooling.

4. The cross-modal target tracking method based on a fully convolutional twin network according to claim 3 is characterized in that: When t≤L-1, the first frame is used to make up for it until t>L-1.

5. The cross-modal target tracking method based on a fully convolutional twin network according to claim 1, characterized in that: In step 2, the adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network as follows: The first set of features is obtained by element-wise adding the visible light features with the thermal infrared features after channel attention enhancement and spatial attention enhancement, and the second set of features is obtained by element-wise adding the thermal infrared features with the visible light features after channel attention enhancement and spatial attention enhancement. The first and second sets of features are then concatenated according to the channels and convolved for dimensionality reduction.

6. The cross-modal target tracking method based on a fully convolutional twin network according to claim 5, characterized in that: In step 2, the channel attention enhancement is performed by splicing the visible light feature and the thermal infrared feature to obtain the connection feature, and then performing the channel attention enhancement. The visible light feature after the channel attention enhancement is Thermal infrared characteristics are is the connection feature, where δ is the global pooling, ω is the fully connected layer, ε is Sigmoid, is the channel product, and Split is the operation of splitting features along the channel dimension.

7. The cross-modal target tracking method based on a fully convolutional twin network according to claim 6, characterized in that: The spatial attention enhancement in step 2 is to splice the visible light features and thermal infrared features after channel attention enhancement along the channel dimension to obtain the union feature U s , the visible light feature after spatial attention enhancement is Thermal infrared characteristics are Among them, ρ is the average pooling operation, φ is the maximum pooling operation, and Cat is the operation of splicing features along the channel dimension. is a two-dimensional convolution operation, and W is the calculated spatial attention weight.

8. The cross-modal target tracking method based on a fully convolutional twin network according to claim 1, characterized in that: The preprocessing in step 4 is to perform image rotation, image flipping, image scaling, image translation and grayscale operations.

9. The cross-modal target tracking method based on a fully convolutional twin network according to claim 1, characterized in that: The loss function when training the model is: L cls represents the weighted focal loss, L iou represents IoU loss, L1 represents L1 loss, λ iou is the hyperparameter representing the IoU loss weight, is the hyperparameter representing the L1 loss weight.

10. A cross-modal target tracking system based on a fully convolutional twin network, characterized in that: Includes the following modules: A training data construction module is used to construct a data set containing visible light and thermal infrared video pairs as a training set for the network model; A network model construction module, used to construct a cross-modal target tracking network model; the cross-modal target tracking network model includes a feature extraction backbone network, an adaptive complementary fusion module and a target classification regression network, the feature extraction backbone network includes a visible light backbone network and a thermal infrared backbone network with the same structure, the visible light backbone network and the thermal infrared backbone network respectively extract features of the template and the search area through convolution and perform adaptive convolution to extract timing information, and the adaptive complementary fusion module performs channel attention enhancement and spatial attention enhancement and fusion on the visible light features and thermal infrared features extracted by the feature extraction backbone network; An initialization module is used to initialize the parameters of the cross-modal target tracking network model; The training module is used to preprocess the training set data and perform model training. During training, a video sequence is selected, and two frames of visible light images and thermal infrared images are randomly taken out at the same time. The visible light and thermal infrared images are used as templates respectively, and the corresponding thermal infrared images and visible light images are used as search areas respectively. And, an output module is used to input the data to be tracked into the trained cross-modal target tracking network model to obtain the tracking result.

Images