RGBT real-time tracking method and system based on multi-mode interactive fusion

Abstract

The invention discloses an RGBT real-time tracking method and system based on multi-mode interactive fusion. The method comprises the following steps: obtaining a first feature corresponding to a visible light image and a first feature corresponding to an infrared image; obtaining a first fusion feature; obtaining a second feature corresponding to the visible light image and a second feature corresponding to the infrared image; obtaining a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; obtaining a second fusion feature; obtaining a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; acquiring a third fusion feature; performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; and carrying out pooling processing on the fourth fusion feature, and carrying out target identification through a full-connection network to obtain a real-time tracking result. According to the method, the RGBT real-time tracking performance and speed can be considered.

Description

Technical Field

[0001] The present invention relates to the field of target tracking, and in particular to an RGBT real-time tracking method and system based on multi-modal interactive fusion. Background Art

[0002] Target tracking is to predict the position and size of the target in the subsequent video sequence based on the state of the target in the first frame, so as to obtain the final motion trajectory. Since the object information provided by thermal infrared and visible light is complementary, RGBT target tracking fully integrates the two, which improves the tracking performance compared to single-modal target tracking, so RGBT target tracking has received widespread attention and application. For example, image recognition and tracking in unmanned driving, automatic detection of illegal turns of urban vehicles, and human-computer interaction require target tracking as a basis. Although RGBT target tracking has been widely studied and applied, how to effectively integrate the features of the two modalities to improve tracking accuracy and how to ensure the speed of accuracy improvement are still issues that need to be solved.

[0003] In response to the above problems, many algorithms have emerged to solve them. Among them, the algorithm based on multi-domain network (MDNet) has received extensive attention and research due to its high accuracy. In the prior art, a tracking framework of cross-modal mode propagation (CMPP) integrates intra-modal mode propagation (IMPP) and long-term context propagation (LTCP) modules, which can utilize the correlation between modalities and reduce the model drift during online tracking. In order to cope with the challenges of modality sharing and modality specific challenges, a challenge-aware neural network (CAT) has emerged in the prior art. In order to enhance the model to distinguish confusing samples, a multi-modal multi-interval metric learning method (M5L) is proposed in the prior art. In order to obtain more discriminative feature information, the prior art proposes a trident fusion network (TFNet). In order to make full use of the complementary characteristics between modalities, the prior art proposes a method based on modality-aware network and competitive learning. In order to effectively utilize implicit attribute information to improve model discriminability, the prior art proposes a network ADRNet based on attribute drive and representation. This attribute-driven design increases the sensitivity to target changes. In order to make full use of the discriminable information of low-quality modalities, the prior art proposes a binary gated mutual conditional network (DMCNet), which introduces a binary gating mechanism to improve the quality of the generated conditions and improve the robustness in complex scenarios. The prior art proposes a differentiated reinforcement and global collaboration network (DRGCNet), which effectively mines and integrates cross-modal global information. The prior art proposes a knowledge-driven learning framework (JTPMA), which improves the performance of the main task by making full use of information from different modes. In order to effectively model the appearance representation of the target under various challenging attributes and improve the robustness of tracking, the prior art designs a challenge-based appearance disentanglement and interaction network. Although the above algorithms have high accuracy, most of them cannot meet real-time requirements. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an RGBT real-time tracking method and system based on multimodal interactive fusion, which solves the problem that it is difficult to balance tracking performance and speed in the prior art during RGBT real-time tracking.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: A RGBT real-time tracking method based on multimodal interactive fusion is provided, which comprises the following steps: S1. Obtain a visible light image and an infrared image to be processed respectively; perform convolution processing on the visible light image and the infrared image respectively through two first convolution layers to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; S2, processing the first feature corresponding to the visible light image and the first feature corresponding to the infrared image through a gating mechanism after cascading to obtain a first fusion feature; S3, performing information interaction fusion on the first fusion feature, and processing the corresponding information interaction fusion result through a gating mechanism, and then adding the result element by element to the first feature corresponding to the visible light image and the first feature corresponding to the infrared image, so as to obtain a second feature corresponding to the visible light image and a second feature corresponding to the infrared image; S4. Perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image through two second convolution layers to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; S5, processing the third feature corresponding to the visible light image and the third feature corresponding to the infrared image through a gating mechanism after cascading to obtain a second fusion feature; S6, performing information interactive fusion on the second fusion feature, and adding the corresponding information interactive fusion result to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image element by element, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; S7, performing convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively through two third convolution layers, and cascading the processing results to obtain a third fusion feature; S8, performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; S9. After the fourth fusion feature is pooled, target recognition is performed through a fully connected network to obtain real-time tracking results.

[0006] Furthermore, in step S1, the two first convolutional layers have the same structure but different weights, and the convolution kernel size is 7×7; in step S4, the two second convolutional layers have the same structure but different weights, and the convolution kernel size is 5×5; in step S7, the two third convolutional layers have the same structure but different weights, and the convolution kernel size is 3×3.

[0007] Furthermore, the expression of the gating mechanism in step S2 and step S5 is:

[0008] in is the processing result of the gating mechanism; It is the processing object of the gating mechanism; It is a 1×1 convolution; Represents the Sigmoid function; Represents element-wise multiplication.

[0009] Furthermore, the expression of information interaction fusion in step S3 and step S6 is:

[0010]

[0011]

[0012]

[0013]

[0014]

[0015] in It is the processing object of information interaction and fusion; represents global average pooling; It is a 3×3 convolution; Represents the Sigmoid function; is the channel weight; , , and All are intermediate characteristics; is the position weight, ; Represents element-wise multiplication; Indicates that the processing object is divided into two parts; represents average pooling; represents maximum pooling; Indicates cascade; It is a 1×1 convolution; represents the softmax function; It means element-by-element addition; It is the result of information interaction fusion.

[0016] Furthermore, the expression of channel space fusion in step S8 is:

[0017]

[0018]

[0019]

[0020]

[0021] in It is the third fusion feature; is the first frame of visible light image; represents global average pooling; and All are 1×1 convolution; is the Relu function; represents the softmax function; represents the correlation calculation; represents channel maximum pooling; is the channel weight; and All are intermediate characteristics; is the weight of the spatial channel; and All are 3×3 convolutions; and It is two 3×3 deconvolutions; Represents the Sigmoid function; Indicates cascade; Represents element-wise multiplication; It means element-by-element addition; It is the channel space fusion result, that is, the fourth fusion feature.

[0022] Furthermore, in step S9, the fully connected network includes three fully connected layers connected in series.

[0023] Provided is an RGBT real-time tracking system based on multimodal interactive fusion, which includes a feature extraction network, a first information interactive fusion module, a second information interactive fusion module, a channel space fusion module and a target recognition module; The feature extraction network includes two first convolutional layers, a first cascade module, a second cascade module, a third cascade module, a first gating mechanism module, a second gating mechanism module, a third gating mechanism module, a first element-by-element addition module, a second element-by-element addition module, two second convolutional layers, a third element-by-element addition module, a fourth element-by-element addition module and two third convolutional layers; Two first convolutional layers, used to perform convolution processing on the visible light image and the infrared image respectively, to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; A first cascading module, used to cascade the first feature corresponding to the visible light image and the first feature corresponding to the infrared image; A first gating mechanism module, used to process the output of the first cascade module through a gating mechanism to obtain a first fusion feature; A first information interaction fusion module, used for performing information interaction fusion on the first fusion feature; A second gating mechanism module, used for performing gating mechanism processing on the output of the first information interaction fusion module; A first element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the visible light image element-by-element to obtain a second feature corresponding to the visible light image; A second element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the infrared image element-by-element to obtain a second feature corresponding to the infrared image; Two second convolutional layers, used to perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image, respectively, to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; A second cascading module, used for cascading the third feature corresponding to the visible light image and the third feature corresponding to the infrared image; A third gating mechanism module is used to perform gating mechanism processing on the output of the second cascade module to obtain a second fusion feature; A second information interaction fusion module, used for performing information interaction fusion on the second fusion feature; A third element-by-element adding module is used to perform element-by-element addition of the output of the second information interaction fusion module to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image, respectively, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; Two third convolutional layers, used to perform convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively; The fourth element-by-element addition module is used to cascade the outputs of the two third convolutional layers to obtain a third fusion feature; A channel space fusion module, used for performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; The target recognition module includes a pooling processing unit and a fully connected network; A pooling processing unit, used for performing pooling processing on the fourth fusion feature; The fully connected network is used to identify the target on the output of the pooling processing unit and obtain real-time tracking results.

[0024] Furthermore, each gating mechanism module includes a first 1×1 convolutional layer, a first sigmoid activation layer and a first element-by-element multiplication layer connected in sequence; in the same gating mechanism module, the input end of the first 1×1 convolutional layer and the other input end of the first element-by-element multiplication layer are jointly used as the input end of the gating mechanism module, and the output end of the first element-by-element multiplication layer is the output end of the gating mechanism module.

[0025] Furthermore, each information interaction fusion module includes a first global average pooling layer, a first 3×3 convolution layer, a second sigmoid activation layer, a second element-by-element multiplication layer and a separation layer connected in sequence; the first output end of the separation layer is respectively connected to the third element-by-element multiplication layer, the first average pooling layer and the first maximum pooling layer; the second output end of the separation layer is respectively connected to the fourth element-by-element multiplication layer, the second average pooling layer and the second maximum pooling layer; the output ends of the first average pooling layer, the first maximum pooling layer, the second average pooling layer and the second maximum pooling layer are respectively connected to the input end of the third cascade module; the output end of the third cascade module is sequentially connected to the second 1×1 convolution layer and the first softmax layer; the output end of the first softmax layer is respectively connected to another input end of the third element-by-element multiplication layer and another output end of the fourth element-by-element multiplication layer; the output end of the third element-by-element multiplication layer and the output end of the fourth element-by-element multiplication layer are respectively connected to the input end of the fifth element-by-element addition module; the output end of the fifth element-by-element addition module is the output end of the information interaction fusion module; the input end of the first global average pooling layer is the input end of the information interaction fusion module.

[0026] Further, the channel space fusion module includes a channel fusion branch and a space fusion branch; The channel fusion branch includes a second global average pooling layer, a third 1×1 convolution layer, a relu layer, a fourth 1×1 convolution layer, and a fifth element-by-element multiplication layer connected in sequence, another input of the fifth element-by-element multiplication layer is the third fusion feature, and the output of the fifth element-by-element multiplication layer is the output of the channel fusion branch; The spatial fusion branch includes a correlation calculation layer and a channel maximum pooling layer connected in sequence, and a second 3×3 convolution layer, a third 3×3 convolution layer, a first 3×3 deconvolution layer, and a second 3×3 deconvolution layer connected in sequence; the two inputs of the correlation calculation layer are the third fusion feature and the first frame of visible light image, respectively; the input of the second 3×3 convolution layer is the third fusion feature; the output end of the channel maximum pooling layer and the output end of the second 3×3 deconvolution layer are respectively connected to the input end of the fourth cascade module; the output end of the fourth cascade module is connected to the second softmax layer and the third sigmoid activation layer in sequence; the output of the third sigmoid activation layer is the output of the spatial fusion branch; The output of the channel fusion branch and the output of the spatial fusion branch are both inputs to the sixth element-by-element multiplication layer; the output of the sixth element-by-element multiplication layer and the third fusion feature are both inputs to the sixth element-by-element addition module; the output of the sixth element-by-element addition module is the output of the channel space fusion module.

[0027] The beneficial effects of the present invention are: 1. The present invention can take into account both the performance and speed of RGBT real-time tracking.

[0028] 2. The information interaction fusion / information interaction fusion module adopted in the present invention can fuse and enhance feature information at different levels, so as to obtain richer feature information and improve tracking performance.

[0029] 3. The channel space fusion / channel space fusion module adopted in the present invention finally fuses the global information from two dimensions, channel and space, so as to obtain a more robust feature expression and improve the tracking performance.

[0030] 4. The gating mechanism / gating mechanism module adopted by the present invention can reduce the propagation of noise and improve the tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of the process of this method; Figure 2 This is the overall structural diagram of the system; Figure 3 It is the structural block diagram of the gating mechanism module; Figure 4 It is the structural block diagram of the information interaction fusion module; Figure 5 It is the structural block diagram of the channel space fusion module; Figure 6 This is a graph showing the accuracy evaluation results of the present invention on the RGBT234 dataset; Figure 7 This is a graph showing the success rate evaluation results of the present invention on the RGBT234 dataset; Figure 8 This is a graph showing the accuracy evaluation results of the present invention on the LasHer dataset; Fig. 9 This is a graph showing the success rate evaluation results of the present invention on the LasHer dataset. DETAILED DESCRIPTION

[0032] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0033] like Figure 1 As shown, the RGBT real-time tracking method based on multimodal interactive fusion includes the following steps: S1. Obtain a visible light image and an infrared image to be processed respectively; perform convolution processing on the visible light image and the infrared image respectively through two first convolution layers to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; S2, processing the first feature corresponding to the visible light image and the first feature corresponding to the infrared image through a gating mechanism after cascading to obtain a first fusion feature; S3, performing information interaction fusion on the first fusion feature, and processing the corresponding information interaction fusion result through a gating mechanism, and then adding the result element by element to the first feature corresponding to the visible light image and the first feature corresponding to the infrared image, so as to obtain a second feature corresponding to the visible light image and a second feature corresponding to the infrared image; S4. Perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image through two second convolution layers to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; S5, processing the third feature corresponding to the visible light image and the third feature corresponding to the infrared image through a gating mechanism after cascading to obtain a second fusion feature; S6, performing information interactive fusion on the second fusion feature, and adding the corresponding information interactive fusion result to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image element by element, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; S7, performing convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively through two third convolution layers, and cascading the processing results to obtain a third fusion feature; S8, performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; S9. After the fourth fusion feature is pooled, target recognition is performed through a fully connected network to obtain real-time tracking results.

[0034] In step S1, the two first convolutional layers have the same structure but different weights, and the convolution kernel size is 7×7; in step S4, the two second convolutional layers have the same structure but different weights, and the convolution kernel size is 5×5; in step S7, the two third convolutional layers have the same structure but different weights, and the convolution kernel size is 3×3.

[0035] like Figure 3 As shown, the expression of the gating mechanism in step S2 and step S5 is:

[0036] in is the processing result of the gating mechanism; It is the processing object of the gating mechanism; It is a 1×1 convolution; Represents the Sigmoid function; Represents element-wise multiplication.

[0037] like Figure 4 As shown, the expression of information interaction fusion in step S3 and step S6 is:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] in It is the processing object of information interaction and fusion; represents global average pooling; It is a 3×3 convolution; Represents the Sigmoid function; is the channel weight, which is used to improve the representation ability of the network; , , and All are intermediate characteristics; is the position weight, ; Represents element-wise multiplication; Indicates that the processing object is divided into two parts; represents average pooling; represents maximum pooling; Indicates cascade; It is a 1×1 convolution; represents the softmax function; It means element-by-element addition; It is the result of information interaction fusion.

[0044] like Figure 5 As shown, the expression of channel space fusion in step S8 is:

[0045]

[0046]

[0047]

[0048]

[0049] in It is the third fusion feature; is the first frame of visible light image; represents global average pooling; and All are 1×1 convolution; is the Relu function; represents the softmax function; represents the correlation calculation; represents channel maximum pooling; is the channel weight; and All are intermediate characteristics; is the weight of the spatial channel; and All are 3×3 convolutions; and It is two 3×3 deconvolutions; Represents the Sigmoid function; Indicates cascade; Represents element-wise multiplication; It means element-by-element addition; It is the channel space fusion result, that is, the fourth fusion feature.

[0050] The fully connected network in step S9 includes three fully connected layers connected in series.

[0051] like Figure 2 As shown, corresponding to the RGBT real-time tracking method based on multimodal interactive fusion, the RGBT real-time tracking system based on multimodal interactive fusion includes a feature extraction network, a first information interactive fusion module, a second information interactive fusion module, a channel space fusion module and a target recognition module; The feature extraction network includes two first convolutional layers, a first cascade module, a second cascade module, a third cascade module, a first gating mechanism module, a second gating mechanism module, a third gating mechanism module, a first element-by-element addition module, a second element-by-element addition module, two second convolutional layers, a third element-by-element addition module, a fourth element-by-element addition module and two third convolutional layers; Two first convolutional layers, used to perform convolution processing on the visible light image and the infrared image respectively, to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; A first cascading module, used to cascade the first feature corresponding to the visible light image and the first feature corresponding to the infrared image; A first gating mechanism module, used to process the output of the first cascade module through a gating mechanism to obtain a first fusion feature; A first information interaction fusion module, used for performing information interaction fusion on the first fusion feature; A second gating mechanism module, used for performing gating mechanism processing on the output of the first information interaction fusion module; A first element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the visible light image element-by-element to obtain a second feature corresponding to the visible light image; A second element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the infrared image element-by-element to obtain a second feature corresponding to the infrared image; Two second convolutional layers, used to perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image, respectively, to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; A second cascading module, used for cascading the third feature corresponding to the visible light image and the third feature corresponding to the infrared image; A third gating mechanism module is used to perform gating mechanism processing on the output of the second cascade module to obtain a second fusion feature; A second information interaction fusion module, used for performing information interaction fusion on the second fusion feature; A third element-by-element adding module is used to perform element-by-element addition of the output of the second information interaction fusion module to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image, respectively, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; Two third convolutional layers, used to perform convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively; The fourth element-by-element addition module is used to cascade the outputs of the two third convolutional layers to obtain a third fusion feature; A channel space fusion module, used for performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; The target recognition module includes a pooling processing unit and a fully connected network; A pooling processing unit, used for performing pooling processing on the fourth fusion feature; The fully connected network is used to identify the target on the output of the pooling processing unit and obtain real-time tracking results.

[0052] like Figure 3 As shown ( Indicates element-by-element multiplication), each gating mechanism module includes the first 1×1 convolution layer, the first sigmoid activation layer and the first element-by-element multiplication layer connected in sequence; in the same gating mechanism module, the input end of the first 1×1 convolution layer and the other input end of the first element-by-element multiplication layer are used as the input end of the gating mechanism module, and the output end of the first element-by-element multiplication layer is the output end of the gating mechanism module. The gating mechanism module is mainly used to reduce the propagation of noise.

[0053] like Figure 4 As shown, each information interaction fusion module includes a first global average pooling layer, a first 3×3 convolution layer, a second sigmoid activation layer, a second element-by-element multiplication layer and a separation layer connected in sequence; the first output end of the separation layer is respectively connected to the third element-by-element multiplication layer, the first average pooling layer and the first maximum pooling layer; the second output end of the separation layer is respectively connected to the fourth element-by-element multiplication layer, the second average pooling layer and the second maximum pooling layer; the output ends of the first average pooling layer, the first maximum pooling layer, the second average pooling layer and the second maximum pooling layer are respectively connected to the input end of the third cascade module; the output end of the third cascade module is sequentially connected to the second 1×1 convolution layer and the first softmax layer; the output end of the first softmax layer is respectively connected to another input end of the third element-by-element multiplication layer and another output end of the fourth element-by-element multiplication layer; the output end of the third element-by-element multiplication layer and the output end of the fourth element-by-element multiplication layer are respectively connected to the input end of the fifth element-by-element addition module; the output end of the fifth element-by-element addition module is the output end of the information interaction fusion module; the input end of the first global average pooling layer is the input end of the information interaction fusion module.

[0054] like Figure 5 As shown, in order to better fuse the global features, the channel space fusion module includes a channel fusion branch and a space fusion branch; The channel fusion branch includes a second global average pooling layer, a third 1×1 convolution layer, a relu layer, a fourth 1×1 convolution layer, and a fifth element-by-element multiplication layer connected in sequence, another input of the fifth element-by-element multiplication layer is the third fusion feature, and the output of the fifth element-by-element multiplication layer is the output of the channel fusion branch; The spatial fusion branch includes a correlation calculation layer and a channel maximum pooling layer connected in sequence, and a second 3×3 convolution layer, a third 3×3 convolution layer, a first 3×3 deconvolution layer, and a second 3×3 deconvolution layer connected in sequence; the two inputs of the correlation calculation layer are the third fusion feature and the first frame of visible light image, respectively; the input of the second 3×3 convolution layer is the third fusion feature; the output end of the channel maximum pooling layer and the output end of the second 3×3 deconvolution layer are respectively connected to the input end of the fourth cascade module; the output end of the fourth cascade module is connected to the second softmax layer and the third sigmoid activation layer in sequence; the output of the third sigmoid activation layer is the output of the spatial fusion branch; The output of the channel fusion branch and the output of the spatial fusion branch are both inputs to the sixth element-by-element multiplication layer; the output of the sixth element-by-element multiplication layer and the third fusion feature are both inputs to the sixth element-by-element addition module; the output of the sixth element-by-element addition module is the output of the channel space fusion module.

[0055] The spatial fusion branch is designed as a dual-stream branch. The first branch uses the cross-correlation to calculate the correlation between the initial frame and the current frame, and then uses the channel maximum pooling operation to aggregate and obtain the feature map. In order to adapt it to the target transformation, there is a second branch, which consists of two convolutions with 3×3 kernels and two deconvolutions. The feature map is obtained through this branch. , then the two feature maps are cascaded, and the weights of the spatial channels are obtained using the softmax function and the sigmoid function This can enhance the information while suppressing the interference at the spatial level.

[0056] In one embodiment of the present invention, end-to-end training is adopted. In each training, eight frames of images are first randomly selected from the video sequence in the data set, and each frame of the image is cropped to 107×107. Then, 64 positive samples and 196 negative samples are selected around the target using Gaussian distribution sampling. When the overlap rate (IOU) between the bounding box generated by the sampling and the real bounding box exceeds 0.7, it is a positive sample, and when it is lower than 0.5, it is a negative sample. During the training process, the stochastic gradient descent (SGD) algorithm is sampled to update the network parameters. The learning rate is set to 0.001, the weight decay is 0.0005, the momentum is set to 0.9, and the network is trained for 300 iterations. This embodiment uses the GTOT data set for training and is tested on the RGBT234 and LasHeR data sets.

[0057] When training the fully connected network, first freeze all parameters except the three fully connected layers, and randomly initialize the parameters of the third fully connected layer. Set the learning rate of the first and second fully connected layers to 0.0002, and the learning rate of the third fully connected layer to 0.002, and use the video sequence to fine-tune the fully connected network to adapt to the target changes. Specifically, the first frame of each video sequence is Gaussian sampled to obtain 500 positive samples and 5000 negative samples, and the fully connected network is trained for 50 rounds. This embodiment uses short-term updates and long-term updates to update the parameters of the fully connected network. In order to improve the tracking accuracy, the bounding box regression technology is used, in which the bounding box regressor uses the first frame to adjust the target state when successfully tracked. In subsequent tracking, Gaussian sampling is performed with the target prediction position of the previous frame as the center to obtain 256 candidate boxes, which are used as input to obtain the prediction score, and the fifth candidate box with the highest score is averaged to obtain an average position, which is used as the final prediction result of the current frame. Repeat this until the entire video sequence is tracked.

[0058] To evaluate the effectiveness of the present invention, the performance is compared with 9 trackers on the RGBT234 dataset. Figure 6 and Figure 7 As shown in the figure, the PR and SR of the present invention are 84.9% and 61.8% respectively, achieving the best performance. Compared with the current better tracker CMPP, the PR and SR are improved by 2.1% and 4.3% respectively. From the comparison of the experimental results, it can be seen that the overall performance of the present invention is better.

[0059] In order to further evaluate the effectiveness of the present invention, the performance of the proposed method is compared with that of 9 trackers on the more challenging LasHeR dataset. Figure 8 and Fig. 9 As shown. The overall performance of the present invention is relatively good. Although the success rate is 1.0% lower than that of mfDiMP, the accuracy is improved by 3.4% compared with mfDiMP. Compared with other algorithms, both the accuracy and success rate are greatly improved. The above test results show that the present invention has good robustness in dealing with various challenges.

[0060] In order to further explore the performance of the present invention under different challenge attributes, the present invention is compared with the five trackers of CAT, M5L, ADRNet, DFAT and APFNet under the 12 challenges in RGBT234. The comparison results are shown in Table 1, where the best performance is bolded. It can be seen from Table 1 that the overall performance of the algorithm in this paper is the best, and except for the fast-moving challenge, PR and SR can show the best performance in the comparison algorithms when facing other challenges, which also shows that the algorithm proposed in this paper can better cope with most challenges.

[0061] Table 2 shows the speed comparison between the present invention and other multi-domain network-based algorithms. The speed of the present invention can reach 30.07fps, which is 20.28% faster than the ADRNet tracker. The speed improvement is even higher than other multi-domain network algorithms. It can be seen that the tracking speed of the present invention is not only high but also can meet the real-time requirements, so the present invention has certain advantages in speed.

[0062] Table 1

[0063] Table 2

[0064] In conclusion, the present invention can take into account both the performance and speed of RGBT real-time tracking.

Claims

1. A RGBT real-time tracking method based on multimodal interactive fusion, characterized in that: The following steps are involved: S1. Obtain a visible light image and an infrared image to be processed respectively; perform convolution processing on the visible light image and the infrared image respectively through two first convolution layers to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; S2, processing the first feature corresponding to the visible light image and the first feature corresponding to the infrared image through a gating mechanism after cascading to obtain a first fusion feature; S3, performing information interaction fusion on the first fusion feature, and processing the corresponding information interaction fusion result through a gating mechanism, and then adding the result element by element to the first feature corresponding to the visible light image and the first feature corresponding to the infrared image, so as to obtain a second feature corresponding to the visible light image and a second feature corresponding to the infrared image; S4. Perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image through two second convolution layers to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; S5, processing the third feature corresponding to the visible light image and the third feature corresponding to the infrared image through a gating mechanism after cascading to obtain a second fusion feature; S6, performing information interactive fusion on the second fusion feature, and adding the corresponding information interactive fusion result to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image element by element, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; S7, performing convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively through two third convolution layers, and cascading the processing results to obtain a third fusion feature; S8, performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; S9. After the fourth fusion feature is pooled, target recognition is performed through a fully connected network to obtain real-time tracking results.

2. The RGBT real-time tracking method based on multimodal interactive fusion according to claim 1 is characterized in that: In step S1, the two first convolutional layers have the same structure but different weights, and the convolution kernel size is 7×7; in step S4, the two second convolutional layers have the same structure but different weights, and the convolution kernel size is 5×5; in step S7, the two third convolutional layers have the same structure but different weights, and the convolution kernel size is 3×3.

3. The RGBT real-time tracking method based on multimodal interactive fusion according to claim 1 is characterized in that: The expression of the gating mechanism in step S2 and step S5 is: in is the processing result of the gating mechanism; It is the processing object of the gating mechanism; It is a 1×1 convolution; Represents the Sigmoid function; Represents element-wise multiplication.

4. The RGBT real-time tracking method based on multimodal interactive fusion according to claim 1 is characterized in that: The expression of information interaction fusion in step S3 and step S6 is: in It is the processing object of information interaction and fusion; represents global average pooling; It is a 3×3 convolution; Represents the Sigmoid function; is the channel weight; , , and All are intermediate characteristics; is the position weight, ; Represents element-wise multiplication; Indicates that the processing object is divided into two parts; represents average pooling; represents maximum pooling; Indicates cascade; It is a 1×1 convolution; represents the softmax function; It means element-by-element addition; It is the result of information interaction fusion.

5. The RGBT real-time tracking method based on multimodal interactive fusion according to claim 1 is characterized in that: The expression of channel space fusion in step S8 is: in It is the third fusion feature; is the first frame of visible light image; represents global average pooling; and All are 1×1 convolution; is the Relu function; represents the softmax function; represents the correlation calculation; Indicates channel maximum pooling; is the channel weight; and All are intermediate characteristics; is the weight of the spatial channel; and All are 3×3 convolutions; and It is two 3×3 deconvolutions; Represents the Sigmoid function; Indicates cascade; Represents element-wise multiplication; It means element-by-element addition; It is the channel space fusion result, that is, the fourth fusion feature.

6. The RGBT real-time tracking method based on multimodal interactive fusion according to claim 1 is characterized in that: The fully connected network in step S9 includes three fully connected layers connected in series.

7. A RGBT real-time tracking system based on multimodal interactive fusion, characterized in that: It includes a feature extraction network, a first information interaction fusion module, a second information interaction fusion module, a channel space fusion module and a target recognition module; The feature extraction network includes two first convolutional layers, a first cascade module, a second cascade module, a third cascade module, a first gating mechanism module, a second gating mechanism module, a third gating mechanism module, a first element-by-element addition module, a second element-by-element addition module, two second convolutional layers, a third element-by-element addition module, a fourth element-by-element addition module and two third convolutional layers; Two first convolutional layers, used to perform convolution processing on the visible light image and the infrared image respectively, to obtain a first feature corresponding to the visible light image and a first feature corresponding to the infrared image; A first cascading module, used to cascade the first feature corresponding to the visible light image and the first feature corresponding to the infrared image; A first gating mechanism module, used to process the output of the first cascade module through a gating mechanism to obtain a first fusion feature; A first information interaction fusion module, used for performing information interaction fusion on the first fusion feature; A second gating mechanism module, used for performing gating mechanism processing on the output of the first information interaction fusion module; A first element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the visible light image element-by-element to obtain a second feature corresponding to the visible light image; A second element-by-element adding module, used for adding the output of the second gating mechanism module and the first feature corresponding to the infrared image element-by-element to obtain a second feature corresponding to the infrared image; Two second convolutional layers, used to perform convolution processing on the second feature corresponding to the visible light image and the second feature corresponding to the infrared image, respectively, to obtain a third feature corresponding to the visible light image and a third feature corresponding to the infrared image; A second cascading module, used for cascading the third feature corresponding to the visible light image and the third feature corresponding to the infrared image; A third gating mechanism module is used to perform gating mechanism processing on the output of the second cascade module to obtain a second fusion feature; A second information interaction fusion module, used for performing information interaction fusion on the second fusion feature; A third element-by-element adding module is used to perform element-by-element addition of the output of the second information interaction fusion module to the third feature corresponding to the visible light image and the third feature corresponding to the infrared image, respectively, to obtain a fourth feature corresponding to the visible light image and a fourth feature corresponding to the infrared image; Two third convolutional layers, used to perform convolution processing on the fourth feature corresponding to the visible light image and the fourth feature corresponding to the infrared image respectively; The fourth element-by-element addition module is used to cascade the outputs of the two third convolutional layers to obtain a third fusion feature; A channel space fusion module, used for performing channel space fusion on the third fusion feature and the first frame of visible light image to obtain a fourth fusion feature; The target recognition module includes a pooling processing unit and a fully connected network; A pooling processing unit, used for performing pooling processing on the fourth fusion feature; The fully connected network is used to identify the target on the output of the pooling processing unit and obtain real-time tracking results.

8. The RGBT real-time tracking system based on multimodal interactive fusion according to claim 7 is characterized in that: Each gating mechanism module includes a first 1×1 convolutional layer, a first sigmoid activation layer and a first element-by-element multiplication layer connected in sequence; in the same gating mechanism module, the input end of the first 1×1 convolutional layer and the other input end of the first element-by-element multiplication layer are used as the input end of the gating mechanism module, and the output end of the first element-by-element multiplication layer is the output end of the gating mechanism module.

9. The RGBT real-time tracking system based on multimodal interactive fusion according to claim 7, characterized in that: Each information interaction fusion module includes a first global average pooling layer, a first 3×3 convolution layer, a second sigmoid activation layer, a second element-by-element multiplication layer and a separation layer connected in sequence; the first output end of the separation layer is connected to the third element-by-element multiplication layer, the first average pooling layer and the first maximum pooling layer respectively; the second output end of the separation layer is connected to the fourth element-by-element multiplication layer, the second average pooling layer and the second maximum pooling layer respectively; the output ends of the first average pooling layer, the first maximum pooling layer, the second average pooling layer and the second maximum pooling layer are connected to the input end of the third cascade module respectively; the output end of the third cascade module is connected to the second 1×1 convolution layer and the first softmax layer in sequence; the output end of the first softmax layer is connected to another input end of the third element-by-element multiplication layer and another output end of the fourth element-by-element multiplication layer respectively; the output end of the third element-by-element multiplication layer and the output end of the fourth element-by-element multiplication layer are connected to the input end of the fifth element-by-element addition module respectively; the output end of the fifth element-by-element addition module is the output end of the information interaction fusion module; the input end of the first global average pooling layer is the input end of the information interaction fusion module.

10. The RGBT real-time tracking system based on multimodal interactive fusion according to claim 7, characterized in that: The channel space fusion module includes a channel fusion branch and a space fusion branch; The channel fusion branch includes a second global average pooling layer, a third 1×1 convolution layer, a relu layer, a fourth 1×1 convolution layer, and a fifth element-by-element multiplication layer connected in sequence, another input of the fifth element-by-element multiplication layer is the third fusion feature, and the output of the fifth element-by-element multiplication layer is the output of the channel fusion branch; The spatial fusion branch includes a correlation calculation layer and a channel maximum pooling layer connected in sequence, and a second 3×3 convolution layer, a third 3×3 convolution layer, a first 3×3 deconvolution layer, and a second 3×3 deconvolution layer connected in sequence; the two inputs of the correlation calculation layer are the third fusion feature and the first frame of visible light image, respectively; the input of the second 3×3 convolution layer is the third fusion feature; the output end of the channel maximum pooling layer and the output end of the second 3×3 deconvolution layer are respectively connected to the input end of the fourth cascade module; the output end of the fourth cascade module is connected to the second softmax layer and the third sigmoid activation layer in sequence; the output of the third sigmoid activation layer is the output of the spatial fusion branch; The output of the channel fusion branch and the output of the spatial fusion branch are both inputs to the sixth element-by-element multiplication layer; the output of the sixth element-by-element multiplication layer and the third fusion feature are both inputs to the sixth element-by-element addition module; the output of the sixth element-by-element addition module is the output of the channel space fusion module.

Images