Cross-modal visual tracking method and device based on adaptive convolution

Abstract

The present invention discloses a cross-modal visual tracking method and device based on adaptive convolution, which belongs to the field of computer vision technology, including: inputting a pair of registered multi-modal images, generating a weight tensor corresponding to the size of the feature map after each layer of convolution through an adaptive convolution module, adaptively fusing the inputs between different modalities pixel by pixel, and adaptively fusing the fusion result with the single input modal feature again to realize cross-modal information interaction and single modal information enhancement; collecting samples according to the first frame of each video to fine-tune the fully connected layer to cope with instance-specific challenges; and finally sending the samples to the last layer of the fully connected layer for binary classification to obtain the final prediction result. By performing an adaptive fusion operation on the fusion result and the features of a single modality, feature complementarity between modalities is realized, thereby improving the performance of the tracking model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-modal visual tracking method and device based on adaptive convolution. Background Art

[0002] Object tracking is a hot topic in the field of computer vision and one of the key technologies for driverless cars, intelligent transportation and intelligent monitoring. The current detection and tracking method mainly trains a binary classifier to effectively distinguish between background and foreground. First, under the condition of the bounding box of the first frame in the video sequence, a specific threshold is set to determine the positive and negative samples. In the subsequent frames of the video, Gaussian sampling is still performed near the result predicted by the previous frame, and the obtained samples are classified into positive and negative samples. The one with the highest score in the positive sample is the result predicted for the next frame.

[0003] Most of the current tracking algorithms are based on single-mode visible light conditions, but they are greatly affected by some extreme conditions, such as bad weather and strong changes in lighting. Single-mode tracking algorithms often perform unsatisfactorily because visible light images are sensitive to light and are seriously affected by challenging environments such as lighting. Thermal infrared images are mainly based on the surface temperature of objects in the scene and are insensitive to changes in lighting, so they have a natural advantage in dealing with challenges such as lighting. The only drawback is that thermal infrared images also have blurred edges and are greatly affected by temperature.

[0004] Since visible light and infrared thermal imaging data can show different feature information in different scenes, making full use of the complementary information under multimodality can enable the tracker to track the target more accurately in complex situations. In the process of multimodal feature fusion, the current RGBT tracking algorithm uses two methods: one is to splice or add together multiple modal inputs as one input; the other is to process the features of each modality independently, and add or splice the processed features together as the result of multimodal feature fusion. If the imaging effect of a certain modality is very poor, and the importance of each modality is treated equally, it will bring noise to the network and reduce the tracking effect.

[0005] Existing feature fusion methods perform a simple weighted sum operation on the two modal features after convolution, and only consider the overall usefulness of a single modality. For example, for the input image, the visible light modal features produce a lot of useless information, while the thermal infrared modal features are more useful. In fact, the features of the visible spectrum model may not be completely useless and may contain some useful information, while the features of the thermal infrared modality may also contain some noise information. This results in insufficient fusion of different modal features and cannot effectively guarantee the accuracy of target tracking.

[0006] In the related technology, the invention patent application with application number 202110359997.X discloses an RGBT target tracking method based on cross-modal sharing and specific representation, which uses the VGG-M network to extract modality-specific features of different modalities, and uses a shared feature extraction module to extract modality shared features between the two modalities. At the same time, a multi-scale feature fusion strategy is used in the shared feature extraction module to enhance the features to improve its robustness.

[0007] However, it connects a separate attention module after each convolution layer, and there is no feature sharing between them. The common feature module only shares weights for different modalities. For a single modality, the common feature modality reduces the size of the second layer features and adjusts the number of channels, and then directly adds them to the corresponding third layer features. The fusion between the two modalities is just a simple splicing of the two modalities together using the splicing operation at the end of the convolution layer. It focuses on achieving robustness to multi-scale objects and the common characteristics of the two modalities.

[0008] In summary, the existing feature fusion methods have the following defects:

[0009] (1) Different modal inputs are concatenated together as a whole input, or the features of each modality are processed independently and the processed features are weighted or concatenated together as a whole as the result of multimodal feature fusion. These methods cannot fully consider the complementary effects of different modal inputs.

[0010] (2) Since the manually designed weight parameters or generation methods are fixed and do not take into account the differences between different inputs and image inputs under different modalities, it is difficult to cope with various challenges in complex scenes. Summary of the invention

[0011] The technical problem to be solved by the present invention is to solve the problem of insufficient fusion of different modal features.

[0012] The present invention solves the above technical problems through the following technical means:

[0013] On the one hand, the present invention proposes a cross-modal visual tracking method based on adaptive convolution, which is used to implement target tracking using a tracking model. The method comprises the following steps:

[0014] Acquire a visible light image and a thermal infrared image, and fuse the visible light image and the thermal infrared image to obtain an initial feature map;

[0015] Performing feature extraction on the visible light image, the thermal infrared image, and the initial feature map respectively to obtain a feature map of the visible light image, a feature map of the thermal infrared image, and a sub-feature map of the initial feature map;

[0016] Performing adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and performing adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image, respectively, to obtain a second fused feature map and a third fused feature map;

[0017] fusing the first fused feature map, the second fused feature map, and the third fused feature map to obtain a bimodal enhanced feature map;

[0018] Performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map;

[0019] Based on the calibrated feature map, an object position of a current frame in the multimodal video is predicted.

[0020] The present invention performs feature extraction and feature fusion on different modal inputs pixel by pixel, and adaptively convolves the result of adaptive convolution with the single modal feature again to complete the information supplement and enhancement of the single modal feature, thereby realizing information exchange and information supplement between different modalities, so as to make full use of the complementary advantages of different features in RGBT tracking to further improve the tracking performance.

[0021] Furthermore, the method further comprises:

[0022] A pair of first frame images are obtained from a pair of registered multimodal videos, and Gaussian distribution sampling is performed with the center points of the two true value frames of the first frame images as the mean to generate candidate samples, wherein the candidate samples include the visible light image and the thermal infrared image for acquisition by the tracking model.

[0023] Furthermore, the fusing of the visible light image and the thermal infrared image to obtain an initial feature map includes:

[0024] The visible light image and the thermal infrared image are spliced ​​together and then convolved to obtain the initial feature map.

[0025] Further, the fusing the first fused feature map, the second fused feature map and the third fused feature map to obtain a bimodal enhanced feature map includes:

[0026] Performing feature extraction on the first fused feature map, the second fused feature map, and the third fused feature map respectively to obtain a sub-feature map of the first fused feature map, a sub-feature map of the second fused feature map, and a sub-feature map of the third fused feature map;

[0027] Adaptively convolutionally fuse the sub-feature map of the first fused feature map, the sub-feature map of the second fused feature map, and the sub-feature map of the third fused feature map to obtain a fourth fused feature map, and adaptively convolutionally fuse the fourth fused feature map with the sub-feature map of the second fused feature map and the sub-feature map of the third fused feature map to obtain a fifth fused feature map and a sixth fused feature map;

[0028] Feature extraction is performed on the fourth fused feature map, the fifth fused feature map and the sixth fused feature map respectively, and adaptive convolution fusion is performed to obtain the bimodal enhanced feature map.

[0029] Further, performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map includes:

[0030] Using an adaptive candidate region calibration layer to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a feature map;

[0031] The feature map is processed using a maximum pooling layer to generate the calibration feature map.

[0032] Further, predicting the target position of the current frame in the multimodal video based on the calibration feature map includes:

[0033] Processing the calibration feature map using a fully connected layer to obtain a target position of a current frame in the multimodal video;

[0034] The fully connected layer includes three fully connected modules connected in sequence, a neuron random activation function is added after the first two fully connected modules, and the last fully connected module includes a Softmax layer.

[0035] Furthermore, the process of the adaptive convolution fusion includes:

[0036] After the images to be fused are spliced, the tensors corresponding to the images to be fused are generated by convolution kernel, linear rectification function, batch normalization, and Sigmoid activation function in sequence;

[0037] Use the normalized exponential function to obtain the weight tensor corresponding to the image to be fused by applying the normalized exponential function to each corresponding tensor position by position;

[0038] Each weight tensor is multiplied and added with the corresponding image to be fused to obtain a result of adaptive convolution fusion.

[0039] Further, after predicting the target position of the current frame in the multimodal video based on the calibration feature map, the method further includes:

[0040] Determine whether the target tracking of the current frame is successful;

[0041] If yes, perform positive and negative sample sampling on the current frame and add them to the positive and negative sample data set;

[0042] If not, positive and negative samples are extracted from the positive and negative sample data sets to fine-tune the parameters of the tracking model.

[0043] Furthermore, the method further comprises:

[0044] During the target tracking process, the parameters of the tracking model are updated using samples in the positive and negative sample data sets.

[0045] In addition, the present invention also proposes a cross-modal visual tracking device based on adaptive convolution, the device comprising:

[0046] An acquisition module, used for acquiring a visible light image and a thermal infrared image, and fusing the visible light image and the thermal infrared image to obtain an initial feature map;

[0047] An extraction module, used to perform feature extraction on the visible light image, the thermal infrared image and the initial feature map respectively, to obtain a feature map of the visible light image, a feature map of the thermal infrared image and a sub-feature map of the initial feature map;

[0048] an adaptive fusion module, configured to perform adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and to perform adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image to obtain a second fused feature map and a third fused feature map;

[0049] an enhancement module, configured to fuse the first fused feature map, the second fused feature map, and the third fused feature map to obtain a bimodal enhanced feature map;

[0050] A calibration module, used for performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map;

[0051] A prediction module is used to predict the target position of the current frame in the multimodal video based on the calibration feature map.

[0052] The advantages of the present invention are:

[0053] (1) The present invention performs feature extraction and feature fusion on different modal inputs pixel by pixel, and adaptively convolves the result of adaptive convolution with the single modal feature again to complete the information supplementation and enhancement of the single modal feature, thereby realizing information exchange and information supplementation between different modalities, so as to make full use of the complementary advantages of different features in RGBT tracking to further improve the tracking performance.

[0054] (2) The present invention designs a pixel-level weight parameter to weight the importance of different modalities. The weight parameter generated by the convolution structure can be updated for the current input data, and the weight parameter generated by the scalar can be updated for the entire data set to obtain an effective fusion feature map.

[0055] (3) A neuron random activation function is added after the first two fully connected modules in the fully connected layer to reduce the risk of overfitting.

[0056] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of a cross-modal visual tracking method based on adaptive convolution in a first embodiment of the present invention;

[0058] Figure 2 is an overall flow chart of a cross-modal visual tracking method based on adaptive convolution in the first embodiment of the present invention;

[0059] Figure 3 is a network structure diagram of the tracking model in the present invention;

[0060] Figure 4 It is a structural diagram of a cross-modal visual tracking device based on adaptive convolution in the second embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] like Figure 1 As shown, an embodiment of the present invention proposes a cross-modal visual tracking method based on adaptive convolution, which is used to implement target tracking using a tracking model. The method includes the following steps:

[0063] S10, acquiring a visible light image and a thermal infrared image, and fusing the visible light image and the thermal infrared image to obtain an initial feature map;

[0064] S20, performing feature extraction on the visible light image, the thermal infrared image, and the initial feature map respectively to obtain a feature map of the visible light image, a feature map of the thermal infrared image, and a sub-feature map of the initial feature map;

[0065] S30, performing adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and performing adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image, respectively, to obtain a second fused feature map and a third fused feature map;

[0066] S40, fusing the first fused feature map, the second fused feature map, and the third fused feature map to obtain a bimodal enhanced feature map;

[0067] S50, performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map;

[0068] S60: Predicting a target position of a current frame in the multimodal video based on the calibration feature map.

[0069] It should be noted that this embodiment performs feature extraction and feature fusion on different modal inputs pixel by pixel, and adaptively convolves the result of adaptive convolution with the single modal feature again to complete the information supplementation and enhancement of the single modal feature, thereby realizing information exchange and information supplementation between different modalities, so as to make full use of the complementary advantages of different features in RGBT tracking to further improve the tracking performance.

[0070] In one embodiment, the method further comprises the following steps:

[0071] A pair of first frame images are obtained from a pair of registered multimodal videos, and Gaussian distribution sampling is performed with the center points of the two true value frames of the first frame images as the mean to generate candidate samples, wherein the candidate samples include the visible light image and the thermal infrared image for acquisition by the tracking model.

[0072] It should be noted that the first frame truth boxes provided in the multimodal video sequence are manually annotated. + = 500 (IOU ≥ 0.7) positive samples and S -= 5000 (IOU ≤ 0.3) negative samples are used as training samples, and the number of training iterations is set to 30. These 5500 samples are used to initialize the tracking model, and the following fully connected module of size 2 is obtained. At this time, the learning rate of the convolutional layer is fixed, the learning rate of the fully connected module is set to 0.001, and the learning rate of the first two fully connected layers is set to 0.0005. After the initialization, the target position of the previous frame is averaged, and the Gaussian distribution is used for sampling, with (0.09r 2 ,0.09r 2 ,0.25) is the covariance. Generate 256 candidate samples, where: r is the average width and height of the target box in the previous frame.

[0073] In one embodiment, the step S10 includes:

[0074] The visible light image and the thermal infrared image are spliced ​​together and then convolved to obtain the initial feature map.

[0075] It should be noted that the convolution operation consists of a small 1×1 convolution kernel, a linear rectification function, and batch normalization.

[0076] In one embodiment, in step S20, feature extraction is performed on the visible light image, the thermal infrared image and the initial feature map using conventional convolution blocks respectively. The design of the conventional convolution block is based on the first convolution layer of VGG-M, and its convolution kernel size is 7×7, and weights are not shared with each other.

[0077] In one embodiment, in step S30, an adaptive convolution module is used to perform adaptive convolution fusion on the input image, and the process of adaptive convolution fusion includes:

[0078] After the images to be fused are spliced, the tensors corresponding to the images to be fused are generated by convolution kernel, linear rectification function, batch normalization, and Sigmoid activation function in sequence;

[0079] Use the normalized exponential function to obtain the weight tensor corresponding to the image to be fused by applying the normalized exponential function to each corresponding tensor position by position;

[0080] Each weight tensor is multiplied and added with the corresponding image to be fused to obtain a result of adaptive convolution fusion.

[0081] It should be noted that the adaptive convolution module is used to splice the input feature maps, and generate tensors corresponding to the visible light, thermal infrared and previous layer feature fusion results through a 1×1 small convolution kernel, linear rectification function, batch normalization, and Sigmoid activation function. The three tensors are normalized exponentially one by one to obtain the weight tensors corresponding to the three inputs. The weight tensor and the corresponding feature map are multiplied and added to obtain the feature fusion result of this layer.

[0082] In this embodiment, the tensor generation process is also effective for inputs of different sizes in the data set. If the data size or input size in the data set is always consistent, a new tensor of the corresponding size can be directly randomly initialized and the normalized exponential function can be used to obtain the corresponding weight tensor.

[0083] In one embodiment, the step S40 includes the following steps:

[0084] S41, performing feature extraction on the first fused feature map, the second fused feature map, and the third fused feature map, respectively, to obtain a sub-feature map of the first fused feature map, a sub-feature map of the second fused feature map, and a sub-feature map of the third fused feature map;

[0085] S42, performing adaptive convolution fusion on the sub-feature graph of the first fused feature graph, the sub-feature graph of the second fused feature graph, and the sub-feature graph of the third fused feature graph to obtain a fourth fused feature graph, and performing adaptive convolution fusion on the fourth fused feature graph with the sub-feature graph of the second fused feature graph and the sub-feature graph of the third fused feature graph, respectively, to obtain a fifth fused feature graph and a sixth fused feature graph;

[0086] It should be noted that in the network processing process, the imaging characteristics of different modalities are not consistent, which leads to inconsistent features extracted from each other. When extracting features, the convolution corresponding to a single modality is often limited to the most significant features of the current modality, and some information will be lost. Adaptive convolution can be used again to effectively supplement this part, while indirectly integrating the features of another modality into the current modality to achieve information interaction and information enhancement.

[0087] S43, respectively extracting features from the fourth fused feature map, the fifth fused feature map, and the sixth fused feature map and performing adaptive convolution fusion to obtain the bimodal enhanced feature map.

[0088] In one embodiment, the step S50 includes the following steps:

[0089] S51, using an adaptive candidate region calibration layer to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a feature map;

[0090] S52: Process the feature map using a maximum pooling layer to generate the calibration feature map.

[0091] It should be noted that the bimodal enhanced feature map is sent to the adaptive candidate region calibration layer, and bilinear interpolation is used to generate a 512×7×7 feature map. Finally, a maximum pooling layer is used to generate a 512×3×3 feature map. By extracting the convolution features of each RoI (region of interest), the computational complexity is reduced and the features are controlled to a specified size.

[0092] This embodiment directly introduces adaptive convolution after the convolution of each layer to achieve the feature fusion of the corresponding layer. By using adaptive convolution at different layers, the fusion feature utilization of different levels and scales is realized. After obtaining the fusion feature at each layer, adaptive convolution is used again to fuse the fusion feature with the single modal feature to supplement the single modal feature information and realize information interaction between different modalities. Adaptive fusion of features of different modalities is achieved to fully cope with different scenarios and solve the problem of missing modal features.

[0093] In one embodiment, in step S60:

[0094] Processing the calibration feature map using a fully connected layer to obtain a target position of a current frame in the multimodal video;

[0095] The fully connected layer includes three fully connected modules connected in sequence, a neuron random activation function is added after the first two fully connected modules, and the last fully connected module includes a Softmax layer.

[0096] It should be noted that the fully connected layer consists of three fully connected modules of size 512, 512, and 2. The first two fully connected layers are followed by a regularization method of random activation of neurons to reduce the risk of overfitting. The last fully connected module of size 2 contains a Softmax layer that calculates the positive and negative scores of each candidate sample feature: + (x i ) and f-(x i ), and finally the predicted target position is obtained by the following formula:

[0097]

[0098] Among them, x i represents the i-th sample, f + (x i ) represents the obtained positive sample score, f-(x i ) represents the obtained negative sample score, x * is the predicted target location.

[0099] like Figure 2 As shown, after step S60, the method further includes the following steps:

[0100] Determine whether the target tracking of the current frame is successful;

[0101] It should be noted that, in this embodiment, among the scores of the samples obtained, the one with the highest score in the positive sample is used as the predicted target tracking result, and when the highest score in the positive sample is greater than zero, it is determined that the tracking is successful.

[0102] If yes, perform positive and negative sample sampling on the current frame and add them to the positive and negative sample data set;

[0103] If not, positive and negative samples are extracted from the positive and negative sample data sets to fine-tune the parameters of the tracking model.

[0104] Specifically, the current frame is sampled for positive and negative samples, including 50 positive samples (IOU ≥ 0.6) and 200 negative samples (IOU ≤ 0.3), and they are added to the positive and negative sample data sets. If the number of frames in the positive and negative sample data sets exceeds 100, the positive sample areas of the earliest frames are discarded, and if the number of frames exceeds 20, the negative sample areas of the earliest frames are discarded. When the target score is less than zero, that is, when tracking fails, a short-term update will be performed, and 32 positive samples and 96 negative samples will be extracted from the positive and negative sample sets to fine-tune the parameters of the fully connected layer, iterate 10 times, and the learning rate is set to 0.00003.

[0105] It should be noted that whenever a potential tracking failure is detected, such as when the estimated target is classified as background, a short-term update is used to perform a brief correction to the most recent tracking success state.

[0106] It should be noted that the sample data in the positive and negative sample data sets are the fused feature maps output by the last layer of adaptive convolution module. The weight parameters generated by the convolution structure are updated for the current input data, and the weight parameters generated by the scalar can be updated for the entire data set to obtain the most effective fusion result. The fusion result is then adaptively fused with the features of a single modality to achieve feature complementarity between modalities and further improve the tracker performance.

[0107] In one embodiment, the method further comprises:

[0108] During the target tracking process, the parameters of the tracking model are updated using samples in the positive and negative sample data sets.

[0109] It should be noted that during the target tracking process, the parameters of the tracking model are updated for a long time every ten frames, and when the target tracking fails, the model parameters are updated for a short time.

[0110] It should be noted that during the online tracking process, long-term updates are performed every 10 frames. 32 positive samples and 96 negative samples are extracted from the positive and negative sample sets to fine-tune the parameters of the fully connected layer. The iterations are repeated 10 times, and the learning rate is set to 0.00003. If the conditions for short-term updates and long-term updates are not met, the next frame is tracked directly and the model is not updated.

[0111] Long-term update is also to update the network. Compared with short-term update, short-term update uses the most recent 20 correct samples, and long-term update uses 100 correct samples. In this way, the updated state of the model has stronger robustness and can adapt to more complex changes in the target's motion state. Through short-term update and long-term update, the purpose is to enable the network to automatically adapt to the characteristics of the tracking target in real time according to the content of the input data, and finally form a stable tracking result.

[0112] Specifically, to realize the above visual tracking process, a tracking model is adopted as follows Figure 3 As shown, the tracking model includes an initial fusion module, a first feature extraction module, a first adaptive convolution layer, a second feature extraction module, a second adaptive convolution layer, a third feature extraction module, a third adaptive convolution layer, an adaptive candidate region calibration layer and a fully connected module, wherein:

[0113] The initial fusion module includes a feature splicing layer and a convolution layer conv1 connected in sequence. The feature splicing layer is used to splice the visible light image and the thermal infrared image and output the splicing image. The convolution layer conv1 is used to perform a convolution operation on the splicing image to obtain the initial feature map. The convolution operation consists of a 1×1 small convolution kernel, a linear rectification function, and batch normalization.

[0114] The first feature extraction module includes convolution blocks conv2, conv3 and conv4. Convolution blocks conv2, conv3 and conv4 are based on the first convolution layer of VGG-M. The convolution kernel size is 7×7 and they do not share weights with each other. They are used to extract features of visible light images, initial feature maps and thermal infrared images respectively.

[0115] The first adaptive convolution layer includes a first adaptive convolution module, a second adaptive convolution module and a third adaptive convolution module, the output of the convolution block conv2 is connected to the first adaptive convolution module and the second adaptive convolution module respectively, the output of the convolution block conv3 is connected to the first adaptive convolution module, the second adaptive convolution module and the third adaptive convolution module respectively, and the output of the convolution block conv4 is connected to the second adaptive convolution module and the third adaptive convolution module respectively;

[0116] The second feature extraction module includes convolution blocks conv5, conv6 and conv7. Convolution blocks conv5, conv6 and conv7 refer to the second convolution layer of VGG-M. The convolution kernel size is 5×5 and weights are not shared with each other. They are used to be connected to the output of the first adaptive convolution module, the output of the second adaptive convolution module and the output of the third adaptive convolution module respectively.

[0117] The second adaptive convolution layer includes a fourth adaptive convolution module, a fifth adaptive convolution module and a sixth adaptive convolution module, the output of the convolution block conv5 is connected to the fourth adaptive convolution module and the fifth adaptive convolution module, respectively, the output of the convolution block conv6 is connected to the fourth adaptive convolution module, the fifth adaptive convolution module and the sixth adaptive convolution module, respectively, and the output of the convolution block conv7 is connected to the fifth adaptive convolution module and the sixth adaptive convolution module, respectively;

[0118] The third feature extraction module includes convolution blocks conv8, conv9 and conv10, which are respectively connected to the output of the fourth adaptive convolution module, the output of the fifth adaptive convolution module and the output of the sixth adaptive convolution module. Convolution blocks conv8, conv9 and conv10 refer to the third convolution layer of VGG-M, with a convolution kernel size of 3×3, and do not share weights with each other.

[0119] The third adaptive convolution layer includes a seventh adaptive convolution module, the outputs of convolution blocks conv8, conv9 and conv10 are all connected to the input of the seventh adaptive convolution module, and the output of the seventh adaptive convolution module is connected to the fully connected module via the adaptive candidate region calibration layer;

[0120] The adaptive candidate region calibration layer includes connecting the adaptive region of interest calibration layer and the maximum pooling layer in sequence, and using the adaptive region of interest calibration layer to extract the convolution features of each RoI (region of interest), reduce the computational complexity, and control the features within a specified size;

[0121] The fully connected module consists of three fully connected modules of size 512, 512, and 2 respectively. The first two fully connected layers are followed by a regularization method of random activation of neurons to reduce the risk of overfitting. The last fully connected module of size 2 contains a Softmax layer.

[0122] Specifically, the adaptive convolution module includes concatenating the feature maps used for its input, and generating a tensor corresponding to the input feature map through a 1×1 small convolution kernel, a linear rectification function, batch normalization, and a Sigmoid activation function, and applying a normalized exponential function to the tensor position by position to obtain a weight tensor corresponding to the input image. The weight tensor and the corresponding feature map are multiplied and added to obtain the feature fusion result of this layer.

[0123] It should be noted that when generating tensors, this method is also effective for inputs of different sizes in the dataset. If the data size or input size in the dataset is always consistent, you can also directly randomly initialize a new tensor of the corresponding size and use the normalized exponential function to obtain the corresponding weight tensor.

[0124] This embodiment designs a pixel-level weight parameter for the modal features extracted by the convolution layers of different modalities, and weights the importance of different modalities. The weight parameters generated by the convolution structure can be updated for the current input data, and the weight parameters generated by the scalar can be updated for the entire data set, so as to obtain the most effective fusion result, and perform an adaptive fusion operation on the fusion result and the feature of a single modality, so as to achieve feature complementarity between modalities and further improve the tracker performance.

[0125] In this embodiment, the tracking model of this embodiment and the existing tracker are tested on the public data sets GTOT and RGBT234, and the test results are evaluated with other trackers in terms of SR (success rate) and PR (accuracy), as shown in Table 1:

[0126] Table 1

[0127] method GTOT-PR GTOT-SR RGBT234-PR RGBT234-SR Siamese FC 0.655 0.54 Struck+RGBT 0.681 0.533 ADNet 0.718 0.629 DAT 0.771 0.618 DAT+RGBT 0.835 0.675 CFNet+RGBT 0.551 0.39 SiamDW+RGBT 0.68 0.565 0.604 0.397 MEEM+RGBT 0.621 0.389 SRDCF 0.641 0.463 CSR-DCF+RGBT 0.695 0.49 SOWP+RGBT 0.696 0.451 ECO 0.77 0.631 0.702 0.514 RT-MDNet 0.745 0.613 0.714 0.5 MDNet 0.719 0.502 SGT 0.851 0.628 0.72 0.472 MDNet+RGBT 0.8 0.637 0.722 0.495 RT-MDNet+RGBT 0.829 0.667 0.734 0.483 DAPNet 0.882 0.707 0.766 0.537 MACNet 0.888 0.717 0.783 0.537

[0128] Among them, the first column in the table is the PR accuracy on the GTOT dataset, the second column is the SR accuracy on the GTOT dataset, the third column is the PR accuracy on the RGBT234 dataset, and the fourth column is the SR accuracy on the RGBT234 dataset. Among them, PR (Precision plot): the percentage of video frames whose distance between the center point of the target position (bounding box) and the center point of the manually annotated (ground-truth) target is less than a given threshold. SR (Success Plot): First, the overlap score (OS) is defined. The bounding box obtained by the tracking algorithm (denoted as a) and the box given by the ground-truth (denoted as b) have an overlap rate defined as: OS = |a∩b| / |a∪b|, |·| represents the number of pixels in the area. When the OS of a frame is greater than the set threshold, the frame is considered successful (Success), and the percentage of total successful frames to all frames is the success rate (Success rate). Among them, MACNet represents the tracking result accuracy of this embodiment. It can be clearly seen that compared with other methods, its tracking performance has achieved good results.

[0129] In one embodiment, the method further includes: training the tracking model, the specific steps are as follows:

[0130] (1) The convolutional block parameters of the feature extraction module are initialized using the first three layers of the VGG-M pre-trained model. The adaptive convolutional module and the fully connected module are randomly initialized. The convolutional layer of the feature extraction module consists of three convolutional blocks and Relu activation functions. The first two layers have a local response function LRN and a maximum pooling layer function MaxPool, and the convolution kernel sizes are 7×7×96, 5×5×256, and 3×3×512, respectively.

[0131] (2) Use the manually annotated visible light thermal infrared dataset to train the entire network. Select 250 samples near the true value box, including S + =50 (IOU ≥ 0.7) positive samples and S - = 200 (IOU ≤ 0.5) negative samples, where IOU represents the intersection-over-union ratio between the true value box and the sample.

[0132] (3) For the obtained positive and negative samples, the stochastic gradient method is used for 100 iterations during training. In each iteration, each video sequence corresponds to a new fc6 fully connected layer for training, and the remaining layers of the video sequence are shared. During the Kth iteration, the minibatch (batchsize = 128, containing 32 positive samples and 96 negative samples) is randomly selected from the (kmod K)th video sequence. Eight frames of paired visible light and thermal infrared images are generated by Gaussian distribution sampling, and the (kmod K)th fc6 layer of the corresponding video sequence is replaced. The learning rate of the convolutional layer is set to 0.0001, and the learning rate of the fully connected layer is set to 0.0002. Save the final model for the online tracking stage.

[0133] In addition, if Figure 4 As shown, the embodiment of the present invention also proposes a cross-modal visual tracking device based on adaptive convolution, and the device includes:

[0134] An acquisition module 10 is used to acquire a visible light image and a thermal infrared image, and fuse the visible light image and the thermal infrared image to obtain an initial feature map;

[0135] An extraction module 20, configured to perform feature extraction on the visible light image, the thermal infrared image and the initial feature map, respectively, to obtain a feature map of the visible light image, a feature map of the thermal infrared image and a sub-feature map of the initial feature map;

[0136] The adaptive fusion module 30 is used to perform adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and perform adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image to obtain a second fused feature map and a third fused feature map;

[0137] An enhancement module 40, configured to fuse the first fused feature map, the second fused feature map and the third fused feature map to obtain a bimodal enhanced feature map;

[0138] A calibration module 50, configured to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map;

[0139] The prediction module 60 is used to predict the target position of the current frame in the multimodal video based on the calibration feature map.

[0140] In one embodiment, the device further comprises:

[0141] An image generation module is used to obtain a pair of first frame images in a pair of registered multimodal videos, and perform Gaussian distribution sampling with the center points of the two true value frames of the first frame images as the mean to generate candidate samples, wherein the candidate samples include the visible light image and the thermal infrared image for acquisition by the tracking model.

[0142] In one embodiment, the acquisition module 10 is specifically used for:

[0143] The visible light image and the thermal infrared image are spliced ​​together and then convolved to obtain the initial feature map.

[0144] In one embodiment, the enhancement module 40 includes:

[0145] a feature extraction unit, configured to perform feature extraction on the first fused feature map, the second fused feature map, and the third fused feature map, respectively, to obtain a sub-feature map of the first fused feature map, a sub-feature map of the second fused feature map, and a sub-feature map of the third fused feature map;

[0146] An adaptive convolution fusion unit, configured to perform adaptive convolution fusion on the sub-feature map of the first fused feature map, the sub-feature map of the second fused feature map, and the sub-feature map of the third fused feature map to obtain a fourth fused feature map, and to perform adaptive convolution fusion on the fourth fused feature map with the sub-feature map of the second fused feature map and the sub-feature map of the third fused feature map, respectively, to obtain a fifth fused feature map and a sixth fused feature map;

[0147] The enhancement unit is used to extract features from the fourth fused feature map, the fifth fused feature map and the sixth fused feature map respectively and then perform adaptive convolution fusion to obtain the bimodal enhanced feature map.

[0148] In one embodiment, the calibration module 50 is specifically configured to:

[0149] Using an adaptive candidate region calibration layer to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a feature map;

[0150] The feature map is processed using a maximum pooling layer to generate the calibration feature map.

[0151] In one embodiment, the prediction module 60 is specifically configured to:

[0152] The calibration feature map is processed using a fully connected layer to obtain a target position of a current frame in the multimodal video.

[0153] In one embodiment, the device further comprises:

[0154] A judgment module, used to judge whether the target tracking of the current frame is successful;

[0155] A sampling module, used for sampling positive and negative samples of the current frame and adding them to the positive and negative sample data sets when the output result of the judgment module is yes;

[0156] The first updating module is used to extract positive and negative samples from the positive and negative sample data sets and fine-tune the parameters of the tracking model when the output result of the judgment module is negative.

[0157] In one embodiment, the device further comprises:

[0158] The second updating module is used to update the parameters of the tracking model using samples in the positive and negative sample data sets during the target tracking process.

[0159] It should be noted that the cross-modal visual tracking device based on adaptive convolution described in this embodiment corresponds to the step flow of the cross-modal visual tracking method based on adaptive convolution disclosed in the above embodiment, which will not be repeated here.

[0160] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0161] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0163] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0164] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A cross-modal visual tracking method based on adaptive convolution, characterized in that: For realizing target tracking by using a tracking model, the method comprises: Acquire a visible light image and a thermal infrared image, and fuse the visible light image and the thermal infrared image to obtain an initial feature map, wherein the visible light image and the thermal infrared image are generated by a pair of registered multimodal videos; Performing feature extraction on the visible light image, the thermal infrared image, and the initial feature map respectively to obtain a feature map of the visible light image, a feature map of the thermal infrared image, and a sub-feature map of the initial feature map; Performing adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and performing adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image, respectively, to obtain a second fused feature map and a third fused feature map; The first fused feature map, the second fused feature map and the third fused feature map are fused to obtain a dual-modal enhanced feature map, including extracting features from the first fused feature map, the second fused feature map and the third fused feature map to obtain a sub-feature map of the first fused feature map, a sub-feature map of the second fused feature map and a sub-feature map of the third fused feature map; performing adaptive convolution fusion on the sub-feature map of the first fused feature map, the sub-feature map of the second fused feature map and the sub-feature map of the third fused feature map to obtain a fourth fused feature map, and performing adaptive convolution fusion on the fourth fused feature map with the sub-feature map of the second fused feature map and the sub-feature map of the third fused feature map to obtain a fifth fused feature map and a sixth fused feature map; extracting features from the fourth fused feature map, the fifth fused feature map and the sixth fused feature map and performing adaptive convolution fusion on them to obtain the dual-modal enhanced feature map; Performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map; Based on the calibrated feature map, an object position of a current frame in the multimodal video is predicted.

2. The cross-modal visual tracking method based on adaptive convolution as claimed in claim 1, characterized in that: The method further comprises: A pair of first frame images are obtained from a pair of registered multimodal videos, and Gaussian distribution sampling is performed with the center points of the two true value frames of the first frame images as the mean to generate candidate samples, wherein the candidate samples include the visible light image and the thermal infrared image for acquisition by the tracking model.

3. The cross-modal visual tracking method based on adaptive convolution as claimed in claim 1, characterized in that: The step of fusing the visible light image and the thermal infrared image to obtain an initial feature map includes: The visible light image and the thermal infrared image are spliced ​​together and then convolved to obtain the initial feature map.

4. The cross-modal visual tracking method based on adaptive convolution as claimed in claim 1, characterized in that: The performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map includes: Using an adaptive candidate region calibration layer to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a feature map; The feature map is processed using a maximum pooling layer to generate the calibration feature map.

5. The cross-modal visual tracking method based on adaptive convolution as claimed in claim 1, characterized in that: The predicting, based on the calibration feature map, a target position of a current frame in the multimodal video includes: Processing the calibration feature map using a fully connected layer to obtain a target position of a current frame in the multimodal video; The fully connected layer includes three fully connected modules connected in sequence, a neuron random activation function is added after the first two fully connected modules, and the last fully connected module includes a Softmax layer.

6. The cross-modal visual tracking method based on adaptive convolution according to any one of claims 1 to 5, characterized in that: The process of adaptive convolution fusion includes: After the images to be fused are spliced, the tensors corresponding to the images to be fused are generated by convolution kernel, linear rectification function, batch normalization, and Sigmoid activation function in sequence; Use the normalized exponential function to obtain the weight tensor corresponding to the image to be fused by applying the normalized exponential function to each corresponding tensor position by position; Each weight tensor is multiplied and added with the corresponding image to be fused to obtain a result of adaptive convolution fusion.

7. The cross-modal visual tracking method based on adaptive convolution according to any one of claims 1 to 5, characterized in that: After predicting the target position of the current frame in the multimodal video based on the calibration feature map, the method further includes: Determine whether the target tracking of the current frame is successful; If yes, perform positive and negative sample sampling on the current frame and add them to the positive and negative sample data set; If not, positive and negative samples are extracted from the positive and negative sample data sets to fine-tune the parameters of the tracking model.

8. The cross-modal visual tracking method based on adaptive convolution according to any one of claims 1 to 5, characterized in that: The method further comprises: During the target tracking process, the parameters of the tracking model are updated using samples in the positive and negative sample data sets.

9. A cross-modal visual tracking device based on adaptive convolution, characterized in that: The device comprises: an acquisition module, used for acquiring a visible light image and a thermal infrared image, and fusing the visible light image and the thermal infrared image to obtain an initial feature map, wherein the visible light image and the thermal infrared image are generated by a pair of registered multimodal videos; An extraction module, used to perform feature extraction on the visible light image, the thermal infrared image and the initial feature map respectively, to obtain a feature map of the visible light image, a feature map of the thermal infrared image and a sub-feature map of the initial feature map; an adaptive fusion module, configured to perform adaptive convolution fusion on the feature map of the visible light image, the feature map of the thermal infrared image, and the sub-feature map of the initial feature map to obtain a first fused feature map, and to perform adaptive convolution fusion on the first fused feature map with the feature map of the visible light image and the feature map of the thermal infrared image to obtain a second fused feature map and a third fused feature map; an enhancement module, configured to fuse the first fused feature map, the second fused feature map, and the third fused feature map to obtain a bimodal enhanced feature map; A calibration module, used for performing bilinear interpolation calculation on the bimodal enhanced feature map to generate a calibration feature map; A prediction module, configured to predict a target position of a current frame in the multimodal video based on the calibration feature map; The enhancement module comprises: a feature extraction unit, configured to perform feature extraction on the first fused feature map, the second fused feature map, and the third fused feature map, respectively, to obtain a sub-feature map of the first fused feature map, a sub-feature map of the second fused feature map, and a sub-feature map of the third fused feature map; An adaptive convolution fusion unit, configured to perform adaptive convolution fusion on the sub-feature map of the first fused feature map, the sub-feature map of the second fused feature map, and the sub-feature map of the third fused feature map to obtain a fourth fused feature map, and to perform adaptive convolution fusion on the fourth fused feature map with the sub-feature map of the second fused feature map and the sub-feature map of the third fused feature map, respectively, to obtain a fifth fused feature map and a sixth fused feature map; The enhancement unit is used to extract features from the fourth fused feature map, the fifth fused feature map and the sixth fused feature map respectively and then perform adaptive convolution fusion to obtain the bimodal enhanced feature map.

10. The cross-modal visual tracking device based on adaptive convolution according to claim 9, characterized in that: The calibration module is specifically used for: Using an adaptive candidate region calibration layer to perform bilinear interpolation calculation on the bimodal enhanced feature map to generate a feature map; The feature map is processed using a maximum pooling layer to generate the calibration feature map.

Images