A visible light thermal infrared vision tracking method for weakly registered data

Abstract

A visible light and thermal infrared visual tracking method for weakly registered data belongs to the field of computer vision technology and solves the problem of multimodal trackers' dependence on registration data. The method of the present invention introduces a progressive feature registration module to realize modal feature-level registration. The registered multimodal features are further fused through a transformer fusion module. The fused features are input into a model predictor to obtain the optimal prediction model of the current frame. A convolution operation is performed with the full-image features to obtain a score weight map. Positions with high response values ​​represent the predicted positions of the targets. At the same time, a modulation vector module is used to select bounding boxes to complete the accurate positioning of the targets in the current frame. The method of the present invention can mine the complementary information of visible light and thermal infrared modalities without relying on strictly registered multimodal data, effectively realize the modal complementarity between visible light and thermal infrared, and further improve the accuracy of multimodal target tracking.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a visible light thermal infrared vision tracking method for weak registration data. Background Art

[0002] Object tracking is a fundamental task in computer vision. Its goal is to predict the size and position of an object in subsequent frames, given its size and position in the initial frame of a video sequence. In recent years, object tracking technology has achieved significant breakthroughs and progress, and has been widely applied in fields such as intelligent transportation, autonomous driving, and medical imaging. However, the reliability of visible light data is significantly reduced under harsh conditions such as extreme lighting, rain, snow, and fog, which in turn affects tracker performance. To compensate for the shortcomings of visible light data, some researchers have recently conducted research on multimodal tracking by introducing thermal infrared modalities to achieve robust visual tracking performance under all-day and all-weather conditions. Training advanced multimodal trackers requires the preparation of registered multimodal data. Existing multimodal trackers rely on registered multimodal data for training and testing. However, since visible light and thermal infrared data are captured by different sensors, obtaining registered multimodal data requires a time-consuming and labor-intensive manual registration process. This difficulty in obtaining registered multimodal data has severely limited the development of multimodal tracking. Furthermore, existing multimodal trackers fail to exploit the complementary advantages of weakly registered data. Summary of the Invention

[0003] The purpose of the present invention is to design a visible light thermal infrared vision tracking method for weak registration data to solve the problem of multimodal tracker's dependence on registration data.

[0004] The present invention solves the above technical problems through the following technical solutions:

[0005] A visible light and thermal infrared visual tracking method for weakly registered data adopts a network model comprising: a feature extraction network, a progressive feature fusion registration module, a transformer fusion module, and a model predictor; the feature extraction network is used to extract multimodal features of visible light and thermal infrared; the progressive feature fusion registration module uses weakly registered image pairs to predict a homography matrix to distort thermal infrared modal features to achieve registration of thermal infrared modal features to visible light modal features; the transformer fusion module is composed of a multi-head attention mechanism to fuse the registered multimodal features; the model predictor is composed of a model prediction module and a modulation vector module to use the fused multimodal features to predict the final tracking result;

[0006] The tracking method comprises the following steps:

[0007] Step 1: Generate a search area around the target in the initial frame or the previous frame, and extract the multimodal features of the search area;

[0008] Step 2: The multimodal features extracted by the feature extraction network are coarsely registered using the progressive feature registration module, and the homography matrix is ​​predicted at the same time, thereby achieving fine registration of thermal infrared features to visible light features and updating the homography matrix at the same time;

[0009] Step 3: The finely registered visible light, thermal infrared modal features are fused through the transformer feature fusion module for multimodal features;

[0010] Step 4: Use the fused features to input into the model predictor to obtain the optimal prediction model for the current frame. Perform a convolution operation with the full image features to obtain a score weight map. The position with a high response value represents the predicted position of the target. At the same time, the bounding box is selected through the modulation vector module to complete the accurate positioning of the target in the current frame.

[0011] Step 5: Track the next frame.

[0012] Furthermore, the method described in step 2 for performing rough feature registration using the progressive feature registration module and predicting the homography matrix to achieve fine registration of thermal infrared features to visible light features is as follows: First, according to the main homography matrix H M The spatial transformer network STN is used to achieve coarse registration of thermal infrared modal features to visible light modal features, and the visible light features and the coarsely registered thermal infrared features are spliced ​​in the channel dimension and sent to the homography estimation module to estimate the secondary homography matrix H. R The homography estimation module is composed of a residual module, a global average pooling, a fully connected layer and a direct linear transformation. The spatial transformer network STN is used to achieve fine registration of the thermal infrared modality and the predicted secondary homography matrix H is R For supervision, the loss function is as follows:

[0013]

[0014] Among them, L M is the homography loss function, H T represents the supervisory signal matrix, n represents the dimension of the homography matrix, is the element in the i-th row and j-th column of the secondary homography matrix, is the element in the i-th row and j-th column of the supervisory semaphore matrix.

[0015] Furthermore, the method for updating the homography matrix in step 2 is as follows: according to the predicted secondary homography matrix H R Update the main homography matrix H M The value of is as follows:

[0016]

[0017] in, is the updated main homography matrix.

[0018] Furthermore, the method of performing multimodal feature fusion of the finely registered visible light thermal infrared modal features through the transformer feature fusion module in step 3 is as follows: the registered visible light modal features F RGB and thermal infrared mode F T As the input of the transformer fusion module, F RGB Combined with the position code as the query set, with F RGB Connection F T Combined with the position code as the key set, F RCB Connection F T As the value, a multi-head self-attention operation is performed, and the residual is added, and then the thermal infrared and visible light fusion modal features are obtained through a feedforward neural network and normalization operation.

[0019] Furthermore, the calculation method of the optimal prediction model described in step 4 is as follows:

[0020] The features S of the training set train The features are input into the initialized model prediction module to obtain the prediction model f, and the prediction model f is used to predict the features S of the test set. test The loss is calculated based on the formula as follows:

[0021]

[0022] Among them, s represents the sample M of the test set test The predicted score value, z represents the sample label value, T represents the foreground and background, and l(s, z) is the loss calculation function;

[0023] Constrain the loss of each iteration and use the Hing-like loss function for optimization. The formula is as follows:

[0024]

[0025] Among them, L cls is the classification loss of the target position, N iter represents the number of iterations, z c represents the sample label value as the Gaussian function of the target c, x is the input search area feature, f i is the prediction model obtained in the i-th iteration.

[0026] Furthermore, the method for selecting the bounding box by the modulation vector module in step 4 is as follows:

[0027] The model is trained using the mean square error between the bounding box predicted by the modulation vector module and the true bounding box. The formula is as follows:

[0028]

[0029] Among them, L bb is the target bounding box regression loss, IOU P IOU is the intersection-over-union ratio between the predicted bounding box and the true value. T is the intersection-over-union ratio of the true bounding box and the true value;

[0030] The final optimized loss function L tot It is the weighted sum of homography loss, target location classification loss and target bounding box regression loss, and the formula is as follows:

[0031] L tot =αL M +βL cls +L bb

[0032] Among them, α and β are hyperparameters.

[0033] The advantages of the present invention are:

[0034] In view of the fact that multimodal data has natural offset and distortion and that registration requires a lot of manpower, the method of the present invention introduces a progressive feature registration module to realize modal feature-level registration. First, the homography matrix is ​​calculated according to the target position of the first frame or the previous frame to realize the coarse registration of thermal infrared image features to visible light image features. Then, the visible light features and the coarsely registered thermal infrared features are spliced ​​according to the channel dimension and sent to the homography estimation subnetwork to predict the secondary homography transformation matrix of the two modal features. Then, the coarsely registered thermal infrared features are finely registered according to the predicted homography matrix, and the main homography matrix is ​​updated according to the predicted secondary homography matrix, thereby finally realizing the finely registered multimodal features. The registered multimodal features are further fused through the transformer fusion module, and the visible light combined with the position encoding is used as the query set. RGB Connection F T Combined with the position code as the key set, F RGB Connection F TAs Value, multi-head self-attention operation is performed, and the thermal infrared and visible light fusion modal features are obtained through residual addition, feedforward neural network and normalization operation; the method of the present invention can mine the complementary information of visible light and thermal infrared modalities without relying on strictly aligned multimodal data, effectively realize the modal complementarity between visible light and thermal infrared, and further improve the accuracy of multimodal target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a visible light thermal infrared visual tracking method for weak registration data according to a first embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the network model structure of the visible light thermal infrared visual tracking method for weak registration data according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] The technical solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0039] Example 1

[0040] like Figure 1 As shown in FIG. 1 , a visible light thermal infrared visual tracking method for weak registration data according to an embodiment of the present invention includes the following steps:

[0041] Step 1: Generate a search area around the target in the initial frame or the previous frame, and extract the multimodal features of the search area.

[0042] Step 2: The multimodal features extracted by the feature extraction network are coarsely registered using the progressive feature registration module, and the homography matrix is ​​predicted at the same time, thereby achieving fine registration of thermal infrared features to visible light features and updating the homography matrix at the same time.

[0043] Step 3: The finely aligned visible light, thermal infrared modal features are fused through the transformer feature fusion module for multimodal feature fusion.

[0044] Step 4: Use the fused features to input into the model predictor to obtain the optimal prediction model for the current frame, and perform a convolution operation with the full image features to obtain a score weight map. The position with a high response value represents the predicted position of the target. At the same time, the bounding box is selected through the modulation vector module to complete the accurate positioning of the target in the current frame.

[0045] Step 5: Track the next frame.

[0046] like Figure 2 As shown, this is a schematic diagram of the network model structure of the visible light thermal infrared visual tracking method for weakly registered data in embodiment 1 of the present invention. The network model includes four parts: the first part is a feature extraction network, which is used to extract multimodal features of visible light and thermal infrared; the second part is a progressive feature fusion registration module, which uses weakly registered images to predict the homography matrix, which is used to distort the thermal infrared modal features to achieve registration of the thermal infrared modal features to the visible light modal features; the third part is a transformer fusion module, which is composed of a multi-head attention mechanism and is used to fuse the multimodal features after registration; the fourth part is a model predictor, which is composed of a model prediction module and a modulation vector module and is used to use the fused multimodal features to predict the final tracking results.

[0047] The specific implementation is divided into training process and testing process, which include the following:

[0048] 1. Training process

[0049] (1) The LasHeR non-registration dataset is selected as the training set. First, multiple multimodal image pairs are randomly selected from the same sequence as samples (M train ,,M test ), where M train is the sample of the training set, M test is a sample of the test set; then Resnet50 is used as the feature extraction network to extract features (S train , S test ), where the feature S of the training set train and the test set features S test By the visible light modal characteristics S RGB and thermal infrared modal characteristics S T Then, the homography matrix of each sample is calculated based on the true value of the visible light and thermal infrared modes, and the homography matrix is ​​split into the main homography matrix H M and the subhomography matrix H R .

[0050] (2) Use the progressive feature registration module for registration: First, according to the main homography matrix H MThe spatial transformer network STN is used to achieve coarse registration of thermal infrared modal features to visible light modal features, and the visible light features and the coarsely registered thermal infrared features are spliced ​​in the channel dimension and sent to the homography estimation module (composed of the residual module, global average pooling G, fully connected layer and direct linear transformation DLT) to estimate the secondary homography matrix H R , and then use the spatial transformer network STN to achieve fine registration of thermal infrared modalities, and the predicted subhomography matrix H R For supervision, the loss function is as follows:

[0051]

[0052] Among them, L M is the homography loss function, H T represents the supervisory signal matrix, n represents the dimension of the homography matrix, is the element in the i-th row and j-th column of the secondary homography matrix, is the element in the j-th row and j-th column of the supervisory semaphore matrix.

[0053] In addition, according to the predicted secondary homography matrix H R Update the main homography matrix H M The value of is as follows:

[0054]

[0055] in, is the updated main homography matrix.

[0056] (3) The registered visible light modal feature F RGB and thermal infrared mode F T As the input of the transformer fusion module, F RGB Combined with the position code as the query set, with F RGB Connection F T Combined with the position code as the key set, F RGB Connection F T As the Value, multi-head self-attention operation is performed, and the thermal infrared and visible light fusion modal features are obtained through residual addition, feedforward neural network and normalization operation. The feature S of the training set train and the test set features S test The features are finely aligned and multimodally fused in the same way as above.

[0057] (4) The feature S of the training set train The features are input to the initialization model prediction module to obtain the prediction model f, and the prediction model f is used to calculate the features S of the test set. test The test is performed on the loss, and the formula is as follows:

[0058]

[0059] Among them, s represents the sample M of the test set test The predicted score value, z represents the sample label value, T represents the foreground and background, and l(s, z) is the loss calculation function.

[0060] In order to improve the robustness of the model, the loss of each iteration is constrained, so the Hing-like loss function is used for further optimization. The formula is as follows:

[0061]

[0062] Among them, L cls is the classification loss of the target position, N iter represents the number of iterations, z c represents the sample label value as the Gaussian function of the target c, x is the input search area feature, f i is the prediction model obtained in the i-th iteration.

[0063] (5) For the selection of the bounding box, the model training is performed by calculating the mean square error between the bounding box predicted by the modulation vector module and the true bounding box. The formula is as follows:

[0064]

[0065] Among them, L bb is the target bounding box regression loss, IOU P IOU is the intersection-over-union ratio between the predicted bounding box and the true value. T is the intersection-over-union ratio of the true bounding box and the true value.

[0066] The final optimized loss function L tot It is the weighted sum of homography loss, classification loss and bounding box regression loss, and the formula is as follows:

[0067] L tot =αL M +βL cls +L bb

[0068] Among them, α and β are hyperparameters.

[0069] 2. Testing process

[0070] During the test, most of the model parameters that have been trained in the training phase are fixed, and only the parameters are initialized in the final model predictor part. In the test phase, only one pair of weakly registered thermal infrared and visible light modality videos are input each time.

[0071] (1) Using the initial frame, we first obtain 15 pairs of initial samples S in thermal infrared and visible light modes through the same data enhancement method. train , the multimodal features are registered through the progressive feature registration module, and then the multimodal features are fused through the transformer fusion module. The fused features are used to initialize the prediction model f = D (S train ).

[0072] (2) Obtain the initialization prediction model, perform convolution operation on the background features and the initialization model to locate the target in the subsequent frame search area, and use the modulation vector module to select the target bounding box to obtain the accurate positioning of the target object in each frame.

[0073] (3) In addition, during the tracking process, a sample pool of size 50 is set up to keep the latest samples. In addition, the optimizer recursive model predictor is executed every 20 frames or when an interference target is detected.

[0074] 3. Analysis of experimental results

[0075] Tables 1 and 2 present experimental results from this paper, representing test results on the publicly available datasets RGBT234 and LasHeR weak registration test set, respectively. These results are compared with other trackers in terms of PR (Precision) and SR (Success Rate). It can be clearly seen that compared to the baseline and other methods, the tracking performance of this paper has improved to a certain extent.

[0076] Table 1

[0077] Trackers MaCNet MANet ToMP50-baseline The present invention PR (accuracy) 76.4 77.7 81.6 85.3 SR (success rate) 53.2 53.9 60.5 62.9

[0078] Table 2

[0079] Trackers MaCNet MANet ToMP50-baseline The present invention PR (accuracy) 38.4 32.9 44.0 54.8 SR (success rate) 27.0 24.1 33.6 42.6

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A visible light thermal infrared visual tracking method for weakly registered data, characterized in that: The network model used includes: a feature extraction network, a progressive feature fusion registration module, a transformer fusion module and a model predictor; the feature extraction network is used to extract multimodal features of visible light and thermal infrared; the progressive feature fusion registration module uses weakly registered images to predict the homography matrix, which is used to distort the thermal infrared modal features to achieve registration of the thermal infrared modal features to the visible light modal features; the transformer fusion module is composed of a multi-head attention mechanism and is used to fuse the multimodal features after registration; the model predictor is composed of a model prediction module and a modulation vector module and is used to use the fused multimodal features to predict the final tracking results; The tracking method comprises the following steps: Step 1: Generate a search area around the target in the initial frame or the previous frame, and extract the multimodal features of the search area; Step 2: The multimodal features extracted by the feature extraction network are coarsely registered using the progressive feature registration module, and the homography matrix is ​​predicted at the same time, thereby achieving fine registration of thermal infrared features to visible light features and updating the homography matrix at the same time; The method of using the progressive feature registration module to perform rough feature registration and predict the homography matrix to achieve fine registration of thermal infrared features to visible light features is as follows: First, according to the main homography matrix H M The spatial transformer network STN is used to achieve coarse registration of thermal infrared modal features to visible light modal features, and the visible light features and the coarsely registered thermal infrared features are spliced ​​in the channel dimension and sent to the homography estimation module to estimate the secondary homography matrix H. R The homography estimation module is composed of a residual module, a global average pooling, a fully connected layer and a direct linear transformation. The spatial transformer network STN is used to achieve fine registration of the thermal infrared modality and the predicted secondary homography matrix H is R For supervision, the loss function is as follows: in, is the homography loss function, H T represents the supervisory signal matrix, n represents the dimension of the homography matrix, is the element in the i-th row and j-th column of the secondary homography matrix, is the element in the i-th row and j-th column of the supervisory semaphore matrix; Step 3: The finely registered visible light, thermal infrared modal features are fused through the transformer feature fusion module for multimodal features; Step 4: Use the fused features to input into the model predictor to obtain the optimal prediction model for the current frame. Perform a convolution operation with the full image features to obtain a score weight map. The position with a high response value represents the predicted position of the target. At the same time, the bounding box is selected through the modulation vector module to complete the accurate positioning of the target in the current frame. Step 5: Track the next frame.

2. The visible light thermal infrared visual tracking method for weak registration data according to claim 1, characterized in that: The method for updating the homography matrix described in step 2 is as follows: According to the predicted secondary homography matrix H R Update the main homography matrix H M The value of is as follows: in, is the updated main homography matrix.

3. The visible light thermal infrared visual tracking method for weak registration data according to claim 2, characterized in that: The method of multimodal feature fusion of the finely registered visible light thermal infrared modal features through the transformer feature fusion module in step 3 is as follows: the registered visible light modal features F RGB and thermal infrared mode F T As the input of the transformer fusion module, F RGB Combined with the position code as the query set, with F RGB Connection F T Combined with the position code as the key set, F RGB Connection F T As the value, a multi-head self-attention operation is performed, and the residual is added, and then the thermal infrared and visible light fusion modal features are obtained through a feedforward neural network and normalization operation.

4. The visible light thermal infrared visual tracking method for weak registration data according to claim 3, characterized in that: The calculation method of the optimal prediction model described in step 4 is as follows: The features S of the training set train The features are input into the initialized model prediction module to obtain the prediction model f, and the prediction model f is used to predict the features S of the test set. test The loss is calculated based on the formula as follows: Among them, s represents the sample M of the test set test The predicted score value, z represents the sample label value, T represents the foreground and background, Calculate the loss function; Constrain the loss of each iteration and use the Hing-like loss function for optimization. The formula is as follows: in, is the classification loss of the target location, represents the number of iterations, Represents the sample label value as the Gaussian function of the target c, is the input search area feature, is the prediction model obtained in the i-th iteration.

5. The visible light thermal infrared visual tracking method for weak registration data according to claim 4, characterized in that: The method for selecting the bounding box by the modulation vector module described in step 4 is as follows: The model is trained using the mean square error between the bounding box predicted by the modulation vector module and the true bounding box. The formula is as follows: in, is the target bounding box regression loss, is the intersection-over-union ratio of the predicted bounding box and the true value, is the intersection-over-union ratio of the true bounding box and the true value; The final optimized loss function It is the weighted sum of homography loss, target location classification loss and target bounding box regression loss, and the formula is as follows: in, and are all hyperparameters.

Images