Robust visual tracking method

Abstract

The application provides a robust visual tracking method, comprising the following steps: preprocessing existing video data; training a pre-trained convolutional neural network with multiple video sequences; randomly initializing a full connection layer; projecting input features; constructing a graph model G, which represents the result re-projected into the original representation space; establishing a global inference model; adding the first frame to the short-term tracking and long-term tracking sets; from the second frame, drawing positive samples and negative samples as training samples; using meta-learning to perform online learning on the parameter matrix P; fine-tuning the global inference model with short-term updated output features; obtaining three kinds of samples from the last layer of the convolutional neural network; training a triplet loss function to adjust the multi-modal similarity and minimize the distance function. Compared with the prior art, the application can improve the performance of computer vision tracking by strengthening the nested global inference model and using part of the multi-domain target information for scene perception update.

Description

Technical Field

[0001] This invention relates to a robust visual tracking method, belonging to the field of computer vision technology. Background Technology

[0002] RGBT tracking (visual tracking) performs far from satisfactory in challenging scenarios such as occlusion or background clutter. For example, in cases of severe occlusion, the target's appearance itself cannot provide sufficiently useful target information. Therefore, exploring the relationship between the target and its surrounding background in dynamic scenes becomes a promising solution.

[0003] In view of this, it is indeed necessary to propose a robust visual tracking method to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a robust visual tracking method that can utilize partial multi-domain target information for scene perception updates.

[0005] To achieve the above objectives, the present invention provides a robust visual tracking method, which mainly includes the following steps:

[0006] Step 1: Preprocess the existing video data by cropping each frame of the image with the target as the center;

[0007] Step 2: Train a pre-trained convolutional neural network using multiple video sequences to initialize the target bounding boxes;

[0008] Step 3: Randomly initialize fully connected layers;

[0009] Step 4: Collect 500 positive samples and 5000 negative samples from the first frame of the video data as training samples;

[0010] Step 5: Project input features X∈R H ×W×C is input into a low-dimensional subspace to generate low-dimensional features.

[0011] Step 6: Construct a graph model G, which represents the result being reprojected into the original representation space;

[0012] Step 7: Establish a global inference model and learn the parametric graph model G of the global inference model;

[0013] Step 8: Add the first frame to the set of short-term and long-term tracking;

[0014] Step 9: Starting from the second frame, draw positive and negative samples as training samples for short-term and long-term updates of the global inference model.

[0015] Step 10: Use meta-learning to learn the parameter matrix P online and output the features. Make fine adjustments;

[0016] Step 11: Fine-tune the global inference model by updating the output features in the short term;

[0017] Step 12: Obtain three types of samples from the last layer of the convolutional neural network;

[0018] Step 13: Train the triplet loss function to adjust the multimodal similarity and minimize the distance function.

[0019] As a further improvement of the present invention, in step 1, the resolution of the size of each frame image cropped is 107x107x3.

[0020] As a further improvement of the present invention, in step 2, the convolutional neural network includes conv1, conv2 and conv3, and each of conv1, conv2 and conv3 includes 5 convolutional layers and 3 fully connected layers.

[0021] As a further improvement of the present invention, in step 4, the region overlap rate of the positive sample is greater than 0.7, and the region overlap rate of the negative sample is less than 0.5.

[0022] As a further improvement of the present invention, in step 7, the graphical model G is...

[0023]

[0024] In this model, the graph model G is composed of nested graph convolutional networks G1 and G2, A1 is the adjacency matrix of graph convolutional network G1, and A2 is the adjacency matrix of graph convolutional network G2.

[0025] As a further improvement of the present invention, the graph convolutional network G1 is

[0026]

[0027] Where W1 is the weight matrix, used to represent the parameters of graph convolution.

[0028] As a further improvement of the present invention, the graphical model G is represented as follows:

[0029]

[0030] W2 is a weight matrix used to represent the relationships between semantic regions.

[0031] As a further improvement of the present invention, in step 9, the region overlap rate of the positive sample is greater than 0.7, and the region overlap rate of the negative sample is less than 0.3.

[0032] As a further improvement of the present invention, in step 10, the score of the k-th category is updated after the first frame.

[0033]

[0034] Where ⊙ represents the Hadamard product, M k This represents the weight vector for the k-th category.

[0035] As a further improvement of the present invention, in step 12, the three samples are (s, s + ,s - ), where s represents the target template of the RGB image, s + s represents the target template for a thermal infrared image. - Indicates a mixed background.

[0036] The beneficial effects of this invention are: by strengthening the nested global reasoning model and utilizing some multi-domain target information for scene perception updates, this invention can improve the performance of computer vision tracking. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the network structure of the robust visual tracking method of the present invention.

[0038] Figure 2 This is a schematic diagram of the global inference module in the robust visual tracking method of the present invention.

[0039] Figure 3 This is a visualization example of the robust visual tracking method based on meta-self-supervised learning in this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0042] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0043] like Figures 1 to 3As shown, this invention discloses a robust visual tracking method, which is a module transfer method based on unsupervised target tracking. The main technologies include: graph-based global reasoning, unsupervised learning, and meta-learning.

[0044] This invention consists of three main parts. The first part is an RGBT tracking network for multi-task learning, which is the first attempt to explore the consistency problem of multimodal global reasoning under the multi-task learning framework. The second part is a new global reasoning model (named NGloRe) proposed under the multi-task learning framework. The third part is a meta-unsupervised learning strategy that uses a small number of samples and unlabeled samples to update the proposed global reasoning model, enabling it to use partial information of the target to explore semantic region relationships in the scene.

[0045] The robust visual tracking method mainly includes the following steps:

[0046] Step 1: Preprocess the existing video data by cropping each frame of the image into a 107x107x3 resolution image centered on the target.

[0047] Step 2: Train a pre-trained CNN (Convolutional Neural Network) using multiple video sequences, such as... Figure 1 As shown, conv1, conv2, and conv3 (the first three convolutional layers) are similar to the VGG-M network (which contains 5 convolutional layers and 3 fully connected layers), and the target bounding boxes are initialized.

[0048] Step 3: Randomly initialize FC6 (fully connected layer).

[0049] Step 4: Collect 500 positive samples (region overlap rate greater than 0.7) and 5000 negative samples (region overlap rate less than 0.5) as training samples in the first frame.

[0050] Step 5, as follows Figure 2 As shown, the projected input X∈R H ×W×C (input features) are generated in a low-dimensional subspace. (Low-dimensional features).

[0051] Step 6: A graph model G is constructed to explore semantic region relationships in the tracking scene, and the graph representation results are reprojected into the original representation space.

[0052] Step 7: Learn the parameters of the global inference model (NGloRe), i.e., the graph model G. Our proposed graph model is expressed as follows:

[0053]

[0054] Here, A1 is the adjacency matrix of graph G1, and A2 is the adjacency matrix of graph G2. G1, based on this adjacency matrix, explores the relationships between image patches, grouping similar nodes together to highlight semantic regions. The output of G1 is considered the prior knowledge for generating A2. After G2, the relationships between semantic regions in the tracking scene can be further explored. Inspired by traditional GCN methods, graph G1 can be represented as...

[0055]

[0056] Where W1 is the weight matrix, used to represent the parameters of the graph convolution. Based on the above formula, we can obtain...

[0057]

[0058] W2 is a weight matrix used to represent the relationships between semantic regions.

[0059] Step 8: Add the first frame to the short-term tracking and long-term tracking sets.

[0060] Step 9: In subsequent frames, we plot positive samples (region overlap rate greater than 0.7) and negative samples (region overlap rate less than 0.3) as training samples for short-term and long-term updates of the global inference model (NGloRe).

[0061] Step 10: Utilize meta-learning to perform online learning on P (parameter matrix). (Output features) are fine-tuned. In this case, c k The update formula for the score of the k-th category after the first frame is expressed as follows: Where ⊙ represents the Hadamard product, M k This represents the weight vector for the k-th category.

[0062] Step 11: Fine-tune the global inference model (NGloRe) by updating the output features in the short term.

[0063] Step 12, as follows Figure 3 As shown, three types of samples (s, s) are obtained from the last layer of the convolutional neural network. + ,s - ), where s represents the target template of the RGB image, s + s- represents the target template in the thermal infrared image, and s- represents the mixed background. Unlike the traditional three-group loss, which only compares the differences between positive and negative samples within a single image domain.

[0064] Step 13: Train the triplet loss function to adjust the multimodal similarity by minimizing the distance function D(s,s). +This makes the target representations of the RGB image and the thermal infrared image more similar. Simultaneously, we also maximize D(s,s) - The distance.

[0065] The testing phase process includes the following steps:

[0066] S1. Select authoritative test set data and convert the video data into image data to input into the network model.

[0067] S2. Calculate the position error and overlap rate based on the Euclidean distance between the bounding box tested by the algorithm and the real labels manually annotated in the video image. Compare the corresponding curves with other algorithms to complete the verification of this invention.

[0068] In the experiments, we used VGG-16 (a convolutional neural network containing 5 convolutional layers and 3 fully connected layers) as the backbone embedding for features. In the implementation, the first frame collected 500 positive samples (region overlap greater than 0.7) and 5000 negative samples (region overlap less than 0.5) as training samples to learn the parameters of the global inference model (NGloRe). The learning rate was set to 0.0001, and the learning rate for FC6 was also set to 0.001. The training iterations were set to 30. In subsequent frames, we plotted positive samples (region overlap greater than 0.7) and negative samples (region overlap less than 0.3) as training samples for short-term and long-term updates of the global inference model (NGloRe). The learning rates for FC4-5 (fully connected layers 4-5) and FC6 (fully connected layer 6) were set to 0.0002. We set the image input to 107*107*3 and selected the SGD optimizer to optimize the classifier.

[0069] In summary, this invention introduces a global inference model (NGloRe) into the CNN backbone network, achieving semantic relationship inference between regions through coordinate interaction space mapping. Regardless of the changes in the current RGB and thermal infrared images, the current RGB-thermal pair should have the same category as the previous RGB-thermal pair. This invention designs its update strategy based on this principle. The detailed design stems from fully utilizing the advantages of meta-learning and unsupervised learning. By strengthening the nested global inference model and utilizing some multi-domain target information for scene perception updates, this invention can improve the performance of computer vision tracking.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A robust visual tracking method, characterized in that, This tracking method is the RGBT visual tracking method, which mainly includes the following steps: Step 1: Preprocess the existing video data by cropping each frame of the image with the target as the center; Step 2: Train a pre-trained convolutional neural network using multiple video sequences to initialize the target bounding boxes; In step 2, the convolutional neural network includes conv1, conv2 and conv3, and each of conv1, conv2 and conv3 includes 5 convolutional layers and 3 fully connected layers; Step 3: Randomly initialize fully connected layers; fully connected layers FC4, FC5, and FC6 are located at the end of the convolutional neural network, after conv3; Step 4: Collect 500 positive samples and 5000 negative samples from the first frame of the video data as training samples; Step 5: Project input features X∈R H ×W×C is input into a low-dimensional subspace to generate low-dimensional features. Step 6: Construct a graph model G, which represents the result being reprojected into the original representation space; Step 7: Build a global inference model and learn the parametric graph model G of the global inference model; the global inference model is embedded in the middle layer of the convolutional neural network, located between conv2 and conv3; In step 7, the graphical model G is Wherein, the graph model G is composed of nested graph convolutional networks G1 and G2, A1 is the adjacency matrix of graph convolutional network G1, and A2 is the adjacency matrix of graph convolutional network G2. The graph convolutional network G1 is Where W1 is the weight matrix, used to represent the parameters of graph convolution; The graphical model G is represented as Wherein, W2 is the weight matrix, used to represent the relationship between semantic regions; Step 8: Add the first frame to the set of short-term and long-term tracking; Step 9: Starting from the second frame, draw positive and negative samples as training samples for short-term and long-term updates of the global inference model. Step 10: Use meta-learning to learn the parameter matrix P online and output the features. Fine-tuning is performed; in step 10, the score of the k-th category is updated after the first frame. Where ⊙ represents the Hadamard product, M k This represents the weight vector for the k-th category; Step 11: Fine-tune the global inference model by updating the output features in the short term; Step 12: Obtain three types of samples from the last layer of the convolutional neural network; In step 12, the three types of samples are (s, s) + ,s - ), where s represents the target template of the RGB image, s + s represents the target template for a thermal infrared image. - Indicates a mixed background; Step 13: Train the triplet loss function to adjust the multimodal similarity. By minimizing the distance function D(s,s+), the target representations of the RGB image and the thermal infrared image are made more similar, and the distance of D(s,s-) is maximized.

2. The robust visual tracking method according to claim 1, characterized in that: In step 1, the resolution of each frame image is cropped to 107x107x3.

3. The robust visual tracking method according to claim 1, characterized in that: In step 4, the region overlap rate of the positive sample is greater than 0.7, and the region overlap rate of the negative sample is less than 0.

5.

4. The robust visual tracking method according to claim 1, characterized in that: In step 9, the region overlap rate of the positive sample is greater than 0.7, and the region overlap rate of the negative sample is less than 0.3.

Images