A video-level object tracking method based on hidden state temporal information transfer

Abstract

A video-level target tracking method based on hidden state timing information transmission belongs to the fields of machine learning, computer vision, and target tracking. The steps are as follows: Step 1: Obtain data for model training. Step 2: Build the overall network architecture of the model. Step 3: Model training. Step 4: Model reasoning. The video-level high-performance tracker proposed in the present invention can make more full use of the contextual information in the video stream to assist tracking, thereby greatly improving the robustness of the tracking model and improving its performance in various complex scenarios. The present invention explores for the first time the use of hidden states to transmit timing information in video streams. Compared with previous methods, the architecture of the present invention can transmit richer timing information, which provides a precedent for the development of subsequent video-level target trackers.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, computer vision and target tracking, and relates to a state space model, video stream timing information transmission and video-level target tracking training; specifically, it is a video-level target tracking method based on hidden state timing information transmission. Background Art

[0002] Visual target tracking is a basic task in computer vision. Its goal is to specify a specific target in the initial frame given a video sequence. The target tracking algorithm locates the target in subsequent frames by combining the target's features, background information, and timing information. Visual target tracking is the application basis for vision-related problems such as video object detection, action recognition, target positioning, and pedestrian detection and tracking. In addition, visual target tracking has extremely wide applications in video surveillance, drones, robot vision, and security protection.

[0003] Existing target tracking methods usually use image-level matching methods based on twin networks (SiamFC, SiamRPN, SiamRPN++, etc.). These methods usually use two backbone networks with shared parameters to extract features of the template image and the search area image, then match the features of the two, and finally make predictions based on the matching results. These methods only focus on the initial appearance information of the target and do not use the temporal information in the video stream, which makes them insensitive to the appearance changes of the target and background, resulting in poor robustness. In order to improve the robustness of the model, some subsequent tracking methods use additional template images (Stark, ToMP, TransT_M, etc.) to record the changes in the target appearance in the video stream. These methods can obtain more target appearance information by continuously updating additional dynamic templates, thereby enhancing the robustness of the tracker. Although this type of method can alleviate the challenges brought by the change of target appearance to a certain extent, they still focus on the appearance information of the target and do not make full use of the temporal information in the video stream to assist tracking. Recently, some methods (ODTrack, ARTrackV2, EVPTrack, etc.) have begun to explore video-level target tracking. Unlike previous methods that only focus on the appearance information of the target, these methods begin to use the timing information in the video stream to assist tracking and enhance the performance of the model. This type of method usually uses a small number of additional tokens to capture and transmit the timing information in the video stream, and continuously updates these tokens as the video stream advances. Compared with previous methods that track only based on the appearance of the target, this type of method greatly improves the performance of the model. However, this type of method has a flaw. A small number of tokens can only transmit limited timing information, which will lead to the loss of timing information in the video stream.

[0004] Therefore, how to efficiently capture and transmit richer timing information in video streams to further improve the performance of the model is an urgent problem to be solved. Summary of the invention

[0005] The present invention aims to provide a video-level target tracking method based on hidden state temporal information transmission. By designing a context information fusion module, the hidden state of Mamba is used to capture and transmit richer temporal information to assist tracking, thereby improving the performance of the model. The present invention has been tested on multiple dataset benchmarks, and has achieved better performance than other high-performance trackers on these benchmarks, effectively solving the problem that existing trackers cannot efficiently capture and transmit rich temporal information in video streams, and achieving the goal of robust tracking in various scenarios.

[0006] The technical solution of the present invention:

[0007] A video-level object tracking method based on hidden state temporal information transmission, the steps are as follows:

[0008] Step 1: Get the data for model training.

[0009] The data used for model training includes video datasets and image datasets.

[0010] Furthermore, four video datasets LaSOT, TrackingNet, GOT-10k, VastTrack and one image dataset COCO are preferentially used for model training.

[0011] Step 2: Build the overall network architecture of the model. Specifically:

[0012] The overall network architecture of the model includes image segmentation module, backbone network, context information fusion module and tracking head.

[0013] The input of the model includes a video clip and a search area image. The video clip is used to indicate the target to be tracked, including various appearance information of the target. The target is located on the search area image.

[0014] The processing process of the model is as follows: First, the template image and the search area image in the video clip are downsampled through the image segmentation module. Then, the template image and the search area image after downsampling are reshaped and spliced ​​in the spatial dimension, and then input into the subsequent backbone network to extract visual features. The backbone network consists of N backbone network blocks, each of which consists of L Transformer layers. In order to deeply capture the temporal information, a context information fusion module is set up. The context information fusion module includes N temporal information fusion blocks, and each temporal information fusion block is paired with the corresponding backbone network block. The temporal information fusion block integrates the historical context information into the backbone network block paired with it, and the backbone network block extracts more accurate features based on the historical context information. At the same time, the temporal information fusion block will also update its hidden state according to the output of the current backbone network to achieve the purpose of updating the temporal information. Finally, the backbone network outputs more accurate visual features under the guidance of the temporal information, and then passes the visual features to the tracking head to obtain the tracking results.

[0015] Furthermore, the context information fusion module: stores the temporal information in the hidden state and integrates it into the backbone network to enhance the extraction of visual features. The temporal information fusion block includes a Mamba layer, an input attention layer, an output attention layer, and a feedforward network layer. The Mamba layer is responsible for storing the temporal information. The input attention layer integrates the temporal information into the backbone network, and the output attention layer extracts the current frame information from the features of the backbone network.

[0016] Output of the previous time series information fusion block First, the context information is extracted through the Mamba layer to obtain the output of the Mamba layer of the current time series information fusion block At the same time, the Mamba layer will Update its own hidden state to achieve the purpose of updating the timing information. Next, the visual features output by the previous backbone network block As query Q, As the key K and value V, they are input into the attention layer to obtain the features after temporal information enhancement. ,then It is input into the backbone network block corresponding to the current time series information fusion block for further feature extraction to obtain the visual features output by the current backbone network block. , then As key K and value V, As query Q, they are passed through the residual connection output attention layer and feedforward network layer to obtain the current time series information fusion block output The processing process of the context information fusion module is as follows:

[0017]

[0018]

[0019]

[0020]

[0021] in , , , Respectively represent the output of the Mamba layer of the i-th temporal information fusion block, the hidden state at the current moment, the hidden state at the previous moment, and the visual features output by the temporal information fusion block; represents the output of the i-1th time series information fusion block; represents the output of the output attention layer in the i-th temporal information fusion block; , Represent the visual features output by the i-th and i-1-th backbone network blocks respectively; , , They represent the Mamba layer, input attention layer, output attention layer, and feedforward network layer of the i-th temporal information fusion block respectively; It represents the i-th backbone network block.

[0022] Furthermore, the Mamba layer is responsible for storing timing information. It includes an x ​​branch and a z branch. First, the input of the Mamba layer After normalization by RMS normalization layer, we get . Next, it is processed by the x branch and the z branch. In the x branch, After passing through the linear mapping layer, convolution layer, activation function, and finally input into the state space layer to extract the time series information in the features, and update the hidden state according to the context information to update the stored time series information; after passing through the x branch, the visual features that integrate the previously stored time series information will be output In the z branch The features will be linearly mapped and activated by the output of the x branch. Reweighting; the reweighted features are then passed through the linear mapping layer to obtain the output of the z branch ,at last Connect via jump Combine to get the output of the Mamba layer.

[0023] Furthermore, the state space layer: The state space layer is formed by the hidden function of the system The one-dimensional function Mapping The continuous-time state-space layer is represented by a set of linear differential equations, specifically the state equation and the output equation as shown below:

[0024]

[0025]

[0026] in , , , is the weight parameter of the continuous-time state-space layer; Represents the updated hidden state; to use the state space layer in deep learning, the state space layer must be discretized. Usually, the zero-order hold method is used for discretization to convert continuous parameters into discrete parameters, as shown below:

[0027]

[0028]

[0029] in and is the discretization parameter; is the time scale parameter. After discretization, the state space layer used is as follows:

[0030]

[0031]

[0032] in , , , is the weight parameter of the discretized state space layer; , , Respectively represent the hidden state, output, and input at the current moment; Represents the hidden state at the previous moment.

[0033] Furthermore, the tracking head consists of a classification head and a regression head. Both parts contain three sub-networks, each of which consists of a convolutional layer. After these sub-networks, a classification score is obtained. , a target box size , an offset dimension , by combining these three results, we get the tracking result.

[0034] Step 3: Model training.

[0035] The objective function during training includes classification loss, focal loss and regression loss, where the regression loss consists of L1 loss and GIoU loss.

[0036] Furthermore, in the video-level training method, each frame will generate an independent loss. The total loss is summarized as follows:

[0037]

[0038] in , , They represent the classification loss, L1 loss and GIoU loss of the i-th frame respectively. Represents the total loss. , , represents a hyperparameter.

[0039] Step 4: Model inference. During the inference process, a fixed-length video clip is input, which is consistent with the training process. At the same time, a memory library is established to store reliable frames, which are frames whose classification scores are greater than the set threshold to provide more appearance information. When the update interval T is reached, the frames stored in the memory library are used to update the video clip. Updating the hidden state when the tracking result is inaccurate will introduce erroneous contextual information, which will reduce the performance of the model in subsequent frames. To alleviate this problem, it is necessary to set another hidden state threshold to update the hidden state: the hidden state will only be updated when the classification score exceeds the hidden state threshold.

[0040] Beneficial effects of the present invention:

[0041] (1) The video-level high-performance tracker proposed in the present invention can make more effective use of contextual information in the video stream to assist tracking, thereby greatly improving the robustness of the tracking model and improving its performance in various complex scenarios.

[0042] (2) This invention explores for the first time the use of hidden states to convey timing information in video streams. Compared with previous methods, the architecture of this invention can convey richer timing information, which provides a precedent for the subsequent development of video-level target trackers. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A framework diagram for transferring temporal information using hidden states.

[0044] Figure 2 This is the overall network architecture diagram of the model.

[0045] Figure 3 This is the architecture diagram of the timing information fusion block.

[0046] Figure 4 This is the Mamba layer network architecture diagram. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0048] Example

[0049] A video-level object tracking method based on hidden state temporal information transmission, the steps are as follows:

[0050] Step 1: Get data for model training. Four video datasets LaSOT, TrackingNet, GOT-10k, VastTrack and one image dataset COCO are used for model training.

[0051] Step 2: Build the overall network architecture of the model. Figure 1 , extract the visual features of the template image and the search area image through the encoder, and then use the hidden state layer to interact with the encoder in a two-way manner. Update the hidden state information of the hidden state layer based on the image feature information extracted by the encoder, and at the same time, merge the time series information stored in the hidden state layer into the features extracted by the encoder to achieve the purpose of using the hidden state to transmit the time series information. Figure 1 The framework diagram shown in the figure builds the overall network architecture of the model. Specifically: Figure 2 As shown in the figure, the overall network architecture of the model uses the pre-trained Transformer network Fast-iTPN as the backbone network and is equipped with a context information fusion module to transmit context information in the video stream.

[0052] The overall network architecture of the model includes image segmentation module, backbone network, context information fusion module and tracking head.

[0053] The input of the model includes a video clip and a search area image. The video clip is used to indicate the target to be tracked, including various appearance information of the target. The target is located on the search area image.

[0054] The processing process of the model is as follows: First, the template image and search area image in the video clip are downsampled through the image segmentation module. (The specific steps of the image segmentation module are: the template image and search area image in the video clip are extracted with a convolution layer and downsampled by 4 times; then the template image and search area image after downsampling are processed by a multi-layer perceptron and two convolution merging layers respectively, and after processing, image blocks downsampled by 16 times are obtained.) Then the template image block and the search area image block are reshaped and spliced ​​in the spatial dimension, and then input into the subsequent backbone network to extract visual features. The backbone network consists of N backbone network blocks, each of which consists of L Transformer layers. In order to deeply capture the temporal information, a context information fusion module is set, which includes N temporal information fusion blocks, each of which is paired with a corresponding backbone network block. The temporal information fusion block integrates the temporal information into the backbone network block paired with it, and the backbone network block extracts more accurate features based on the temporal information. At the same time, the time series information fusion block will also update its hidden state according to the output of the current backbone network to achieve the purpose of updating the time series information. Finally, the backbone network outputs more accurate visual features under the guidance of the time series information, and then passes the visual features to the tracking head to obtain the tracking results.

[0055] like Figure 2 As shown, the context information fusion module is the core component of the present invention. It stores the temporal information into the hidden state and integrates it into the backbone network to enhance the extraction of visual features. Each temporal information fusion block in the context information fusion module integrates the captured temporal information into the features of the backbone network before the corresponding backbone network block performs feature extraction, and updates its hidden state according to the features of the current frame. This deep integration enables the context information fusion module to extract multiple levels of temporal information, providing richer implicit context information to improve the prediction of the model. The structure of the temporal information fusion block is shown in Figure 3 As shown in Figure 1, it mainly includes the Mamba layer, input attention layer, output attention layer and feedforward network layer. The specific processing process is as follows: the output of the previous time series information fusion block First, the context information is extracted through the Mamba layer to obtain the output of the Mamba layer of the current time series information fusion block At the same time, the Mamba layer will Update its own hidden state to achieve the purpose of updating the timing information. Next, the visual features output by the previous backbone network block As query Q, As the key K and value V, they are input into the attention layer to obtain the features after temporal information enhancement. ,then It is input into the backbone network block corresponding to the current time series information fusion block for further feature extraction to obtain the visual features output by the current backbone network block. , then As key K and value V, As query Q, they are passed through the residual connection output attention layer and feedforward network layer to obtain the current time series information fusion block output The processing of the context information fusion module can be summarized as follows:

[0056]

[0057]

[0058]

[0059]

[0060] in , , , Respectively represent the output of the Mamba layer of the i-th temporal information fusion block, the hidden state at the current moment, the hidden state at the previous moment, and the visual features output by the temporal information fusion block; represents the output of the i-1th time series information fusion block; represents the output of the output attention layer in the i-th temporal information fusion block; , Represent the visual features output by the i-th and i-1-th backbone network blocks respectively; , , They represent the Mamba layer, input attention layer, output attention layer, and feedforward network layer of the i-th temporal information fusion block respectively; It represents the i-th backbone network block.

[0061] The Mamba layer is a key component of the time series information fusion block. Its structure is as follows Figure 4 As shown, first enter After normalization by RMS normalization layer, we get . Next, it is processed through two branches: x branch and z branch. In the x branch, After the linear mapping layer, the dimension is expanded, and then the local features are extracted through the convolution layer. After being activated by the SiLU activation function, it is sent to the state space layer to extract and update the stored context information. After the x branch, we get In the z branch, The dimension is expanded and activated through the linear mapping layer and SiLU activation function, and then the output of the x branch The weighted value is then reduced through the linear mapping layer to obtain the output of the z branch. .at last Connect via jump Combined to get the output of the Mamba layer .

[0062] The state space layer is a key part of the Mamba layer. Inspired by continuous systems, the state space layer uses the hidden functions of the system to The one-dimensional function Mapping The continuous-time state-space layer can be represented by a set of linear differential equations: the state equation and the output equation are shown below:

[0063]

[0064]

[0065] in , , , is the weight parameter of the continuous-time state-space layer; Represents the updated hidden state; to use the state space layer in deep learning, the state space layer must be discretized. Usually, the zero-order hold method is used to discretize it, converting continuous parameters into discrete parameters, as shown below:

[0066]

[0067]

[0068] in and is the discretization parameter; is the time scale parameter. After discretization, we can get the state space layer that can be used as follows:

[0069]

[0070]

[0071] in , , , is the weight parameter of the discretized state space layer; , , Respectively represent the hidden state, output and input at the current moment; Represents the hidden state at the previous moment.

[0072] The tracking head consists of two parts: the classification head and the regression head. Both parts contain three sub-networks, each of which consists of a convolutional layer. After these sub-networks, a classification score is obtained. , a target box size , an offset dimension ,By combining these three results, we can get the tracking result.

[0073] Step 3: Model training.

[0074] After the complete network structure is built, the five selected datasets can be used to perform offline training on the built network. During training, the input of the model consists of video clips and search area images. For the video dataset, images are sampled from random video sequences. For the image dataset COCO, an image is randomly selected, and video clips and search area images are generated through data augmentation technology. Common data augmentation methods such as scaling, translation, and jittering are applied to these images. The search area image and video clip image are obtained by expanding the target box by 4 times and 2 times respectively, and then the search area image is adjusted to pixels, the video clip image is adjusted to Pixels. For the backbone network, the pre-trained Transformer network Fast-iTPN parameters are used to initialize it, and other parts are randomly initialized. The AdamW optimizer is used for training, and the weight decay is set to The initial learning rate is . The model training was performed for 300 rounds, each round contained 60,000 samples, and the learning rate was reduced by a factor of 10 at the 240th round. Two Tesla A800 GPUs were used in the training process, and the total batch size was set to 128. The objective function during training includes classification loss, focal loss loss, and regression loss, where the regression loss consists of L1 loss and GIoU loss. In the video-level training method, each frame will generate an independent loss. The total loss can be summarized as follows:

[0075]

[0076] in , , They represent the classification loss, L1 loss and GIoU loss of the i-th frame respectively. Represents the total loss. , , represents a hyperparameter.

[0077] Step 4: Model inference. During inference, a video clip of length 5 is input, which is consistent with the training process. At the same time, a memory library is established to store reliable frames (reliable frames are those with a classification score greater than a set threshold) to provide more appearance information. When the update interval T is reached, the frames stored in the memory library are used to update the video clip. Updating the hidden state when the tracking results are inaccurate may introduce erroneous contextual information, which may reduce the performance of the model in subsequent frames. To alleviate this problem, a hidden state threshold needs to be set to update the hidden state: the hidden state will only be updated when the classification score exceeds the hidden state threshold.

[0078] Table 1 shows the detailed information of the five models.

[0079]

[0080] The present invention has developed a total of 5 different models. The backbone networks, input resolutions, parameter amounts, computational complexity, and speeds used by the 5 models are shown in Table 1. The speed of the model was tested on an NVIDIA RTX 2080 GPU.

[0081] Table 2 shows the comparison with other high performance trackers

[0082]

[0083] Furthermore, we also use LaSOT, TrackingNet, GOT-10k, The five models of the present invention are tested on the test sets of the four datasets. Table 2 shows the comparison of the success rates of the five models of the present invention and other high-performance tracking models on the four datasets.

Claims

1. A video-level target tracking method based on hidden state temporal information transmission, characterized in that: Here are the steps: Step 1: Get data for model training; The data used for model training includes video datasets and image datasets; Step 2: Build the overall network architecture of the model; specifically: The overall network architecture of the model includes image segmentation module, backbone network, context information fusion module and tracking head; The input of the model includes a video clip and a search area image. The video clip is used to indicate the target to be tracked, including various appearance information of the target; the target is located on the search area image; The processing process of the model is as follows: first, the template image and the search area image in the video clip are downsampled through the image segmentation module; then the downsampled template image and the search area image are reshaped and spliced ​​in the spatial dimension, and then input into the subsequent backbone network to extract visual features; The backbone network consists of N backbone network blocks, each of which consists of L Transformer layers. In order to deeply capture temporal information, a context information fusion module is set up. The context information fusion module includes N temporal information fusion blocks, each of which is paired with a corresponding backbone network block. The temporal information fusion block integrates historical context information into the paired backbone network block, and the backbone network block extracts more accurate features based on the historical context information. At the same time, the temporal information fusion block also updates its hidden state according to the output of the current backbone network to achieve the purpose of updating the temporal information. Finally, the backbone network outputs more accurate visual features under the guidance of the temporal information, and then passes the visual features to the tracking head to obtain the tracking results. Step 3: Model training; The objective function during training includes classification loss, focal loss and regression loss, where the regression loss consists of L1 loss and GIoU loss; Step 4: Model inference; During the inference process, a fixed-length video clip is input, which is consistent with the training process; At the same time, a memory library is established to store reliable frames. Reliable frames refer to frames whose classification scores are greater than the set threshold to provide more appearance information; When the update interval T is reached, the frames stored in the memory library are used to update the video clip; When the tracking result is inaccurate, updating the hidden state will introduce erroneous contextual information, which will reduce the performance of the model in subsequent frames; To alleviate this problem, it is necessary to set a hidden state threshold to update the hidden state: the hidden state will only be updated when the classification score exceeds the hidden state threshold.

2. According to claim 1, a video-level target tracking method based on hidden state timing information transmission is characterized in that: Four video datasets LaSOT, TrackingNet, GOT-10k, VastTrack and an image dataset COCO are used for model training.

3. The video-level target tracking method based on hidden state temporal information transmission according to claim 1 is characterized in that: The context information fusion module: stores the temporal information in the hidden state and integrates it into the backbone network to enhance the extraction of visual features; the temporal information fusion block includes a Mamba layer, an input attention layer, an output attention layer, and a feedforward network layer; the Mamba layer is responsible for storing the temporal information; the input attention layer integrates the temporal information into the backbone network, and the output attention layer extracts the current frame information from the features of the backbone network; Output of the previous time series information fusion block First, the context information is extracted through the Mamba layer to obtain the output of the Mamba layer of the current time series information fusion block ; At the same time, the Mamba layer will Update its own hidden state to achieve the purpose of updating the timing information; then the visual features output by the previous backbone network block As query Q, As the key K and value V, they are input into the attention layer to obtain the features after temporal information enhancement. ,then It is input into the backbone network block corresponding to the current time series information fusion block for further feature extraction to obtain the visual features output by the current backbone network block. , then As key K and value V, As query Q, they are passed through the residual connection output attention layer and feedforward network layer to obtain the current time series information fusion block output ; The processing process of the context information fusion module is as follows: in , , , Respectively represent the output of the Mamba layer of the i-th temporal information fusion block, the hidden state at the current moment, the hidden state at the previous moment, and the visual features output by the temporal information fusion block; represents the output of the i-1th time series information fusion block; represents the output of the output attention layer in the i-th temporal information fusion block; , Respectively represent the visual features output by the i-th and i-1-th backbone network blocks; , , They represent the Mamba layer, input attention layer, output attention layer, and feedforward network layer of the i-th temporal information fusion block respectively; It represents the i-th backbone network block.

4. The video-level target tracking method based on hidden state timing information transmission according to claim 3 is characterized in that: The Mamba layer is responsible for storing timing information, including x branch and z branch. First, the input of the Mamba layer After normalization by RMS normalization layer, we get ; Next, it is processed by the x branch and the z branch; In the x branch, After going through the linear mapping layer, convolution layer, activation function, and finally input into the state space layer to extract the time series information in the features, and update the hidden state according to the context information to update the stored time series information; After the x branch, the visual features that integrate the previously stored temporal information are output ; In the z branch The features will be linearly mapped and activated by the output of the x branch. Reweighting; the reweighted features are then passed through the linear mapping layer to obtain the output of the z branch ,at last Connect via jump Combine to get the output of the Mamba layer.

5. The video-level target tracking method based on hidden state timing information transmission according to claim 4 is characterized in that: The state space layer: The state space layer is through the hidden function of the system The one-dimensional function Mapping ; The continuous-time state-space layer is represented by a set of linear differential equations. The state equation and output equation are shown below: in , , , is the weight parameter of the continuous-time state-space layer; represents the updated hidden state; To use the state space layer in deep learning, the state space layer must be discretized; usually the zero-order hold method is used for discretization to convert continuous parameters into discrete parameters, as shown below: in and is the discretization parameter; is the time scale parameter; after discretization, the state space layer used is as follows: in , , , is the weight parameter of the discretized state space layer; , , Respectively represent the hidden state, output, and input at the current moment; Represents the hidden state at the previous moment.

6. The video-level target tracking method based on hidden state temporal information transmission according to claim 1 is characterized in that: The tracking head consists of two parts: a classification head and a regression head; both parts contain three sub-networks, each of which consists of a convolutional layer; after passing through these sub-networks, a classification score is obtained , a target box size , an offset dimension , by combining these three results, we get the tracking result.

7. The video-level target tracking method based on hidden state temporal information transmission according to claim 1 is characterized in that: In the video-level training approach, each frame incurs an independent loss; the total loss is summarized as follows: in , , Represent the classification loss, L1 loss and GIoU loss of the i-th frame respectively; represents the total loss; , , Represents a hyperparameter.

Images