A video target tracking method using pixel-level spatiotemporal feature memory

Abstract

The present invention belongs to the field of computer vision technology, and specifically is a video target tracking method using a pixel-level spatiotemporal feature memory library. The method of the present invention includes the following steps: cropping dynamic target images and search area images, extracting image features, pixel-level online prediction, pixel-level update strategy "fusion-replacement", feature enhancement and pixel-level matching, and head network structure to achieve target tracking. After inputting the initial frame of the target in the initial frame, the present invention saves storage resources and computing resources by efficiently utilizing the memory library, can effectively cope with scenarios such as changes in the appearance of the target in the video, and has high adaptability and robustness. At the same time, the method can achieve rapid deployment for the currently popular convolutional neural network model and Transformer model. The method is suitable for automatically generating and tracking the bounding box of any target in the video in a computer-assisted video analysis scenario, and can be used for security monitoring analysis, automatic driving, human-computer interaction, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a video target tracking method using a pixel-level spatiotemporal feature memory library. Background Art

[0002] Object tracking is a key area in computer vision and a crucial component of video analysis. Video object tracking involves continuously detecting and identifying the position and shape of a specific object in a video using bounding boxes, given its initial state. With the widespread adoption of computer technology and the availability of advanced imaging equipment, video has become a mainstream medium for information dissemination. Therefore, analyzing the image content in videos is of great significance in real-world production and life. Video object tracking utilizes computer technology to analyze and utilize the geometric information of objects in videos, including shape, texture, and posture. This information can be used to mine the objects' motion, thereby studying their motion patterns and supporting system decision-making. Consequently, video object tracking currently has a wide range of applications, such as security monitoring and analysis, autonomous driving, and human-computer interaction. It also plays a significant role in the military.

[0003] Existing target tracking methods use known target information to extract features from the known target information and the current video frame through a Siamese neural network (SiameseNetwork), which is two neural networks with the same network architecture and parameters. The two are encoded into the same embedding space, and then the target is retrieved and matched in the current video frame features to determine the target position and shape. Among them, the initial state of the target is the most important source of known target information. However, in the actual application of target tracking, the change in the appearance of the target limits the performance of existing target tracking methods. As the target deforms in the video or the video perspective changes, the appearance information of the target changes significantly compared to the initial information. It is difficult to effectively use the initial target information for matching in the current video frame, resulting in tracking drift or target loss, and poor adaptability and robustness to the target appearance.

[0004] In most video sequences, the appearance of a target does not change suddenly, but rather gradually as the sequence progresses. Therefore, the most effective way to address deformation in target tracking methods is to better utilize the temporal information of the target in the video, analyze the target's state in the video sequence, and thus grasp the target's deformation. In recent years, researchers have enriched target information and improved the ability of target tracking methods to cope with target deformation by storing the target's historical state in real time. However, in practical applications, target tracking methods find it difficult to store excessive target information due to limited storage space. Furthermore, target tracking tasks place high demands on the real-time performance of the methods. Storing too much target information will result in greater consumption of computing resources, thus affecting the method's speed and hindering its application in real-world scenarios.

[0005] In summary, how to more efficiently store the historical state of the target, achieve a balance between performance and the consumption of storage and computing resources, and then make full use of the target's timing information in the video, and improve the adaptability and robustness of the tracking method when dealing with changes in the target's appearance, has high research significance and practical application value. Summary of the Invention

[0006] The purpose of the present invention is to propose a video target tracking method using a pixel-level spatiotemporal feature memory bank with high memory bank storage efficiency, good adaptability and robustness to target appearance changes.

[0007] The video target tracking method provided by this invention utilizes a pixel-level spatiotemporal feature memory library to store and update target feature pixels in the time domain. While maintaining the memory size, it fully utilizes the target's historical information in the video time sequence, adapts to target deformation in the current video frame, and determines the target's position and posture through pixel-level matching in the current video frame, achieving more robust tracking. This method is inspired by the human brain's mechanism of target tracking. Specifically, during long-term target tracking, the human brain first memorizes the target's basic features, such as texture and color, which are not easily changed in video. Subsequently, it continuously adapts to new changes in the target in the video to accurately track it.

[0008] The video target tracking method using a pixel-level spatiotemporal feature memory library provided by the present invention comprises the following specific steps:

[0009] Step (1): Based on the target reference frame of the initial frame or the tracking result bounding box of the previous frame, the target is cut out as a dynamic target image; based on the target state of the previous frame, the search area is cut out in the current frame;

[0010] Step (2): Use the twin neural network to extract features from the dynamic target image and the search area, and map them to the same feature space; the initial frame target features are used to construct an initial pixel-level spatiotemporal feature memory bank (hereinafter referred to as the memory bank), and the features of the dynamic target image (hereinafter referred to as the dynamic target features) are used to update the memory bank;

[0011] Step (3): Use the pixel-level online prediction module to predict the basic features and change-related features of the dynamic target features to guide the update process;

[0012] Step (4): Based on the basic features and change-related features of the target retrieved in step (3), a pixel-level update strategy is designed to fuse the basic features with the basic features in the memory library to form new basic features; and the change-related features are used to replace the outdated change features in the memory library;

[0013] Step 5: Enhance the memory and search area features, then use the pixel-level matching and fine-tuning module to match the target information in the memory with the search area features, generate a target position weight map, and fine-tune it.

[0014] Step (6): Input the target position weight map into the head network module, use the classification network to distinguish the target from the background, use the intersection-over-union network to predict the intersection-over-union ratio of each bounding box with the true target box, and use the regression network to regress the specific coordinates of each bounding box;

[0015] Step (seven): Combine the classification score and the score obtained by intersection and union, index the position of the point with the highest score in the position weight map, and find the corresponding regression coordinates, which are the coordinates of the tracking target bounding box.

[0016] Each step is further described below.

[0017] In step (1), this method uses the target's state in the previous frame to determine the search area for the current frame, based on the property that the target's displacement and deformation do not change drastically between consecutive frames. Simultaneously, the cropped dynamic target image and search area are resized to a fixed size to facilitate the design of the tracking network structure.

[0018] In step (2): use the twin neural network to extract features of the dynamic target image and the search area, and map the two to the same feature space, that is, input the two into a network with the same network structure and weights; the initial frame target image features construct the initial pixel-level spatiotemporal feature memory bank (hereinafter referred to as: memory bank), and the features of the dynamic target image (hereinafter referred to as: dynamic target features) are used to update the memory bank.

[0019] In step (3), the "using pixel-level online prediction module to predict the basic features and change-related features of the target in the dynamic target features to guide the update process" specifically inputs the dynamic image features into the pixel-level online prediction module. The prediction module is a dynamic filter F that calculates the correlation between the dynamic image features and the target features in the memory library to achieve the prediction function. This method is based on the property that convolution in the time domain is equal to the product in the frequency domain. First, the memory library M and the dynamic target features f are correlated. z Perform Fourier transform and get M * and Then, the memory bank M * Calculate the matrix Hadamard product with the dynamic filter F to obtain the response map R. The response map R here is obtained by fitting the dynamic target features To achieve the prediction function, the specific formula is as follows, where ⊙ is the Hadamard product, that is, the dot product of the corresponding elements of the matrix:

[0020] R=M * ⊙F (1);

[0021] In order to ensure the fitting effect of the response graph R, this method uses online learning to minimize the dynamic target features after the response graph R and Fourier transform. The squared error loss is used to improve the prediction accuracy. The specific formula is as follows, where n is the n units in the memory bank:

[0022]

[0023] Subsequently, this method uses the Wirtinger derivative to solve the optimization problem and obtains the general expression of the dynamic filter F, which is as follows, where η is the adjustment factor hyperparameter and F′ is the filter before update:

[0024]

[0025] When tracking each frame, this method repeats step (3) continuously, so that the response map R and the dynamic target feature Therefore, the response scores in the response graph R correspond to the correlation between the dynamic target features and the memory bank. That is, higher scores correspond to the basic features in the dynamic target features, while lower scores correspond to the features related to changes in the dynamic target features. The pixels in the dynamic target features are sorted according to the scores to reconstruct the dynamic target feature sequence.

[0026] The step (four) described in "based on the basic features and change-related features of the target retrieved in step (three), design a pixel-level update strategy, correspondingly fuse the basic features with the basic features in the memory bank to form new basic features; replace the outdated change features in the memory bank with the change-related features" includes designing a "fusion-replacement" update strategy. Specifically, after step (three), a reconstructed dynamic target feature sequence is obtained, that is, it has been sorted according to the correlation with the variable features in the memory bank (hereinafter referred to as the "variable memory bank"). Step (four) aims to fuse the top-ranked dynamic target feature pixels with the pixels in the variable memory bank to form new pixels, and at the same time replace the bottom-ranked dynamic target feature pixels with the pixels in the variable memory bank to update the memory bank. Therefore, the present method designs a pixel pair indexing module based on cosine similarity to index the corresponding dynamic target feature pixels and pixels in the variable memory bank for the update process to form pixel pairs. Specifically, step (four) first calculates the reconstructed dynamic target feature sequence f z The cosine similarity matrix D with the memory library M, where ||·||2 is the L2 norm or Euclidean distance:

[0027]

[0028] Then, the maximum index of the columns of matrix D is calculated to form d. Here, d stores the position index of the dynamic target feature pixel and the memory pixel with the most similar semantic information. Pixel pairs are constructed based on the index. That is, the two pixels in the pixel pair, although from different features, have the most similar semantic information. The specific formula for this step is as follows, where argmax refers to the maximum index and row represents the row index:

[0029] d = argmax row D (5);

[0030] Where argmax refers to the maximum value index;

[0031] Based on the obtained pixel pairs, this method fuses the top half of the pixel pairs into a new pixel, replaces the pixels in the memory bank with the dynamic target feature pixels in the bottom half of the pixel pairs, and updates the memory bank.

[0032] The step (5) described in "feature enhancement of the memory bank and search area features, and then matching the target information in the memory bank with the search area features through the pixel-level matching and fine-tuning module to generate a target position weight map and fine-tune it" includes using the Transformer-based feature enhancement module to enhance the memory bank and search area features. Specifically, the self-attention mechanism is used to enhance the memory bank and search area features respectively, and the mutual attention mechanism is used to build the relationship between the memory bank and the search area features. The pixel-level matching operation is based on matrix multiplication. Specifically, the memory bank M and the search area feature f s Multiply them together to get the target position weight map h. Then, a two-layer fully convolutional neural network is used Fine-tune the number of channels of the target position weight map h. Here, in order to adapt to the pixel-level matching process, the convolution kernel size of the full convolution network is set to 1. The formula for this step is as follows, where For a fully convolutional neural network:

[0033]

[0034] In step (6), the target position weight map is input into the head network module, the classification network is used to distinguish the target from the background, the intersection-over-union (IoU) network is used to predict the IoU ratio of each bounding box with the true target box, and the regression network is used to regress the specific coordinates of each bounding box. The head network includes a classification network, an IoU network, and a regression network. These networks are all composed of multi-layer perceptrons. These three networks are used to predict the classification score s of the target and background, predict the IoU ratio i of each bounding box with the true target box, and regress the specific coordinates r of each bounding box.

[0035] The step (seven) "combining the classification score and the score obtained by the intersection and union ratio, indexing the position of the highest score point in the position weight map, and finding the corresponding regression coordinates, which are the coordinates of the tracking target bounding box" is specifically done by combining the classification score s and the intersection and union ratio i to obtain the score, indexing the position of the highest score point pos in the position weight map, and obtaining the coordinates coord of the target box in the regression coordinate r, which can be expressed by the following formula:

[0036] pos=argmax(s τ ·i 1-τ ),coord=r[pos] (7);

[0037] Here, argmax refers to the maximum value index.

[0038] Features and positive effects of the present invention

[0039] According to the video target tracking method of the present invention, the specific steps are as follows: step 1, based on the target reference frame of the initial frame or the tracking result bounding box of the previous frame, the dynamic target image is cut out, and based on the target state of the previous frame, the search area is cut out in the current frame; step 2, using the twin neural network to extract features of the dynamic target image and the search area, wherein the dynamic target features will be used to construct and update the pixel-level spatiotemporal feature memory library (hereinafter referred to as: memory library); step 3, using the pixel-level online prediction module to predict the basic features of the target and the features related to the change in the dynamic target features to guide the update process; step 4, based on the basic features and the features related to the change of the target retrieved in step 3, when updating the memory library, Design a pixel-level update strategy, integrate it with the corresponding basic features of the memory bank, and replace the features related to the changes in the memory bank; Step five, through the pixel-level matching and fine-tuning module, match the target information in the memory bank with the search area features, generate a target position weight map, and fine-tune it; Step six, input the target position weight map into the head network module, use the classification network to distinguish the target from the background, use the intersection-over-union network to predict the intersection-over-union ratio of each bounding box to the true target box, and the regression network regresses the specific coordinates of each bounding box; Step seven, obtain the score by combining the classification score and the intersection-over-union ratio, index the position of the highest score point in the position weight map, and find the corresponding regression coordinates, which are the coordinates of the tracking target bounding box.

[0040] Therefore, the method of the present invention takes into account the consumption of memory and computing resources in the actual tracking process, utilizes pixel-level storage and updating, compresses the storage space, improves the storage efficiency of the memory bank, and makes the memory bank update process more flexible. This idea can be widely applied to other video target tracking methods.

[0041] In addition, unlike other tracking methods that use simple frame-level updates to construct a memory library, this method uses online learning to achieve more accurate pixel-level updates. Step three predicts the basic features of the target and the features related to changes in the dynamic target features to guide the update process. Specifically, a dynamically updated filter is designed to calculate the matrix Hadamard product with the memory library to obtain a response map, and construct the squared difference loss between the response map and the dynamic target features. The online learning minimizes the loss process and updates the filter. This allows the response map in this step to continuously fit the dynamic target features during the tracking process. The high and low response scores in the response map correspond to the correlation between the dynamic target features and the memory library, that is, higher scores correspond to the basic features in the dynamic target features, and conversely, lower scores correspond to the features related to changes in the dynamic target features. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a main algorithm flow chart of the video target tracking method based on the pixel-level spatiotemporal feature memory library in an embodiment of the present invention.

[0043] Figure 2 4 is a flowchart of the pixel-level online prediction module in step three of an embodiment of the present invention.

[0044] Figure 3 This is the pixel-level update strategy "fusion-update" in step four of the embodiment of the present invention.

[0045] Figure 4 This is a feature enhancement flow chart of step five in an embodiment of the present invention.

[0046] Figure 5 These are effect diagrams of the embodiments of the present invention in some actual videos. DETAILED DESCRIPTION

[0047] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described in detail in the following embodiments with reference to the accompanying drawings.

[0048] Figure 1 FIG is an algorithm flow chart of the video target tracking method based on the pixel-level spatiotemporal feature memory library in this embodiment. Figure 1 As shown, the method of the present invention includes the following steps.

[0049] (1) Crop the dynamic target image and the search area image. This example uses the given reference frame on the first frame or the bounding frame obtained by tracking to crop the target image as the dynamic target image z on the previous frame. To facilitate the design of subsequent modules, this example scales the dynamic target image and adjusts its size to 128×128 in RGB format. For the search area image, this example uses the property that "the target will not undergo drastic deformation and displacement between two adjacent frames" and refers to the target state of the previous frame, including position and size, to magnify the target length and width by three times and crop out the search area. Consistent with the settings of the dynamic target image, adjust the size of the search area image to 256×256 in RGB format.

[0050] (2) Use the twin neural network to extract features from the dynamic target image and search area. In the implementation of this example, the twin neural network is converted into a logical structure, that is, a single network model is reused twice to simulate two networks with the same network structure and weights. This example uses a pre-trained ResNet50 neural network as the backbone network of the twin neural network, in which the fifth layer and the fully connected layer of the network are discarded, and only the first four layers are retained. At the same time, this example adjusts the convolution step size of the fourth layer of the network from 2 to 1, and adjusts the 3×3 convolution kernel to a hollow convolution. The downsampling multiple of this network is 8 times. The dynamic target feature obtained in this way is 16×16×1024, and is used to construct a pixel-level spatiotemporal feature memory bank (memory bank). The size of the search area feature is 32×32×1024.

[0051] (3) Use the pixel-level online prediction module to predict the basic features and change-related features of the target in the dynamic target features. Figure 2 This is the algorithm flow chart of the pixel-level online prediction module. Before tracking begins, the instance has constructed an initial memory bank M with a size of 3×16×16×1024 by repeating the target features of the first frame. This instance performs a z Perform fast Fourier transform and get M * and This example constructs a dynamic filter F with a size of 16×16×1024. This example keeps the first feature in the memory unchanged because the first feature is drawn by the given initial box and has the highest credibility to avoid tracking drift. The remaining memory M * It is called a variable memory bank with a size of 2×16×16×1024. It calculates the matrix Hadamard product with the dynamic filter F in turn, converts the convolution in the time domain into the product in the frequency domain, and accumulates the results to obtain the response graph R, which is consistent with the size of the dynamic target feature. The response graph R here is obtained by fitting the dynamic target feature. Implement the prediction function, where the memory bank has 3 basic units, so n is 2, 3:

[0052]

[0053] In order to ensure the fitting effect of the response graph R, this method uses online learning to minimize the dynamic target features after the response graph R and Fourier transform. The square error loss is used to improve the prediction accuracy. The specific formula is as follows:

[0054]

[0055] Subsequently, the Wirtinger derivative is used to solve the optimization problem and the general expression of the dynamic filter F is obtained. The specific formula is as follows:

[0056]

[0057] Here, this example uses η as the update weight, which is 0.4 in the implementation. When tracking each frame, this example repeats step 3 so that the response graph R is consistent with the dynamic target feature. Therefore, the response scores in the response graph R correspond to the correlation between the dynamic target features and the memory bank. That is, higher scores correspond to the basic features in the dynamic target features, while lower scores correspond to the features related to changes in the dynamic target features. The pixels in the dynamic target features are sorted according to the scores to reconstruct the dynamic target feature sequence.

[0058] (4) Design a pixel-level “fusion-replacement” update strategy. Step 3 obtains a reconstructed dynamic target feature sequence, which is sorted according to its relevance to the memory library. Figure 3 This is a flowchart for the "fusion-replacement" update strategy. Based on the target's basic features and change-related features retrieved in step 3, the corresponding basic features are fused with the basic features in the variable memory to form new basic features; the change-related features replace the outdated change features in the memory. This example calculates cosine similarity to index features in the memory that share semantic information with the dynamic target features. This yields a similarity matrix D of size 256×512, where 512 is the size of the variable memory. The following formula calculates cosine similarity, where ||·||2 is the L2 norm, or Euclidean distance:

[0059]

[0060] This example then finds the maximum index of the columns of matrix D. The matrix d stores the position indices of the dynamic target feature pixels and memory pixels with the most similar semantic information. Based on these indices, pixel pairs are constructed. In other words, two pixels from different features in a pixel pair have the most similar semantic information. The size of d is 256×1, and each row represents the index of the pixel in the variable memory with the most similar semantic information for each dynamic target feature pixel. Here, argmax refers to the maximum index, and row represents the row-wise index:

[0061] d = argmax row D (5);

[0062] In this example, the pixel pairs ranked 128 are fused into a new pixel with a fusion ratio of f z :M=1:9. The dynamic target feature pixel in the pixel pair ranked 128 replaces the pixel in the variable memory bank and updates the memory bank.

[0063] (5) A feature enhancement module is constructed based on Transformer to enhance the features of the memory library and search area. Figure 4 This is the flowchart of the feature enhancement module. Specifically, before feature enhancement, this example constructs a convolutional layer to reduce the number of channels of the memory bank and search area features from 1024 to 256. Then, the self-attention mechanism is used to enhance the memory bank and search area features respectively, and the mutual attention mechanism is used to build the relationship between the memory bank and the search area features. Pixel-level matching is based on matrix multiplication. The memory bank M and the search area feature f s Multiply them together to get the target position weight map h, and use a two-layer fully convolutional neural network Fine-tune the number of channels of the target position weight map h. Here, in order to adapt to the pixel-level matching process, the convolution kernel size of the full convolutional network is set to 1:

[0064]

[0065] (6) The head network in the tracking method includes a classification network, an intersection-over-union (IoU) network, and a regression network. These networks are all composed of multi-layer perceptrons, which are used to predict the classification score s of the target and background, predict the IoU ratio i of each bounding box and the true target box, and regress the specific coordinates r of each bounding box.

[0066] (7) Combine the classification score s and the intersection-over-union ratio i to obtain the score, index the position pos of the highest score point in the position weight map, and obtain the coordinate coord of the target box in the regression coordinate r, which can be explained by the following formula, where argmax refers to the maximum value index and τ is the scale factor hyperparameter, which is 0.4 in the implementation:

[0067] pos=argmax(s τ ·i 1-τ ),coord=r[pos] (7).

[0068] Before implementing the tracking process, this example trains a basic framework for video object tracking based on a pixel-level spatiotemporal feature memory. The framework samples four images from any video sequence in the training dataset. Three of these images are cropped to extract the target image based on the annotated bounding box. These images are then fed into a Siamese neural network to extract features and stitch them together, simulating a spatiotemporal feature memory. The remaining image is cropped to extract the search region image based on the annotated bounding box. After a feature enhancement module and pixel-level matching, the target bounding box output by the head network is compared with the annotated bounding box of the search region using a loss function. The classification network and the intersection-over-union (IoU) network use the cross-entropy loss function, while the regression network uses the L1 loss function and the IoU loss function. This process is called a batch. In this example, each iteration consists of 20 batches, and each epoch of parameter learning and updating has 1000 iterations, for a total of 700 training epochs.

[0069] Functions and Effects of the Embodiments

[0070] For example, consider tracking a target in a real-world video. First, the video file is split into several single-frame images. Tracking is performed around the target within these single-frame images. Before tracking, the user or another third party specifies the target. This involves drawing a target reference frame in the initial frame, which provides the target's length, width, and center point.

[0071] When tracking begins, the initial target image is first cropped based on the target reference frame of the initial frame, and the search area is cropped in the second frame based on the target state of the first frame, and input into the twin neural network to extract features. The initial target features will be used to initialize the pixel-level spatiotemporal feature memory bank (hereinafter referred to as the memory bank). Specifically, the initial target features are copied twice and stored in the memory bank together. In the second frame tracking, the memory bank is not updated. Subsequently, the target information in the memory bank is matched with the search area features through the pixel-level matching and fine-tuning module to generate a target position weight map and fine-tune it. The target position weight map is input into the head network module, and the classification network is used to distinguish between the target and the background. The intersection-over-union network predicts the intersection-over-union ratio of each bounding box to the true target box, and the regression network regresses the specific coordinates of each bounding box; in step seven, the score is obtained by combining the classification score and the intersection-over-union ratio, and the position of the highest score point in the position weight map is indexed. The corresponding regression coordinates are found, which are the coordinates of the tracking target bounding box.

[0072] From the third frame onwards, the steps described in the invention will be fully executed. The search area is cut out according to the target state of the previous frame. When the current frame meets the specified frame interval (5 frames), that is, when the memory update interval is met, the dynamic target image is cut out based on the tracking result bounding box of the previous frame, and the dynamic target image and the search area are input into the twin neural network to extract features, and the dynamic target features are input into the memory bank for update. If the current frame does not meet the specified frame interval, only the search area is input into the twin neural network, and the memory bank is not updated.

[0073] As can be seen from the above, the memory update step is optional, and its execution is determined based on the current frame. During the memory update, a pixel-level online prediction module is used to predict the target's basic features and change-related features within the dynamic target feature set, guiding the update process. Based on the predicted and retrieved target basic features and change-related features, the memory update is then integrated with the corresponding basic features in the memory, replacing the change-related features in the memory.

[0074] In the pixel-level matching and fine-tuning module, the target information in the memory bank is matched with the search area features to generate a target position weight map, which is then fine-tuned. This target position weight map is then fed into the head network module. The classification network distinguishes the target from the background, the intersection-over-union (IoU) network predicts the IoU ratio of each bounding box with the true target box, and the regression network regresses the specific coordinates of each bounding box. By combining the classification score and the IoU ratio to obtain a score, the highest-scoring point in the position weight map is indexed and the corresponding regressed coordinates are found, which are the coordinates of the target bounding box.

[0075] The system achieves excellent tracking results in scenarios where the target's appearance changes, including deformation, occlusion, rotation, and changes in video perspective. This improves both adaptability and robustness to changes in target appearance. Furthermore, the tracking speed in real-world videos reaches 28 frames per second (FPS), meeting real-time requirements (i.e., greater than 25 FPS).

[0076] In the commonly used video object tracking dataset LaSOT, this example addresses challenges related to target appearance changes in the dataset, such as deformation, occlusion, rotation, and viewpoint change, and conducts quantitative comparison and analysis with baseline methods based on spatiotemporal memory and Transformer. The quantitative results are shown in the table below:

[0077] Table 1

[0078]

Claims

1. A video target tracking method using a pixel-level spatiotemporal feature memory library, characterized in that: It can store and update the target feature pixels in the time domain, while maintaining the memory size unchanged, fully utilize the historical information of the target in the video time sequence, adapt to the target deformation in the current video frame, and determine the target's position and posture through pixel-level matching in the current video frame to achieve more robust tracking. The specific steps are as follows: Step (1): Based on the target reference frame of the initial frame or the tracking result bounding box of the previous frame, the target is cut out as a dynamic target image; based on the target state of the previous frame, the search area is cut out in the current frame; Step (2): Use the twin neural network to extract features from the dynamic target image and the search area, and map them to the same feature space; the initial frame target features construct an initial pixel-level spatiotemporal feature memory bank, referred to as the memory bank, and the features of the dynamic target image are used to update the memory bank; the features of the dynamic target image are referred to as dynamic target features; Step (3): Use the pixel-level online prediction module to predict the basic features and change-related features of the dynamic target features to guide the update process; Step (4): Based on the basic features and change-related features of the target retrieved in step (3), a pixel-level update strategy is designed to fuse the basic features with the basic features in the memory library to form new basic features; and the change-related features are used to replace the outdated change features in the memory library; Step 5: Enhance the memory and search area features, then use the pixel-level matching and fine-tuning module to match the target information in the memory with the search area features, generate a target position weight map, and fine-tune it. Step (6): Input the target position weight map into the head network module, use the classification network to distinguish the target from the background, use the intersection-over-union network to predict the intersection-over-union ratio of each bounding box with the true target box, and use the regression network to regress the specific coordinates of each bounding box; Step (7): Combine the classification score and the score obtained by intersection and union, index the position of the point with the highest score in the position weight map, and find the corresponding regression coordinates, which are the coordinates of the bounding box of the tracking target; The step (3) of "using a pixel-level online prediction module to predict the basic features and change-related features of the target in the dynamic target features to guide the update process" specifically inputs the dynamic image features into the pixel-level online prediction module; the prediction module is a dynamic filter F, which calculates the correlation between the dynamic image features and the target features in the memory library to achieve the prediction function; based on the property that convolution in the time domain is equal to the product in the frequency domain, the memory library M and the dynamic target features f are firstly z Perform Fourier transform and get M * and Then, the memory bank M * Calculate the matrix Hadamard product with the dynamic filter F to obtain the response map R; the response map R here is obtained by fitting the dynamic target feature To achieve the prediction function, the specific formula is as follows, where ⊙ is the Hadamard product, that is, the dot product of the corresponding elements of the matrix: R=M * ⊙F (1) In order to ensure the fitting effect of the response graph R, online learning is used to minimize the dynamic target features after the response graph R and Fourier transform. The squared error loss is used to improve the prediction accuracy. The specific formula is as follows, where n is the n units in the memory bank: Subsequently, the Wirtinger derivative is used to solve the optimization problem of formula (2), and the general expression of the dynamic filter F is obtained. The specific formula is as follows, where η is the update weight hyperparameter and F′ is the filter before the update: When tracking each frame, repeat step (3) so that the response map R and the dynamic target feature A one-to-one correspondence is formed; therefore, the response score in the response graph R corresponds to the correlation between the dynamic target feature and the memory bank, that is, a higher score corresponds to the basic feature in the dynamic target feature, and conversely, a lower score corresponds to the change-related feature in the dynamic target feature. The pixels in the dynamic target feature are sorted according to the score to reconstruct the dynamic target feature sequence.

2. The video target tracking method according to claim 1, characterized in that: In step (1), based on the property that the displacement and deformation of the target do not change drastically in consecutive frames, the search area of ​​the current frame is determined with reference to the target state of the previous frame. At the same time, the cropped dynamic target image and search area are adjusted to a fixed size to facilitate the design of the tracking network structure.

3. The video target tracking method according to claim 2, characterized in that: In step (2), the twin neural network is used to extract features from the dynamic target image and the search area, and the two are mapped to the same feature space, that is, the two are input into a network with the same network structure and weights; The features of the initial frame target image are used to construct the initial pixel-level spatiotemporal feature memory library, and the features of the dynamic target image are used to update the memory library.

4. The video target tracking method according to claim 3, characterized in that: The step (iv) described in "designing a pixel-level update strategy based on the basic features and change-related features of the target retrieved in step (iii) to correspondingly fuse the basic features with the basic features in the memory bank to form new basic features; and replacing the outdated change features in the memory bank with the change-related features" includes designing a "fusion-replacement" update strategy. Specifically, after step (iii), a reconstructed dynamic target feature sequence is obtained, which has been sorted according to the correlation with the variable features in the memory bank, also known as the "variable memory bank"; this step (iv) aims to fuse the top-ranked dynamic target feature pixels with the pixels in the variable memory bank to form new pixels, and at the same time replace the bottom-ranked dynamic target feature pixels with the pixels in the variable memory bank to update the memory bank; therefore, a pixel pair indexing module based on cosine similarity is designed to index the corresponding dynamic target feature pixels and the pixels in the variable memory bank for the update process to form pixel pairs; specifically, this step first calculates the reconstructed dynamic target feature sequence f z The cosine similarity matrix D with the memory library M, where ||·||2 is the L2 norm or Euclidean distance: Then, the maximum index of the columns of matrix D is calculated to form d. Here, d stores the position index of the dynamic target feature pixel and the memory pixel with the closest semantic information. Pixel pairs are constructed based on the index. That is, the two pixels in the pixel pair, although from different features, have the closest semantic information. The specific formula for this step is as follows: d=argmax row D(5) Among them, argmax refers to the maximum value index, and row represents the row index; Based on the obtained pixel pairs, the top half of the pixel pairs are fused into a new pixel, and the dynamic target feature pixels in the bottom half of the pixel pairs are used to replace the pixels in the memory bank to update the memory bank.

5. The video target tracking method according to claim 4, characterized in that: The step (5) of "enhancing the features of the memory bank and the search area, and then matching the target information in the memory bank with the search area features through the pixel-level matching and fine-tuning module to generate a target position weight map and fine-tune it" includes using a Transformer-based feature enhancement module to enhance the features of the memory bank and the search area. Specifically, the self-attention mechanism is used to enhance the features of the memory bank and the search area respectively, and the mutual attention mechanism is used to build the relationship between the features of the memory bank and the search area. The pixel-level matching operation is based on matrix multiplication, specifically, the memory bank M is matched with the search area feature f s Multiply them together to get the target position weight map h; then, use a two-layer fully convolutional neural network Fine-tune the number of channels of the target position weight map h. Here, in order to adapt to the pixel-level matching process, the convolution kernel size of the full convolutional network is set to 1. The formula for this step is as follows: For a fully convolutional neural network:

6. The video target tracking method according to claim 5, characterized in that: In step (six), "the target position weight map is input into the head network module, the classification network is used to distinguish the target from the background, the intersection-over-union (IoU) network is used to predict the IoU ratio of each bounding box with the true target box, and the regression network is used to regress the specific coordinates of each bounding box". The head network includes a classification network, an IoU network, and a regression network. These networks are all composed of a multi-layer perceptron. These three networks are used to predict the classification score s of the target and the background, predict the IoU ratio i of each bounding box with the true target box, and regress the specific coordinates r of each bounding box.

7. The video target tracking method according to claim 6, characterized in that: The specific process of "combining the classification score and the score obtained by the intersection-and-union ratio to index the position of the highest-scoring point in the position weight map and find the corresponding regression coordinates, which are the coordinates of the tracking target bounding box" in step (seven) is to combine the classification score s and the intersection-and-union ratio i to obtain the score, index the position of the highest-scoring point pos in the position weight map, and obtain the coordinates coord of the target box in the regression coordinate r, which is expressed by the following formula: pos=argmax(s τ ·i 1-τ ),coord=r[pos] (7) Here, argmax refers to the maximum value index and τ is the scaling factor hyperparameter.

Images