Video tracking method, device, terminal and medium

Abstract

The present invention discloses a video tracking method, device, terminal, and medium. The present invention combines the semantic features of the target tracking point itself with the surrounding contextual features, improving the ability to query spatial information. This allows the correct spatial features of the target tracking point to be found even in long videos or when the tracking target undergoes significant changes. Point tracking tasks in long videos are accomplished by forming a point query based on the semantic features, contextual features, and point positions of the target tracking point, and updating the point query frame by frame. The method is applicable to fields such as video editing, augmented reality, 3D reconstruction, and optical flow estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video tracking method, device, terminal and medium. Background Art

[0002] Object tracking is a key task in computer vision. Its goal is to accurately track the motion of a specific target within a continuous image sequence. Point tracking is a common object tracking method. This method detects and tracks feature points within the target area. Feature points are pixels with unique properties in an image, such as corners and edges. By tracking the positional changes of feature points between consecutive frames, the target's trajectory is obtained.

[0003] Current point tracking algorithms rely heavily on single-point features for spatial perception, which is overly limited. When processing long videos and subjecting the target to significant changes, it becomes difficult to retrieve the correct spatial features, resulting in poor tracking results.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a video tracking method, device, terminal and medium in response to the above-mentioned defects of the prior art, aiming to solve the problem that the point tracking algorithm in the prior art mainly relies on the single-point features of the tracking point in spatial perception. When processing long videos or when the appearance of the tracking target changes significantly, it is difficult to query the correct spatial features, resulting in poor tracking effect.

[0006] The technical solutions adopted by the present invention to solve the problem are as follows:

[0007] In a first aspect, an embodiment of the present invention provides a video tracking method, the method comprising:

[0008] Determine the target tracking point and the initial video frame from the target video;

[0009] A point query for the target tracking point is constructed according to the initial video frame, and the point query for the target tracking point is updated based on depth image features of a tracking video frame subsequent to the initial video frame to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point.

[0010] In one embodiment, constructing a point query for the target tracking point based on the initial video frame includes:

[0011] Extracting initial depth image features of the initial video frame through a backbone network;

[0012] Obtaining the point position of the target tracking point in the initial video frame;

[0013] Constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position;

[0014] A point query of the target tracking point is obtained based on the point position, the semantic feature and the context feature.

[0015] In one embodiment, constructing semantic features and context features of the target tracking point based on the depth image features and the point position includes:

[0016] If the depth image feature of the initial video frame is single-scale, performing a first sampling based on the single-scale depth image feature and the point position to obtain a semantic feature of the target tracking point;

[0017] A second sampling is performed based on the single-scale depth image feature and the point position to obtain a context feature of the target tracking point.

[0018] In one embodiment, constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position includes:

[0019] If the depth image feature of the initial video frame is multi-scale, for each single-scale depth image feature in the multi-scale depth image feature, perform a first sampling on each single-scale depth image feature based on the point position to obtain multiple initial semantic features of the target tracking point;

[0020] Performing a second sampling on each of the single-scale depth image features based on the point position to obtain a plurality of initial context features of the target tracking point;

[0021] The multiple initial semantic features and the multiple initial context features are fused respectively by a multi-layer perceptron to obtain the semantic features of the target tracking point and the context features of the target tracking point.

[0022] In one embodiment, updating the point query of the target tracking point based on the depth image features of the tracking video frame after the initial video frame includes:

[0023] Extract the depth image features of the current tracking video frame;

[0024] Determining an optimized point position and optimized semantic features of the target tracking point in the current tracking video frame based on the depth image features of the current tracking video frame and a point query of the target tracking point in the previous tracking video frame;

[0025] The point query of the target tracking point is updated based on the optimized point position.

[0026] In one embodiment, determining the optimized point position and optimized semantic features of the target tracking point in the current tracking video frame based on the depth image features of the current tracking video frame and the point query of the target tracking point in the previous tracking video frame includes:

[0027] Obtaining, based on a point query of the target tracking point in an initial video frame, a depth image feature of the initial video frame, and visibility of a preset initial video frame, an optimized semantic feature and an optimized point position of the target tracking video in a second tracking video frame, and visibility of the second tracking video frame;

[0028] For each tracking video frame after the second tracking video frame, the semantic features and point position of the target tracking point in the current tracking video frame are updated based on the point query of the target tracking point in the current tracking video frame, the depth image features of the current tracking video frame, and the optimized semantic features and visibility of the target tracking point in all historical tracking video frames before the current tracking video frame, to obtain the optimized semantic features and optimized point position of the target tracking point in the current tracking video frame;

[0029] updating the point query in the current tracking video frame according to the optimized point position of the target tracking point in the current tracking video frame;

[0030] The visibility of the target tracking point in the current tracking video frame is calculated according to the optimized semantic features of the target tracking point in the current tracking video frame.

[0031] In one embodiment, obtaining optimized semantic features and optimized point positions in a second tracking video frame and visibility of the second tracking video frame based on a point query of the target tracking point in an initial video frame, a depth image feature of the initial video frame, and visibility of a preset initial video frame includes:

[0032] Optimizing the semantic features and point positions in the initial video frame according to the point query of the target tracking point in the initial video frame and the depth image features of the initial video frame to obtain optimized semantic features and optimized point positions of the target tracking point in the initial video frame;

[0033] updating the point query in the initial video frame according to the optimized point position in the initial video frame to obtain a point query in a second tracking video frame;

[0034] updating the semantic features and point positions of the target tracking point in the second tracking video frame according to the point query of the second tracking video frame, the depth image features of the second tracking video frame, the optimized semantic features in the initial video frame, and the visibility of the preset initial video frame to obtain the optimized semantic features and optimized point positions in the second tracking video frame;

[0035] updating the point query of the second tracking video frame according to the optimized point position in the second tracking video frame;

[0036] The visibility of the target tracking point in the second tracking video frame is calculated according to the optimized semantic features in the second tracking video frame.

[0037] In one embodiment, the method for updating the semantic features of the current tracking video frame includes:

[0038] For each of the historical tracking video frames, calculating an attention weight according to the optimized semantic features of the historical tracking video frame and the semantic features of the current tracking video frame;

[0039] Reweighting the attention weight according to the visibility of the historical tracking video frame to obtain a reweighted attention weight of the historical tracking video frame;

[0040] According to the reweighted attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused to obtain a temporal feature query result;

[0041] The semantic features of the current tracking video frame are updated according to the temporal feature query result.

[0042] In one embodiment, weighted fusion of the optimized semantic features of each of the historical tracking video frames is performed based on the reweighted attention weights of each of the historical tracking video frames, including:

[0043] For each of the historical tracking video frames, obtaining a frame position relationship between the historical tracking video frame and the current tracking video frame;

[0044] updating the reweighted attention weight of the historical tracking video frame according to the frame position relationship to obtain an updated attention weight of the historical tracking video frame;

[0045] According to the updated attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused.

[0046] In one embodiment, the method for updating the point position in the current tracking video frame includes:

[0047] The semantic features of the current tracking video frame are generated into a plurality of sampling offsets by a multi-layer perceptron;

[0048] For each of the sampling offsets, generating a sampling point according to the sampling offset and the point position in the current tracking video frame;

[0049] Sampling is performed according to the depth image features of the current tracking video frame and the position of the sampling point to obtain semantic features and context features of the sampling point;

[0050] Calculating the offset attention weight of the sampling point based on the contextual features in the current tracking video frame and the contextual features of the sampling point;

[0051] Perform weighted fusion on the sampling offsets according to the offset attention weights of the sampling points to obtain a position update result;

[0052] The position of the point in the current tracking video frame is updated based on the position update result.

[0053] In one embodiment, sampling is performed according to the depth image features of the current tracking video frame and the position of the sampling point to obtain the context features of the sampling point, including:

[0054] If the depth image feature of the current tracking video frame is multi-scale, determining a target feature map from the depth image feature of the current tracking video frame according to the scale to which the sampling point belongs;

[0055] A second sampling is performed according to the target feature map and the position of the sampling point to obtain a context feature of the sampling point.

[0056] In one embodiment, calculating the offset attention weight of the sampling point based on the contextual features of the current tracking video frame and the contextual features of the sampling point includes:

[0057] Calculating a first similarity between each feature in the context features of the current point tracking video frame and each feature in the context features of the sampling point;

[0058] Calculating the second similarity of the sampling points through a multi-layer perceptron according to all the calculated first similarities;

[0059] Calculate the offset attention weight of the sampling point according to the second similarity of the sampling point.

[0060] In one embodiment, the method for updating the semantic features of the current tracking video frame further includes:

[0061] Perform weighted fusion on the semantic features of each sampling point according to the offset attention weight of each sampling point to obtain a spatial feature query result;

[0062] The semantic features of the current tracking video frame are updated according to the spatial feature query result.

[0063] In one embodiment, the method further comprises:

[0064] If a cut occurs in the current tracking video frame, a global matching operation is activated in the current tracking video frame to obtain a global matching point;

[0065] The point position in the point query is replaced according to the position of the global matching point.

[0066] In one embodiment, activating a global matching operation in the current tracking video frame to obtain a global matching point includes:

[0067] Using the depth image features of the current tracking video frame as a matching feature map;

[0068] Calculating a similarity map based on each feature in the context features of the current tracking video frame and the matching feature map;

[0069] The similarity graphs of the features in the context features of the current tracking video frame are fused into a target similarity graph, and a global matching point is determined according to the target similarity graph.

[0070] In one embodiment, activating a global matching operation in the current tracking video frame to obtain a global matching point further includes:

[0071] If the depth image feature of the current tracking video frame is multi-scale, then for each single-scale feature map, multiple similarity maps are calculated based on each feature in the context feature of the current tracking video frame and each single-scale feature map;

[0072] fusing the multiple similarity graphs into a target similarity graph;

[0073] A global matching point is determined based on the target similarity graph.

[0074] In a second aspect, an embodiment of the present invention further provides a video tracking device, the device comprising:

[0075] A determination module, used to determine the target tracking point and the initial video frame from the target video;

[0076] A tracking module is configured to construct a point query for the target tracking point based on the initial video frame, and to update the point query for the target tracking point based on the depth image features of a tracking video frame subsequent to the initial video frame, so as to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point.

[0077] In a third aspect, an embodiment of the present invention further provides a terminal comprising a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing any of the video tracking methods described above; and the processor is used to execute the programs.

[0078] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-described video tracking methods.

[0079] Beneficial Effects of the Invention: The embodiments of the present invention combine the semantic features of the target tracking point itself with the surrounding contextual features, improving the ability to query spatial information. This allows the correct spatial features of the target tracking point to be found even in long videos or when the tracking target undergoes significant changes. Point tracking tasks in long videos are accomplished by forming a point query based on the semantic features, contextual features, and point position of the target tracking point, and updating the point query frame by frame. This approach is applicable to fields such as video editing, augmented reality, 3D reconstruction, and optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0081] Figure 1 4 is a flow chart of a video tracking method provided by an embodiment of the present invention.

[0082] Figure 2 This is an overall logical framework diagram of the video tracking method provided by an embodiment of the present invention.

[0083] Figure 3 This is a logical diagram of semantic feature updating provided by an embodiment of the present invention.

[0084] Figure 4 This is a logical diagram of semantic features and point position updates provided by an embodiment of the present invention.

[0085] Figure 5 It is a schematic diagram of the second similarity calculation provided by an embodiment of the present invention.

[0086] Figure 6 This is the experimental result provided by the embodiment of the present invention Figure 1 .

[0087] Figure 7 This is the experimental result provided by the embodiment of the present invention Figure 2 .

[0088] Figure 8 4 is a schematic diagram of a module of a video tracking device provided by an embodiment of the present invention.

[0089] Figure 9 This is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The present invention discloses a video tracking method, device, terminal, and medium. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0091] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0092] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0093] In response to the above-mentioned shortcomings of the prior art, the present invention provides a video tracking method, comprising: determining a target tracking point and an initial video frame from a target video; constructing a point query for the target tracking point based on the initial video frame; and updating the point query for the target tracking point based on the depth image features of a tracking video frame subsequent to the initial video frame to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video containing the target tracking point. The present invention combines the semantic features of the target tracking point itself with the surrounding contextual features to improve the query capability for spatial information, so that the correct spatial features of the target tracking point can still be found in long videos or when the tracking target undergoes significant changes. The point query is composed of the semantic features, contextual features, and point positions of the target tracking point, and the point tracking task in long videos is achieved by updating the point query frame by frame. The method is applicable to fields such as video editing, augmented reality, 3D reconstruction, and optical flow estimation.

[0094] like Figure 1 As shown, the method specifically includes:

[0095] Step S100: Determine target tracking points and initial video frames from the target video.

[0096] Specifically, the target video is the video on which the tracking task is to be performed, and can be a long video with a number of frames greater than a preset value. The target tracking point is the tracked point, and there is at least one target tracking point, which can be any point in any frame of the video. Any movement of the target tracking point in the tracking video frame will be tracked. In practical applications, the target video is first converted into a tracking video frame I in the form of an image, and any frame is I i . It should be noted that the movement here refers to the absolute position movement of the video interface, not the relative position movement in the video. The initial video frame is the video frame where the user specifies the target tracking point, that is, the initial video frame that starts the tracking task, denoted as I0, and there is at least one initial video frame. The user can specify one or more target tracking points. Multiple target tracking points can be located in the same tracking video frame or different tracking video frames. The tracking method of each target tracking point is the same, that is, there can be one or more target tracking points, and the initial video frame contains one video frame or multiple video frames. The user can select a point on any frame in the target video as the target tracking point, for example, by clicking the mouse, touching the screen, etc. The initial point position l0 of the target tracking point can be expressed by two-dimensional coordinates.

[0097] Step S200: construct a point query for the target tracking point based on the initial video frame, and update the point query for the target tracking point based on the depth image features of the tracking video frame after the initial video frame to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point.

[0098] Specifically, the algorithm flow of this embodiment is mainly divided into two stages. The first stage is to construct a point query for the target tracking point in the initial video frame. The depth image features of the initial video frame I0 are extracted. Based on the point position l0 of the target tracking point, features are sampled on the depth image features to describe the target tracking point's intrinsic information, thereby obtaining the semantic feature f. A set of features are then sampled around the point position on the depth image features to describe the contextual information of the target tracking point, thereby obtaining the contextual feature C. The constructed point query is {f, l0, C}, which will be used to initialize the point query for the target tracking point in the next tracking video frame I1. The second stage is the frame-by-frame point tracking process. During the tracking process, when new video frames are received, these video frames become tracking video frames. Each tracking video frame inherits the point query passed by the previous tracking video frame. The point query is updated using the point query and the depth image features of the current tracking video frame to achieve continuous tracking of the target tracking point.

[0099] In one implementation, constructing a point query of the target tracking point based on the initial video frame includes:

[0100] Extracting initial depth image features of the initial video frame through a backbone network;

[0101] Obtaining the point position of the target tracking point in the initial video frame;

[0102] Constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position;

[0103] A point query of the target tracking point is obtained based on the point position, the semantic feature and the context feature.

[0104] Specifically, this embodiment pre-builds a backbone network for extracting 2D depth image features. Once the user specifies a target tracking point, the initial video frame that generates the target tracking point is fed into the backbone network, which then extracts the depth image features of the initial video frame. The semantic features of the target tracking point and a set of contextual features are then extracted from the depth image features based on the target tracking point's location, thereby initializing a point query for the target tracking point.

[0105] For example, this embodiment supports tracking from any frame. Assuming that tracking starts from frame 0, the initial video frame I0 is input into the backbone network to obtain the depth image feature X0 corresponding to the initial video frame I0. The size of the single-layer depth image feature is H×W×D, where H is the image feature height, W is the image feature width, and D is the feature dimension. The semantic feature f∈R of the target tracking point is extracted on the depth image feature X0. D and a set of contextual features N 2 is the number of features in a set of context features, thereby initializing the point query of the target tracking point. At the same time, each frame image also supports the use of multi-scale image features, and the multi-layer depth image features are H1×W1×D, H2×W2×D, etc. Preferably, a four-layer feature can be used (H1×W1×D, H2×W2×D, H3×W3×D, H4×W4×D, where H1, H2, H3, and H4 are the image feature heights of each layer in the four layers, and W1, W2, W3, and W4 are the image feature widths of each layer in the four layers).

[0106] In one implementation, constructing semantic features and context features of the target tracking point based on the depth image features and the point position includes:

[0107] If the depth image feature of the initial video frame is single-scale, performing a first sampling based on the single-scale depth image feature and the point position to obtain a semantic feature of the target tracking point;

[0108] A second sampling is performed based on the single-scale depth image feature and the point position to obtain a context feature of the target tracking point.

[0109] Specifically, when the extracted depth image features of the initial video frame are of a single scale, a first sampling is performed on the depth image feature map X0 of the initial video frame according to the initial point position l0 of the target tracking point to obtain a feature vector of dimension D, which is used as the semantic feature f of the target tracking point. In addition, in order to obtain richer contextual information, a second sampling is performed near the initial point position l0 to obtain the contextual feature C of the target tracking point. The contextual feature C is a set of features, and these features have a certain positional relationship. The first sampling and the second sampling are two different sampling methods. This embodiment does not specifically limit the sampling method. A suitable sampling method can be selected according to user needs or tracking tasks. For example, the first sampling can be bilinear interpolation or region of interest pooling (ROI-Pooling), and the second sampling can be grid sampling. The grid sampling obtains a set of features distributed according to the grid points.

[0110] In another implementation, constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position includes:

[0111] If the depth image feature of the initial video frame is multi-scale, for each single-scale depth image feature in the multi-scale depth image feature, perform a first sampling on each single-scale depth image feature based on the point position to obtain multiple initial semantic features of the target tracking point;

[0112] Performing a second sampling on each of the single-scale depth image features based on the point position to obtain a plurality of initial context features of the target tracking point;

[0113] The multiple initial semantic features and the multiple initial context features are fused respectively by a multi-layer perceptron to obtain the semantic features of the target tracking point and the context features of the target tracking point.

[0114] Specifically, when the extracted depth image features of the initial video frame are multi-scale (multi-layer), the processing process of the image features at each scale can refer to the single-scale depth image features. In this embodiment, the semantic features sampled at each scale are defined as initial semantic features, and the initial semantic features sampled at different scales are fused through a multi-layer perceptron (MLP) to obtain the semantic features f of the target tracking point. In this embodiment, the context features sampled at each scale are defined as initial context features, and the initial context features sampled at different scales are fused through a multi-layer perceptron according to position correspondence to obtain the context features C of the target tracking point.

[0115] In one implementation, updating the point query of the target tracking point based on the depth image features of the tracking video frame subsequent to the initial video frame includes:

[0116] Extract the depth image features of the current tracking video frame;

[0117] Determining an optimized point position and optimized semantic features of the target tracking point in the current tracking video frame based on the depth image features of the current tracking video frame and a point query of the target tracking point in the previous tracking video frame;

[0118] The point query of the target tracking point is updated based on the optimized point position.

[0119] Specifically, during the tracking process, the new tracking video frame currently received is defined as the current tracking video frame. The current tracking video frame is input into the backbone network, and the depth image features are extracted through the backbone network. The current tracking video frame will inherit the point query passed by the previous tracking video frame. Through the point query and the depth image features of the current tracking video frame, the target tracking point is detected in the current tracking video frame. In order to further improve the detection effect, this embodiment will optimize the point position and semantic features of the target tracking point in the current tracking video frame to obtain the correct position and correct spatial information of the target tracking point, thereby improving the detection accuracy. The optimized point position is used to replace the point position in the point query, and the features in the point query are kept unchanged to reduce feature drift, while avoiding large computational overhead through frame-by-frame displacement updates.

[0120] For example, let's take tracking video frame l1 as the current tracking video frame. After obtaining the point query {f, l0, C} in the first stage, the second stage will use the point query and the depth image feature X1 of tracking video frame l1 to obtain the optimized point position l1 and optimized semantic feature f1 of the target tracking point in that frame. l1 is used to update the point position in the point query, resulting in the updated point query {f, l1, C}.

[0121] In one implementation, determining an optimized point position and optimized semantic features of the target tracking point in the current tracking video frame based on the depth image features of the current tracking video frame and a point query of the target tracking point in the previous tracking video frame includes:

[0122] Obtaining, based on a point query of the target tracking point in an initial video frame, a depth image feature of the initial video frame, and visibility of a preset initial video frame, an optimized semantic feature and an optimized point position of the target tracking video in a second tracking video frame, and visibility of the second tracking video frame;

[0123] For each tracking video frame after the second tracking video frame, the semantic features and point position of the target tracking point in the current tracking video frame are updated based on the point query of the target tracking point in the current tracking video frame, the depth image features of the current tracking video frame, and the optimized semantic features and visibility of the target tracking point in all historical tracking video frames before the current tracking video frame, to obtain the optimized semantic features and optimized point position of the target tracking point in the current tracking video frame;

[0124] updating the point query in the current tracking video frame according to the optimized point position of the target tracking point in the current tracking video frame;

[0125] The visibility of the target tracking point in the current tracking video frame is calculated according to the optimized semantic features of the target tracking point in the current tracking video frame.

[0126] In summary, for each tracking video frame, the processing steps typically include optimizing the semantic features and position of the target tracking point, and determining the visibility of the target tracking point in the frame based on the optimized semantic features. The visibility estimate is used to reflect whether the target tracking point is occluded. In this embodiment, the initial video frame is the video frame that initiates tracking, and the second tracking video frame is the frame following the initial video frame. The processing methods for the initial video frame and the second tracking video frame differ from those for other tracking video frames.

[0127] Specifically, since the initial video frame is the one that initiates tracking, the visibility of the target tracking point in the initial video frame does not need to be calculated and a preset value can be used. For the second tracking video frame, the optimized semantic features and optimized point positions of the second tracking video frame are obtained by using the point query inherited from the initial video frame, its own depth image features, and the preset visibility of the initial video frame. The visibility of the target tracking point in the second tracking video frame is then determined based on the optimized semantic features.

[0128] Each tracking video frame after the second is processed in the same manner. For example, a single tracking video frame is defined as the current tracking video frame. The semantic features and point positions of the target tracking point in the current tracking video frame are optimized using the point query inherited from the previous tracking video frame, the depth image features of the current tracking video frame, and the optimized semantic features and visibility of the target tracking point in all historical tracking video frames prior to the current tracking video frame. The visibility of the target tracking point in the current tracking video frame is then determined based on the optimized semantic features.

[0129] In one implementation, obtaining optimized semantic features and optimized point positions in a second tracking video frame and visibility of the second tracking video frame based on a point query of the target tracking point in an initial video frame, a depth image feature of the initial video frame, and visibility of a preset initial video frame includes:

[0130] Optimizing the semantic features and point positions in the initial video frame according to the point query of the target tracking point in the initial video frame and the depth image features of the initial video frame to obtain optimized semantic features and optimized point positions of the target tracking point in the initial video frame;

[0131] updating the point query in the initial video frame according to the optimized point position in the initial video frame to obtain a point query in a second tracking video frame;

[0132] updating the semantic features and point positions of the target tracking point in the second tracking video frame according to the point query of the second tracking video frame, the depth image features of the second tracking video frame, the optimized semantic features in the initial video frame, and the visibility of the preset initial video frame to obtain the optimized semantic features and optimized point positions in the second tracking video frame;

[0133] updating the point query of the second tracking video frame according to the optimized point position in the second tracking video frame;

[0134] The visibility of the target tracking point in the second tracking video frame is calculated according to the optimized semantic features in the second tracking video frame.

[0135] Specifically, after constructing a point query using the initial video frame, this embodiment optimizes the semantic features and point positions in the initial video frame based on the constructed point query and its own depth image features, obtaining the optimized semantic features and optimized point positions in the initial video frame. The constructed point query is then updated using the optimized point positions. When a second tracking video frame is detected, the updated point query is passed to the second tracking video frame. The second tracking video frame updates the semantic features and point positions of the second tracking video frame based on the inherited point query, its own depth image features, the optimized semantic features of the target tracking point in the initial video frame, and the preset visibility of the initial video frame, obtaining the optimized semantic features and optimized point positions of the target tracking point in the second tracking video frame. Similar to the initial video frame, the point query is updated using the optimized point position of the target tracking point in the second tracking video frame. However, starting from the second tracking video frame, the visibility of the target tracking point in that frame is determined based on the optimized semantic features of that frame.

[0136] In one implementation, the method for updating the semantic features of the current tracking video frame includes:

[0137] For each of the historical tracking video frames, calculating an attention weight according to the optimized semantic features of the historical tracking video frame and the semantic features of the current tracking video frame;

[0138] Reweighting the attention weight according to the visibility of the historical tracking video frame to obtain a reweighted attention weight of the historical tracking video frame;

[0139] According to the reweighted attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused to obtain a temporal feature query result;

[0140] The semantic features of the current tracking video frame are updated according to the temporal feature query result.

[0141] Specifically, this embodiment takes into account the visibility of the long-term attention mechanism (VLTA) when updating the semantic features. Figure 3 As shown in the figure, in order to enable the point query to obtain the information in the past tracking video frames, the optimized semantic features {f1, f2...f t-1}Calculate attention. Taking into account that the target tracking point may disappear in the video, the optimized semantic features obtained on the historical tracking video frames where the target tracking point disappears inevitably contain noise. Therefore, this embodiment uses the visibility of the target tracking point calculated in the historical tracking video frames to reweight the attention to obtain reweighted attention. The higher the reweighted attention, the more important the optimized semantic features of the historical tracking video frame are. Finally, a new feature is obtained by weighted fusion through the reweighted attention and optimized semantic features of each historical tracking video frame, that is, the time series feature query result. Finally, the semantic features of the current tracking video frame are updated through the time series feature query result, thereby improving the information query capability of the time dimension and reducing feature drift. In actual application, the implementation method of weighted fusion can be point multiplication and softmax.

[0142] In one implementation, weighted fusion of the optimized semantic features of each of the historical tracking video frames is performed based on the reweighted attention weights of each of the historical tracking video frames, including:

[0143] For each of the historical tracking video frames, obtaining a frame position relationship between the historical tracking video frame and the current tracking video frame;

[0144] updating the reweighted attention weight of the historical tracking video frame according to the frame position relationship to obtain an updated attention weight of the historical tracking video frame;

[0145] According to the updated attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused.

[0146] Specifically, given that historical tracking frames closer to the current tracking frame have greater reference value, this embodiment further updates the attention of each historical tracking frame based on its positional relationship with the current tracking frame, thereby obtaining an updated attention weight. Finally, a weighted fusion approach is used to obtain the temporal feature query result by combining the updated attention weights of each historical tracking frame with the optimized semantic features.

[0147] For example, each time in the history tracking video frame optimized semantic features f i Add the rotation position encoding to update f iBy performing dot product and softmax, the rotation position encoding of each frame can reflect the frame position relationship between the frame and the current tracking video frame, and further reflect the proximity of the frame to the current tracking video frame, so that the attention is more focused on the adjacent frames.

[0148] In one implementation, a method for updating a point position in a current tracking video frame includes:

[0149] The semantic features of the current tracking video frame are generated into a plurality of sampling offsets by a multi-layer perceptron;

[0150] For each of the sampling offsets, generating a sampling point according to the sampling offset and the point position in the current tracking video frame;

[0151] Sampling is performed according to the depth image features of the current tracking video frame and the position of the sampling point to obtain semantic features and context features of the sampling point;

[0152] Calculating the offset attention weight of the sampling point based on the contextual features in the current tracking video frame and the contextual features of the sampling point;

[0153] Perform weighted fusion on the sampling offsets according to the offset attention weights of the sampling points to obtain a position update result;

[0154] The position of the point in the current tracking video frame is updated based on the position update result.

[0155] In summary, this embodiment takes into account the contextual mutual attention mechanism (CCA) and the attention-based position update mechanism (APU) when updating the point position. Figure 4As shown, in traditional mutual attention mechanisms, attention is typically calculated using the semantic features f of the point query. However, since the semantic features f of the point query are all point-level features, the calculated attention can be disrupted when there are repetitive patterns in the image or when the tracking target undergoes significant appearance changes. Therefore, this embodiment proposes using spatial context to assist in attention calculation, thereby obtaining a more robust attention weight. Specifically, a multi-layer perceptron is used to adaptively generate multiple sampling offsets from the semantic features of the current tracking video frame. A sampling point is then generated based on each sampling offset combined with the point position in the current tracking video frame. Similar to the target tracking point, each sampling point also needs to sample point-level features and contextual features based on the depth image features of the current tracking video frame. By comparing the similarity between the contextual features of the current tracking video frame and the contextual features of the sampling point, the offset attention weight of the point is calculated. A higher offset attention weight indicates that the point is more similar to the target tracking point. Finally, the offset attention weights of each sampling point are used to weightedly fuse the sampling offsets to obtain a position update result, thereby accurately achieving position update.

[0156] For example, according to the semantic feature f of the target tracking point in the current tracking video frame t 'Adaptively generate M sampling offsets O through a multilayer perceptron t ∈R M×2 , where 2 represents the coordinates x,y of the sampling point. Based on the sampling offset O t and l' t M sampling points can be obtained, l' t is the point position of the target tracking point in the current tracking video frame, for example, the sampling offset O t and l' t The sampling points are obtained by adding them together. Based on the M sampling points, M semantic features V are obtained by the first sampling on the depth image features of the current tracking video frame. t ∈R M×D , and obtain M context features through the second sampling The size of each context feature is N 2 ×D. Taking a sampling point as an example, when the depth image feature has only one scale, for the mth sampling position among M sampling points In the depth image feature X t The semantic features of the sampling point are sampled by bilinear interpolation method. And in the depth image feature X t N samples are sampled in a grid pattern near the sampling position. 2 features, which constitute the context features of the sampling point (like Figure 5Then, the context features of the query point in the current tracking video frame are used and M context features K t , calculate the more robust offset attention weight W∈R M , to help the mutual attention (CCA) sample more helpful features from the image and help the attention-based position update (APU) get a more accurate position update. The offset attention weight W will be used to weight and sum the sample offset O t , thus obtaining a position update Δl of the current tracking video frame t ∈R 2 , and then update the point position l t '.

[0157] In one implementation, sampling is performed according to the depth image features of the current tracking video frame and the position of the sampling point to obtain the context features of the sampling point, including:

[0158] If the depth image feature of the current tracking video frame is multi-scale, determining a target feature map from the depth image feature of the current tracking video frame according to the scale to which the sampling point belongs;

[0159] A second sampling is performed according to the target feature map and the position of the sampling point to obtain a context feature of the sampling point.

[0160] Specifically, if the depth image feature of the current tracking video frame is multi-scale, it means that the depth image feature is composed of multiple feature maps of different scales. Since the location of each sampling point has its own scale, when the depth image feature is multi-scale, the context feature of each sampling point only needs to be sampled from the feature map corresponding to the scale. Taking a sampling point as an example, the feature map corresponding to the scale of the sampling point among the multiple feature maps of different scales is used as the target feature map. It is only necessary to perform a second sampling on the target feature map according to the sampling point position to obtain the context feature of the sampling point.

[0161] Furthermore, if the depth image features of the current tracking video frame are multi-scale, the method for obtaining the semantic features of each sampling point is the same as the method for obtaining the semantic features of the target tracking point in the same situation: the semantic features obtained by the first sampling based on the position of the sampling point at each scale are defined as the initial semantic features. The initial semantic features of the sampling points sampled at different scales are fused through a multi-layer perceptron (MLP) to obtain the semantic features of the sampling points.

[0162] In one implementation, calculating the offset attention weight of the sampling point based on the contextual features of the current tracking video frame and the contextual features of the sampling point includes:

[0163] Calculating a first similarity between each feature in the context features of the current point tracking video frame and each feature in the context features of the sampling point;

[0164] Calculating the second similarity of the sampling points through a multi-layer perceptron according to all the calculated first similarities;

[0165] Calculate the offset attention weight of the sampling point according to the second similarity of the sampling point.

[0166] Specifically, the context feature includes a set of features, that is, the context feature is a block-level feature. The offset attention weight of each sampling point is determined based on the similarity between the block-level feature of the sampling point and the block-level feature in the point query of the current point tracking video frame. Figure 5 As shown, taking the mth sampling point as an example, the context feature of the mth sampling point The N in the context feature C that will be queried at the point in the current tracking video frame 2 Features and N in 2 Calculate the first similarity between each pair of features to get N 2 ×N 2 These first similarities will be fed into a multi-layer perceptron to calculate the final similarity value, namely the second similarity. Based on the second similarity, the offset attention weight of the mth sampling point can be determined.

[0167] In one implementation, the method for updating the semantic features of the current tracking video frame further includes:

[0168] Perform weighted fusion on the semantic features of each sampling point according to the offset attention weight of each sampling point to obtain a spatial feature query result;

[0169] The semantic features of the current tracking video frame are updated according to the spatial feature query result.

[0170] This embodiment also updates the semantic features of the current tracking video frame through the contextual mutual attention mechanism (CCA). Specifically, Figure 4 As shown, for each sampling point, the semantic feature V of the sampling point is calculated by the attention weight W of the sampling point. t Then, the weighted semantic features of each sampling point are summed to obtain the spatial feature query result Δ Using spatial feature query results Δ To update (optimize) the semantic features of the current tracking video frame, so as to obtain further updated semantic features ft '.

[0171] In one implementation, Figure 2 As shown, this embodiment can use the visibility long-term attention mechanism (VLTA), contextual mutual attention mechanism (CCA), attention-based position update (APU), self-attention and feedforward network to build an optimization module. The optimization module iteratively updates the semantic features and point positions of each tracking video frame. Taking the current tracking video frame as the t-th frame as an example, the semantic features and point positions that have not been fully optimized are f t ' and l t ', the fully optimized semantic features and point positions are f t and l t .

[0172] In one implementation, the method further includes:

[0173] If a cut occurs in the current tracking video frame, a global matching operation is activated in the current tracking video frame to obtain a global matching point;

[0174] The point position in the point query is replaced according to the position of the global matching point.

[0175] Specifically, when the target video is a video collected from the natural world, there is usually no problem of shot cutting. However, when the target video is a video that has been artificially processed, there is usually a problem of shot cutting. When shot cutting occurs, the point position l in the query point is inherited from the previous tracking video frame. t-1 It is no longer applicable. At this time, the global matching operation will be activated to find a more suitable global matching point as the point position.

[0176] In one implementation, activating a global matching operation in the current tracking video frame to obtain a global matching point includes:

[0177] Using the depth image features of the current tracking video frame as a matching feature map;

[0178] Calculating a similarity map based on each feature in the context features of the current tracking video frame and the matching feature map;

[0179] The similarity graphs of the features in the context features of the current tracking video frame are fused into a target similarity graph, and a global matching point is determined according to the target similarity graph.

[0180] Specifically, if the depth image features of the current tracking video frame have only one scale, the single-scale depth feature map is used as the matching feature map. A similarity map is calculated between each feature of the context features of the target tracking point in the point query inherited from the current tracking video frame and the matching feature map. The similarity maps of all features are then fused into a target similarity map. The optimal coordinates are found through this target similarity map, which is the global matching point.

[0181] For example, taking the current tracking video frame as frame t, the context feature C of the target tracking point is N 2 features and the depth image feature X of the current tracking video frame t Calculate the similarity to get N 2 Similarity graph This N 2 Similarity graphs are fed into a multi-layer perceptron, thus N 2 Similarity graphs are fused into a more reliable target similarity graph H t ∈R H×W×1 Calculate SoftArgMax on the target similarity graph to get a coordinate, which is the position of the global matching point. Replace the point position l in the point query with the global matching point. t-1 .

[0182] In one implementation, activating a global matching operation in the current tracking video frame to obtain a global matching point further includes:

[0183] If the depth image feature of the current tracking video frame is multi-scale, then for each single-scale feature map, multiple similarity maps are calculated based on each feature in the context feature of the current tracking video frame and each single-scale feature map;

[0184] fusing the multiple similarity graphs into a target similarity graph;

[0185] A global matching point is determined based on the target similarity graph.

[0186] Specifically, if the depth image features of the current tracking video frame have multiple scales, the context features of the target tracking point are obtained by querying the inherited points of the current tracking video frame. The similarity between each feature of the context features of the target tracking point and the feature map of the single scale is calculated on each feature map of the single scale to obtain multiple similarity maps. Finally, all similarity maps generated at each scale are fused into a target similarity map, and the optimal coordinates are found through this target similarity map to obtain the global matching point.

[0187] For example, taking the current tracking video frame as the t-th frame, if the depth image features of the current tracking video frame have multiple scales, the following calculation process is performed at each scale to obtain N 2 Similarity graph: N in the context feature C of the target tracking point 2 features and the depth image feature X of the current tracking video frame t Calculate the similarity to get N 2 Similarity graph For example, if the number of scales is 4, then we will end up with 4N 2 Similarity graph Similarly, this 4N 2 The similarity graph will be fed into the multi-layer perceptron, and the 4N 2 Similarity graphs are fused into a more reliable target similarity graph H t ∈R H×W×1 . Calculate SoftArgMax on the target similarity graph to get a coordinate, which is the position of the global matching point.

[0188] In one implementation, the method further includes:

[0189] According to the target tracking point and each tracking video frame, a mask and / or a position frame of the target object to which the target tracking point belongs in each tracking video frame is generated.

[0190] Specifically, this embodiment can also achieve multiple functions by combining other algorithms. For example, after completing tracking of a target tracking point, the position of the target object to which the target tracking point belongs in the tracking video frame can be identified using the target tracking point and the tracking video frame to which the target tracking point belongs. This can then be used to generate a mask and / or an indicator frame (e.g., a square frame) of the target object, thereby achieving tracking of the target object at the mask level or the frame level.

[0191] For example, the target tracking point and the tracking video frame to which it belongs are used as input data for the SAM model, which outputs the mask of the target object to which the target tracking point belongs. Alternatively, the target tracking point and the tracking video frame to which it belongs are used as input data for the TRex model, which outputs the box of the target object to which the target tracking point belongs.

[0192] Advantages of the present invention include:

[0193] (1) The present invention is an online algorithm that does not rely on future frames to assist in the processing of the current frame. It can stably track any point in challenging long videos and is more robust in terms of spatial and temporal perception. Furthermore, it does not require the calculation of a similarity matrix, saving significant computational resources. Furthermore, the algorithm structure is more concise and clear, facilitating model training and optimization.

[0194] (2) The ability to query spatial information is improved through the contextual information of the image. The spatial perception of long videos is designed by using single-point features and contextual features around the points, which can query the correct spatial features of the tracking points in long videos.

[0195] (3) The perception of long-term temporal information is achieved through the attention mechanism, and the attention of long-term temporal information is weighted by the visibility of the target tracking point in each tracking video frame, thereby improving the information query capability in the time dimension and reducing feature drift.

[0196] (4) When a mirror cut occurs, a global matching algorithm is called to help reconstruct the tracking and avoid using invalid point queries.

[0197] The effect of the present invention proves that:

[0198] The algorithm of this invention was tested on the task of tracking arbitrary points based on the TAP-Vid-DAVIS and TAP-Vid-Kinetics public datasets. The experimental results are shown in the figure. Figure 6 As shown in FIG. 4 , the algorithm model of the present invention achieves the best results.

[0199] In addition, the algorithm of the present invention has been used to conduct relevant experiments on the task of tracking arbitrary points based on TAP-Vid and RoboTAPbenchumarks (a total of four datasets). The experimental results are as follows: Figure 7 As shown. The algorithm model of the present invention achieved the best results when using the same training data. It should be noted that and Both introduce additional data for training. Specifically, We re-rendered the Kubric video at 64 frames and added 15,000 additional real videos. Trained on an additional 15 million real videos. The algorithm of the present invention achieves state-of-the-art performance on most datasets and remains competitive with algorithms trained on additional internal data. Training data: (Kub24) Kubri dataset with 24 frames per video, (Kub64) Kubri with 64 frames per video, (PO) PointOdyssey, (FT) FlyingThings++. It can be seen that although the algorithm of the present invention was only trained on a small amount of public synthetic data, it is still competitive with those algorithms trained on a large amount of additional internal data, and even surpasses them in some indicators, which reflects the superiority of the algorithm of the present invention.

[0200] Based on the above embodiments, the present invention also provides a video tracking device, such as Figure 8 As shown, the device includes:

[0201] Determination module 01, used to determine the target tracking point and the initial video frame from the target video;

[0202] Tracking module 02 is used to construct a point query of the target tracking point based on the initial video frame, and update the point query of the target tracking point based on the depth image features of the tracking video frame after the initial video frame to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, context features and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point.

[0203] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 9 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video tracking method is implemented. The display screen of the terminal can be a liquid crystal display or an electronic ink display.

[0204] Those skilled in the art will understand that Figure 9 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0205] In one implementation, the terminal has one or more programs stored in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a video tracking method.

[0206] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0207] In summary, the present invention discloses a video tracking method, device, terminal and medium. The method determines a target tracking point and an initial video frame from a target video; constructs a point query of the target tracking point based on the initial video frame, and updates the point query of the target tracking point based on the depth image features of the tracking video frame after the initial video frame to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, context features and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point. The present invention combines the semantic features of the target tracking point itself with the surrounding context features, which can improve the query capability of spatial information, so that the correct spatial features of the target tracking point can still be found in a long video or when the tracking target undergoes significant changes. The point query is composed of the semantic features, context features and point positions of the target tracking point, and the point tracking task of a long video is achieved by updating the point query frame by frame. It is applicable to video editing, augmented reality, 3D reconstruction, optical flow estimation and other fields.

[0208] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A video tracking method, characterized in that: The method comprises: Determine the target tracking point and the initial video frame from the target video; Constructing a point query for the target tracking point according to the initial video frame, and updating the point query for the target tracking point based on depth image features of a tracking video frame subsequent to the initial video frame, so as to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point; Updating the point query of the target tracking point based on the depth image features of the tracking video frame after the initial video frame includes: Extract the depth image features of the current tracking video frame; According to the depth image features of the current tracking video frame and the point query of the target tracking point in the previous tracking video frame, the optimized point position and optimized semantic features of the target tracking point in the current tracking video frame are determined, including: according to the point query of the target tracking point in the initial video frame, the depth image features of the initial video frame and the visibility of the preset initial video frame, the optimized semantic features and optimized point position of the target tracking video in the second tracking video frame and the visibility of the second tracking video frame are obtained; for each tracking video frame after the second tracking video frame, the optimized semantic features and optimized point position of the target tracking point in the current tracking video frame are obtained according to the point query of the target tracking point in the initial video frame, the depth image features of the initial video frame and the visibility of the preset initial video frame ... The method comprises the following steps: updating the semantic features and point positions of the target tracking point in the current tracking video frame based on the query, the depth image features of the current tracking video frame, the optimized semantic features and visibility of the target tracking point in all historical tracking video frames before the current tracking video frame, and obtaining the optimized semantic features and optimized point positions of the target tracking point in the current tracking video frame; updating the point query in the current tracking video frame according to the optimized point positions of the target tracking point in the current tracking video frame; and calculating the visibility of the target tracking point in the current tracking video frame according to the optimized semantic features of the target tracking point in the current tracking video frame. The point query of the target tracking point is updated based on the optimized point position.

2. The video tracking method according to claim 1, characterized in that: Constructing a point query of the target tracking point according to the initial video frame, comprising: Extracting initial depth image features of the initial video frame through a backbone network; Obtaining the point position of the target tracking point in the initial video frame; Constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position; A point query of the target tracking point is obtained based on the point position, the semantic feature and the context feature.

3. The video tracking method according to claim 2, characterized in that: Constructing semantic features and context features of the target tracking point based on the depth image features and the point position, including: If the depth image feature of the initial video frame is single-scale, performing a first sampling based on the single-scale depth image feature and the point position to obtain a semantic feature of the target tracking point; A second sampling is performed based on the single-scale depth image feature and the point position to obtain a context feature of the target tracking point.

4. The video tracking method according to claim 2, wherein: Constructing semantic features and context features of the target tracking point based on the initial depth image features and the point position, including: If the depth image feature of the initial video frame is multi-scale, for each single-scale depth image feature in the multi-scale depth image feature, perform a first sampling on each single-scale depth image feature based on the point position to obtain multiple initial semantic features of the target tracking point; Performing a second sampling on each of the single-scale depth image features based on the point position to obtain a plurality of initial context features of the target tracking point; The multiple initial semantic features and the multiple initial context features are fused respectively by a multi-layer perceptron to obtain the semantic features of the target tracking point and the context features of the target tracking point.

5. The video tracking method according to claim 1, characterized in that: Obtaining optimized semantic features and optimized point positions in a second tracking video frame and visibility of the second tracking video frame based on a point query of the target tracking point in an initial video frame, a depth image feature of the initial video frame, and visibility of a preset initial video frame, including: Optimizing the semantic features and point positions in the initial video frame according to the point query of the target tracking point in the initial video frame and the depth image features of the initial video frame to obtain optimized semantic features and optimized point positions of the target tracking point in the initial video frame; updating the point query in the initial video frame according to the optimized point position in the initial video frame to obtain a point query in a second tracking video frame; updating the semantic features and point positions of the target tracking point in the second tracking video frame according to the point query of the second tracking video frame, the depth image features of the second tracking video frame, the optimized semantic features in the initial video frame, and the visibility of the preset initial video frame to obtain the optimized semantic features and optimized point positions in the second tracking video frame; updating the point query of the second tracking video frame according to the optimized point position in the second tracking video frame; The visibility of the target tracking point in the second tracking video frame is calculated according to the optimized semantic features in the second tracking video frame.

6. The video tracking method according to claim 1, characterized in that: The method for updating the semantic features of the current tracking video frame includes: For each of the historical tracking video frames, calculating an attention weight according to the optimized semantic features of the historical tracking video frame and the semantic features of the current tracking video frame; Reweighting the attention weight according to the visibility of the historical tracking video frame to obtain a reweighted attention weight of the historical tracking video frame; According to the reweighted attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused to obtain a temporal feature query result; The semantic features of the current tracking video frame are updated according to the temporal feature query result.

7. The video tracking method according to claim 6, characterized in that: According to the reweighted attention weights of the historical tracking video frames, weighted fusion is performed on the optimized semantic features of the historical tracking video frames, including: For each of the historical tracking video frames, obtaining a frame position relationship between the historical tracking video frame and the current tracking video frame; updating the reweighted attention weight of the historical tracking video frame according to the frame position relationship to obtain an updated attention weight of the historical tracking video frame; According to the updated attention weights of the historical tracking video frames, the optimized semantic features of the historical tracking video frames are weightedly fused.

8. The video tracking method according to claim 1, characterized in that: The method for updating the point position in the current tracking video frame includes: The semantic features of the current tracking video frame are generated into a plurality of sampling offsets by a multi-layer perceptron; For each of the sampling offsets, generating a sampling point according to the sampling offset and the point position in the current tracking video frame; Sampling is performed according to the depth image features of the current tracking video frame and the position of the sampling point to obtain semantic features and context features of the sampling point; Calculating the offset attention weight of the sampling point based on the contextual features in the current tracking video frame and the contextual features of the sampling point; Perform weighted fusion on the sampling offsets according to the offset attention weights of the sampling points to obtain a position update result; The position of the point in the current tracking video frame is updated based on the position update result.

9. The video tracking method according to claim 8, characterized in that: Sampling is performed according to the depth image feature of the current tracking video frame and the position of the sampling point to obtain the context feature of the sampling point, including: If the depth image feature of the current tracking video frame is multi-scale, determining a target feature map from the depth image feature of the current tracking video frame according to the scale to which the sampling point belongs; A second sampling is performed according to the target feature map and the position of the sampling point to obtain a context feature of the sampling point.

10. The video tracking method according to claim 8, characterized in that: Calculating the offset attention weight of the sampling point based on the context feature of the current tracking video frame and the context feature of the sampling point includes: Calculating a first similarity between each feature in the context features of the current tracking video frame and each feature in the context features of the sampling point; Calculating the second similarity of the sampling points through a multi-layer perceptron according to all the calculated first similarities; Calculate the offset attention weight of the sampling point according to the second similarity of the sampling point.

11. The video tracking method according to claim 8, characterized in that: The method for updating the semantic features of the current tracking video frame further includes: Perform weighted fusion on the semantic features of each sampling point according to the offset attention weight of each sampling point to obtain a spatial feature query result; The semantic features of the current tracking video frame are updated according to the spatial feature query result.

12. The video tracking method according to claim 1, characterized in that: The method further comprises: If a cut occurs in the current tracking video frame, a global matching operation is activated in the current tracking video frame to obtain a global matching point; The point position in the point query is replaced according to the position of the global matching point.

13. The video tracking method according to claim 12, characterized in that: Activate a global matching operation on the current tracking video frame to obtain global matching points, including: Using the depth image features of the current tracking video frame as a matching feature map; Calculating a similarity map based on each feature in the context features of the current tracking video frame and the matching feature map; The similarity graphs of the features in the context features of the current tracking video frame are fused into a target similarity graph, and a global matching point is determined according to the target similarity graph.

14. The video tracking method according to claim 12, characterized in that: Activating a global matching operation on the current tracking video frame to obtain a global matching point further includes: If the depth image feature of the current tracking video frame is multi-scale, then for each single-scale feature map, multiple similarity maps are calculated based on each feature in the context feature of the current tracking video frame and each single-scale feature map; fusing the multiple similarity graphs into a target similarity graph; A global matching point is determined based on the target similarity graph.

15. A video tracking device, characterized in that: The device comprises: A determination module, used to determine the target tracking point and the initial video frame from the target video; a tracking module, configured to construct a point query for the target tracking point based on the initial video frame, and update the point query for the target tracking point based on depth image features of a tracking video frame subsequent to the initial video frame, so as to achieve continuous tracking of the target tracking point, wherein the point query includes semantic features, contextual features, and point positions, and the tracking video frame is a video frame in the target video that contains the target tracking point; Updating the point query of the target tracking point based on the depth image features of the tracking video frame after the initial video frame includes: Extract the depth image features of the current tracking video frame; According to the depth image features of the current tracking video frame and the point query of the target tracking point in the previous tracking video frame, the optimized point position and optimized semantic features of the target tracking point in the current tracking video frame are determined, including: according to the point query of the target tracking point in the initial video frame, the depth image features of the initial video frame and the visibility of the preset initial video frame, the optimized semantic features and optimized point position of the target tracking video in the second tracking video frame and the visibility of the second tracking video frame are obtained; for each tracking video frame after the second tracking video frame, the optimized semantic features and optimized point position of the target tracking point in the current tracking video frame are obtained according to the point query of the target tracking point in the initial video frame, the depth image features of the initial video frame and the visibility of the preset initial video frame ... The method comprises the following steps: updating the semantic features and point positions of the target tracking point in the current tracking video frame based on the query, the depth image features of the current tracking video frame, the optimized semantic features and visibility of the target tracking point in all historical tracking video frames before the current tracking video frame, and obtaining the optimized semantic features and optimized point positions of the target tracking point in the current tracking video frame; updating the point query in the current tracking video frame according to the optimized point positions of the target tracking point in the current tracking video frame; and calculating the visibility of the target tracking point in the current tracking video frame according to the optimized semantic features of the target tracking point in the current tracking video frame. The point query of the target tracking point is updated based on the optimized point position.

16. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the video tracking method according to any one of claims 1 to 14; and the processor is used to execute the programs.

17. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the video tracking method according to any one of claims 1 to 14.

Images