Method and device for tracking points in video, equipment and storage medium

Abstract

The invention discloses a method, device and equipment for tracking a point in a video and a storage medium. The method comprises the following steps: selecting a tracked point in the video; constructing an initial point query of the tracked point in each video frame; and based on the depth feature map of each video frame and the initial point query of the tracked point in each video frame, determining the target point query of the tracked point in each video frame through a decoding module so as to track the tracked point. According to the method, the initial point query is constructed for the tracked point in each video frame, and then the target point query of the tracked point in each video frame is determined in parallel based on the depth feature map of each video frame and each initial point query, so that the tracked point is tracked; therefore, the position information of the tracked point in each video frame can be determined at the same time, and the problem that stable tracking cannot be continued when the tracking point is shielded can be avoided.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for tracking points in a video. Background Art

[0002] Currently, it is common to track points in a video by optical flow estimation to determine the moving trajectory of the points. However, when tracking points by optical flow estimation, if the tracked point is blocked by other objects in an image frame, it is impossible to estimate the position of the tracked point in the image frame by comparing the optical flow of the image frame with its previous image frame, resulting in tracking interruption, and thus it is impossible to achieve long-term tracking of the point.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a method, device, equipment and storage medium for tracking points in a video in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a method for tracking points in a video, wherein the method for tracking points in a video specifically includes:

[0006] In response to the point selection operation, select the tracked point in the video;

[0007] Constructing an initial point query of the tracked point in each video frame in the video, wherein the initial point query includes an initial semantic feature and an initial position;

[0008] Based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, the target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point, wherein the target point query includes a target semantic feature and a target position.

[0009] In the method for tracking points in a video, the tracked point is at least one tracked point determined by a user on at least one video frame in the video.

[0010] In the method for tracking a point in a video, the step of constructing an initial point query of the tracked point in each video frame in the video specifically comprises:

[0011] Reading a starting video frame corresponding to the tracked point;

[0012] Acquire an initial position of the tracked point in the starting video frame, determine an initial semantic feature of the tracked point in the starting video frame based on the initial position and a depth feature map of the starting video frame, and construct an initial point query of the tracked point in the starting video frame based on the initial semantic feature and the initial position;

[0013] The initial point query of the tracked point in the starting video frame is used as the initial point query of the tracked point in each video frame in the video, so as to obtain the initial point query of the tracked point in each video frame in the video.

[0014] In the method for tracking points in a video, the starting video frame is a video frame on which a point selection operation is performed, or the starting video frame is any video frame selected by a user in the video.

[0015] The method for tracking a point in a video, wherein the depth feature map of the starting video frame includes a plurality of depth feature maps of different image scales, and determining the initial semantic features of the tracked point in the starting video frame based on the initial position and the depth feature map of the starting video frame specifically includes:

[0016] Sample the depth features corresponding to the initial position in the depth feature map of each image scale respectively;

[0017] All the sampled deep features are fused to obtain the initial semantic features of the tracked point.

[0018] The method for tracking points in a video, wherein the query based on the depth feature map of each video frame in the video and the initial point of the tracked point in each video frame, and determining the target point query of the tracked point in each video frame through a decoding module specifically includes:

[0019] For each video frame, performing at least one position offset sampling in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point;

[0020] Determining an attention weight of each position offset sampled visual feature based on each position offset sampled visual feature and an initial semantic feature of the tracked point in the video frame;

[0021] Based on the attention weights of the visual features sampled at each position offset, an initial point query of the tracked point in the video frame is updated through a mutual attention mechanism in the decoding module, and a target point query of the tracked point in the video frame is determined according to the updated initial point query.

[0022] The method for tracking a point in a video, wherein the step of performing at least one position offset sampling in a depth feature map of the video frame based on an initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point specifically includes:

[0023] Acquire the segmented area to which the tracked point belongs, and determine the movement pattern of the tracked point based on the segmented area;

[0024] Based on the movement rule, at least one position offset of the tracked point is determined, and position offset sampling is performed in the depth feature map of the video frame according to each position offset and the initial position of the tracked point in the video frame to obtain at least one position offset sampled visual feature corresponding to the tracked point.

[0025] In the method for tracking points in a video, the attention weight of each position offset sampled visual feature is determined based on each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame as follows:

[0026] The similarity between each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame is calculated respectively to obtain the attention weight of each position offset sampled visual feature.

[0027] The method for tracking a point in a video, wherein the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0028] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and initial positions of the tracked points in each video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracked points in the video frame, wherein the updated point query includes an updated semantic feature and an updated position;

[0029] The updated point query of the tracked point in the video frame is used as the target point query of the tracked point in the video frame.

[0030] The method for tracking a point in a video, wherein the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0031] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain the updated semantic features of the tracked point in the video frame;

[0032] The updated semantic features and the initial position of the tracked point in the video frame are updated through the feedforward network in the decoding module to obtain a target point query of the tracked point in the video frame.

[0033] The method for tracking a point in a video, wherein the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0034] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and the initial position of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracking point in the video frame, wherein the updated point query includes an updated semantic feature and an updated position;

[0035] The update point query of the tracked point in the video frame is updated through the feedforward network in the decoding module to obtain the target point query of the tracked point in the video frame.

[0036] The method for tracking a point in a video, wherein the attention weights of the visual features sampled based on each position offset and updating the initial semantic features of the tracked point in the video frame through the mutual attention mechanism in the decoding module specifically include:

[0037] The visual features of the offset samples at each position are weighted based on the attention weights of the visual features of the offset samples at each position to obtain the sampled semantic features;

[0038] The sampled semantic feature is added to the initial semantic feature of the tracked point in the video frame to obtain an updated semantic feature of the tracked point in the video frame.

[0039] The method for tracking a point in a video, wherein the attention weights of the visual features sampled based on each position offset and updating the initial position of the tracked point in the video frame through the mutual attention mechanism in the decoding module specifically include:

[0040] Based on the attention weight of the visual features of each position offset sample, the position offset corresponding to each position offset sample is weighted to obtain the target position offset;

[0041] The target position offset and the initial position of the tracked point in the video frame are added to obtain an updated position of the tracked point in the video frame.

[0042] The method for tracking points in a video, wherein the attention weights of the visual features sampled based on each position offset are updated through the mutual attention mechanism in the decoding module after the initial semantic features of the tracked point in the video frame are updated, the method further comprises:

[0043] The updated semantic features are updated through a spatiotemporal attention mechanism, or through at least one of a spatial attention mechanism and a temporal attention mechanism.

[0044] The method for tracking points in a video, wherein the decoding module includes several decoding layers, and the input items of each decoding layer in the several decoding layers are initial point queries, wherein the initial point query of the first decoding layer is the initial point query of the tracked point in the video frame, and the initial point queries of other decoding layers are the target point queries output by their previous decoding layers.

[0045] The method for tracking points in a video, wherein the query based on the depth feature map of each video frame in the video and the initial point of the tracked point in each video frame, determining the target point query of the tracked point in each video frame by a decoding module to track the tracked point, before the method further comprises:

[0046] Dividing the video into a plurality of video frame groups according to time sequence, wherein each video frame in the video is included in at least one video frame group;

[0047] Each of the video frame groups is taken as a sub-video to be tracked, and a depth feature map of each video frame in the sub-video to be tracked and an initial point query of the tracked point in each video frame are performed on each sub-video to be tracked, and a target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point.

[0048] The method for tracking points in a video, wherein before performing a query based on a depth feature map of each video frame in the sub-video to be tracked and an initial point of the tracked point in each video frame for each sub-video to be tracked, determining a target point query of the tracked point in each video frame by a decoding module to track the tracked point, the method comprises:

[0049] Detect whether there is an overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked;

[0050] If there is no overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked, read the first target point query of the last video frame of the tracked point in the previous sub-video to be tracked, and use the first target point query as the initial point query of each video frame of the tracked point in the current sub-video to be tracked;

[0051] If there are overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, read the overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, and update the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frame.

[0052] In the method for tracking a point in a video, the updating of the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frames specifically comprises:

[0053] For each overlapping video frame in the current sub-video to be tracked, querying the second target point of the tracked point in the overlapping video frame as the initial point query of the tracked point in the overlapping video frame;

[0054] For the non-overlapping video frames in the current sub-video to be tracked, the last overlapping video frame in the overlapping video frames is read in time sequence, and the second target point query of the tracked point in the last overlapping video frame is used as the initial point query of the tracked point in each non-overlapping video frame.

[0055] The method for tracking points in a video, wherein the method further comprises: based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, determining the target point query of the tracked point in each video frame through a decoding module to track the tracked point, and then:

[0056] The visibility category of the tracked point in each video frame is determined based on the target semantic features of the tracked point in each video frame, wherein the visibility category is visible or invisible.

[0057] The method for tracking a point in a video, wherein after determining the visibility category of the tracked point in each video frame based on the target semantic features of the tracked point in each video frame, the method comprises:

[0058] According to the visibility category of the tracked point in each video frame, the tracked point is marked in each video frame based on the target position of the tracked point in each video frame to display the movement trajectory of the tracked point. A second aspect of the present application provides a device for tracking points in a video, wherein the device for tracking points in a video specifically includes:

[0059] The response module is used to respond to the point selection operation and select the tracked point in the video;

[0060] A construction module, used to construct an initial point query of the tracked point in each video frame in the video, wherein the initial point query includes an initial semantic feature and an initial position;

[0061] A control module is used to determine a target point query of the tracked point in each video frame through a decoding module based on a depth feature map of each video frame in the video and an initial point query of the tracked point in each video frame to track the tracked point, wherein the target point query includes a target semantic feature and a target position.

[0062] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the methods for tracking points in a video as described above.

[0063] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0064] The memory stores a computer-readable program executable by the processor;

[0065] When the processor executes the computer-readable program, the processor implements the steps in any of the above methods for tracking points in a video.

[0066] Beneficial effects: Compared with the prior art, the present application provides a method, device, equipment and storage medium for tracking points in a video, the method comprising: selecting a tracked point in a video in response to a point selection operation; constructing an initial point query for the tracked point in each video frame in the video; based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, determining the target point query of the tracked point in each video frame through a decoding module to track the tracked point. The present application constructs an initial point query including semantic features and position information for the tracked point in each video frame, and then determines the target point query of the tracked point in each video frame in parallel based on the depth feature map of each video frame and each initial point query, so as to detect the tracked point from each video frame at the same time, so as to avoid the problem of being unable to continue stable tracking when the tracking point is blocked. At the same time, the present application can track all video frames in the video in parallel, thereby improving the efficiency of point tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A flowchart of a method for tracking points in a video provided in an embodiment of the present application.

[0069] Figure 2 This is an example picture of the point being chased.

[0070] Figure 3 A principle flow chart of an example of a method for tracking points in a video provided in an embodiment of the present application.

[0071] Figure 4 A schematic flow chart of an example of the target point query determination process.

[0072] Figure 5 A schematic diagram of the structure of a device for tracking points in a video provided in an embodiment of the present application.

[0073] Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiments of the present application provide a method, device, equipment and storage medium for tracking points in a video. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0076] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0077] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0078] The inventor has found through research that currently, optical flow estimation is generally used to track points in a video to determine the movement trajectory of the points. However, when tracking points through optical flow estimation, when the tracked point is blocked by other objects in an image frame, it is impossible to estimate the position of the tracked point in the image frame by performing optical flow analysis on the image frame and its previous image frame, resulting in tracking interruption, and thus the tracking point cannot be tracked for a long time.

[0079] In order to solve the above problems, some researchers borrowed the method of optical flow estimation, constructed a similarity matrix by calculating the similarity between the image features of the tracked point and each point in all video frames, and then determined the position of the tracked point in each video frame based on the calculated similarity matrix. Then, the amount of calculation required to calculate the similarity matrix will increase rapidly with the increase of video length, video frame size and the number of tracked points.

[0080] To this end, the present application converts the point tracking problem into a point detection problem, selects a tracked point in the video in response to the user's point selection operation; constructs an initial point query of the tracked point in each video frame in the video; based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, the target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point. The present application detects the target position of the tracked point in each video frame by determining the target point query in each video frame, and then tracks the tracked point according to all the detected target positions. In this way, the tracking point will not be affected by the inability to continue stable tracking when it is blocked, and there is no need to calculate the similarity between the tracked point and each image point in each video frame, which reduces the amount of calculation required for point tracking, thereby improving the efficiency of point tracking.

[0081] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0082] This embodiment provides a method for tracking points in a video, such as Figure 1 As shown, the method includes:

[0083] S10, in response to the point selection operation, a tracked point is selected in the video.

[0084] Specifically, the point selection operation refers to the user determining the tracked point in the video, and the tracked point is the point that needs to be tracked, that is, the tracked point will be tracked in the video as a target point to obtain the movement trajectory of the tracked point in the video. Among them, the tracked point can be at least one tracked point determined on at least one video frame in the video, and the number of tracked points can be determined according to actual needs, for example, the tracked points are 1, 2, 5 or 10, etc. In other words, the point selection operation can be used to select one tracked point, and can also be used to select multiple tracked points. Among them, when there are multiple tracked points, it can be achieved through one point selection operation, and it can also be achieved through multiple point selection operations, and there is no specific limitation here.

[0085] Furthermore, the point selection operation may be a confirmation operation performed by the user on any video frame in the video (for example, by clicking a mouse on any video frame in the video to form a point selection operation, etc.), or may be formed by receiving tracked point information uploaded by the user or sent by other external devices, wherein the tracked point information may include the video frame sequence number, the position coordinates of the tracked point, and the label of the tracked point, etc. In a specific implementation of the present application, the point selection operation is formed by the user clicking or selecting a video frame, that is, the tracked point is at least one tracked point determined by the user through a confirmation operation such as clicking or selecting on at least one video frame in the video.

[0086] Exemplarily, before responding to the tracked point selection instruction and determining the tracked point in the video, the method further includes:

[0087] A point selection operation performed by a user on at least one video frame in the video is received to determine at least one tracked point on at least one video frame in the video.

[0088] Specifically, the point selection operation is a click operation performed by the user on the video frame, for example, Figure 2 As shown, users Figure 2When the tip of the marker pen in a video frame performs a click operation, the target point corresponding to the tip of the marker pen is used as the tracked point. When at least one video frame includes one video frame, the point selection operation can determine multiple tracked points on the video frame; when at least one video frame includes multiple video frames, the point selection operation can determine one or more tracked points on each of the multiple video frames. In other words, when there are multiple tracked points, the multiple tracked points can come from the same video frame or from different video frames. For example, the tracked points include tracked point a, tracked point b, tracked point c and tracked point d, the video includes video frame 1, video frame 2,..., video frame n, n>4, the tracked point a, tracked point b, tracked point c and tracked point d can all originate from video frame 1, or the tracked point a can originate from video frame 1, the tracked point b can originate from video frame 2, the tracked point c can originate from video frame 3, the tracked point d can originate from video frame 4, or the tracked point a and the tracked point b can originate from video frame 1, and the tracked point c and the tracked point d can originate from video frame 2.

[0089] S20: construct an initial point query for the tracked point in each video frame in the video.

[0090] Specifically, the initial point query includes an initial semantic feature and an initial position, wherein the initial semantic feature may be a semantic vector, which is used to represent the depth feature information of the tracked point in the depth feature map corresponding to the video frame, and the vector length of the initial semantic feature is the same as the feature dimension of the depth feature map corresponding to the video frame. For example, if the feature dimension of the depth feature map is C, then the vector length of the initial semantic feature is C. The initial position may be a two-dimensional position vector, which is used to represent the position information of the tracked point in the video frame, and the initial position includes the x-coordinate and y-coordinate of the tracked point in the video frame. For example, if the x-coordinate of the tracked point in the video frame is 5 and the y-coordinate is 10, then the initial position is (5, 10). In addition, since the initial semantic feature represents the depth feature of the tracked point in the depth feature map corresponding to the video frame, before constructing the initial point query for each video frame, feature extraction may be performed on each video frame to obtain the depth feature map corresponding to each video frame, wherein the depth feature map may be obtained by a traditional feature extraction method or by a feature extraction module based on deep learning.

[0091] Exemplarily, the deep feature map is extracted by a feature extraction module, specifically, as Figure 3 As shown, for each video frame X in the video frame t , the video frame X t Input feature extraction module, output video frame X through feature extraction module t The depth feature map F tAmong them, F t It can be a single-scale image feature, for example, F t is a feature map of H×W×C, where H represents the image feature height, W represents the image feature width, and C represents the feature dimension; it can also be a multi-scale graphic feature, for example, F i It includes a feature map of H1×W1×C, a feature map of H2×W2×C, a feature map of H3×W3×C and a feature map of H4×W4×C, wherein H1, H2, H3 and H4 all represent image feature heights, and W1, W2, W3 and W4 all represent image feature widths.

[0092] In one implementation, constructing an initial point query of the tracked point in each video frame in the video specifically includes:

[0093] S21, reading the starting video frame corresponding to the tracked point;

[0094] S22, obtaining an initial position of the tracked point in the starting video frame, determining an initial semantic feature of the tracked point in the starting video frame based on the initial position and a depth feature map of the starting video frame, and constructing an initial point query of the tracked point in the starting video frame based on the initial semantic feature and the initial position;

[0095] S23, taking the initial point query of the tracked point in the starting video frame as the initial point query of the tracked point in each video frame in the video, so as to obtain the initial point query of the tracked point in each video frame in the video.

[0096] Specifically, in step S21, the starting video frame is a video frame used to determine the initial point query, that is, the initial point query of the tracked point is determined based on the starting video frame. The starting video frame may be automatically determined according to a preset selection rule, or may be set by the user. For example, when the starting video frame is automatically determined according to a preset selection rule, the preset selection rule may be to use the video frame where the point selection operation is to be performed as the starting video frame, wherein the point selection operation is a point selection operation for selecting the tracked point; when the starting video frame is set by the user, the starting video frame may be determined based on the starting frame selection operation performed by the user, wherein the starting frame selection operation may be implemented by inputting a video frame sequence number, or by clicking on a video frame, etc.

[0097] Based on this, the starting video frame is the video frame where the point selection operation is performed, and it can also be any video frame in the video. For example, the starting video frame is the video frame where the tracked point appears for the first time in the video, or the starting video frame is the video frame where the tracked point appears for the fifth time in the video. Among them, when the starting video frame is the video frame where the point selection operation is performed, it means that the starting video frame can be selected synchronously when selecting the tracked point, and the starting video frame is used as the first video frame that the tracked point needs to start tracking, so as to track the moving trajectory of the tracked point. It should be noted that the movement of the tracking point refers to the absolute position movement of the tracking point with respect to the video interface, rather than the relative position movement in the video, that is, a stationary object in the video may also cause the movement of the tracked point due to the movement of the lens. When the starting video frame can be any video frame in the video, the tracked point can be any target point in the video, and any movement of it in the image will be tracked by us. It can be seen that any target point can be tracked from any video frame, so that the moving trajectory to be tracked can be set according to the needs, so that the moving trajectory of the tracking point can better meet the needs of users.

[0098] It should be noted that, when there are multiple tracked points, the method for determining the starting video frame of each tracked point may be the same or different. For example, the starting video frame of each tracked point is the video frame on which the point selection operation is performed, or the starting video frame of each tracked point is the video frame determined according to the starting frame selection operation performed by the user, or the starting video frames of some tracked points are the video frames on which the point selection operation is performed, and the starting video frames of some tracked points are the video frames determined according to the starting frame selection operation performed by the user, etc.

[0099] Of course, in practical applications, the starting video frame can be a video frame used only as a query to determine the initial point, or it can be a video frame used as a query to determine the initial point and the first video frame to be tracked as the tracked point. Among them, when the starting video frame is used as the first video frame to be tracked as the tracked point, it means that the video frames before the starting video frame do not need to be tracked. Then, after obtaining the starting video frame, the starting video frame and the video frame after the starting video frame can be selected in the video, and then the target tracking video composed of the selected video frames is used as the video corresponding to the tracked point. In addition, when there are multiple tracked points and there are at least two tracked points with different starting video frames, when determining the target tracking video, the video frame selection can be based on the starting video frame that is at the front in time sequence.

[0100] Further, in step S22, as Figure 3 As shown, the initial position f of the tracked point in the starting video frame eis the position information corresponding to the tracked point in the starting video frame. The initial position of the tracked point in the starting video frame can be determined according to the determination method of the starting video frame. For example, when the starting video frame is a video frame on which a point selection operation is performed, the initial position of the tracked point in the starting video frame can be based on the position information of the image point on which the point selection operation is performed; when the starting video frame is any video frame in the video, the initial position of the tracked point in the starting video frame is input by the user. Of course, in practical applications, no matter how the starting video frame is determined, the initial position of the tracked point in the starting video frame is input by the user. For example, the position coordinates of the tracked point in the starting video frame input by the user are received to obtain the initial position of the tracked point.

[0101] The initial semantic feature l of the tracked point in the starting video frame e It is the depth feature of the tracked point, which can be obtained by sampling the depth feature map of the starting video frame. Specifically, after the initial position is obtained, sampling can be performed in the depth feature map of the starting video frame based on the initial position to obtain the initial semantic feature. For example, the depth feature corresponding to the initial position is sampled in the depth feature map of the starting video frame by bilinear interpolation to obtain the initial semantic feature of the tracked point; or, the depth feature map can be upsampled first to obtain a feature map with the same image size as the image size of the starting video frame, and then the depth feature corresponding to the initial position is selected in the feature map to obtain the initial semantic feature of the tracked point; or, the depth feature of the initial position is obtained in the depth feature map through the region of interest pooling operation to obtain the initial semantic feature of the tracked point, etc. The present application adopts the initial semantic feature in the depth feature map of the starting video frame based on the initial position, so that the initial semantic feature can accurately describe the depth feature of the tracked point, improves the accuracy of the description of the tracked point by the initial semantic feature in each initial point query, and thus can improve the accuracy of the detected target position.

[0102] It should be noted that when the depth feature map adopts a multi-scale depth feature map, the depth features corresponding to the initial position can be sampled in the depth feature map of each scale respectively, and then all the sampled depth features can be fused (for example, through MLP fusion, etc.) to obtain the initial semantic features of the tracked point. That is, when the depth feature map of the starting video frame includes several depth feature maps of different image scales, the process of determining the initial semantic features of the tracked point in the starting video frame based on the initial position and the depth feature map of the starting video frame can be to sample the depth features corresponding to the initial position in the depth feature map of each image scale respectively; and fuse all the sampled depth features to obtain the initial semantic features of the tracked point. Of course, when the depth feature map of the starting video frame includes a depth feature map of an image scale, the depth features sampled in the depth feature map are directly used as the initial semantic features.

[0103] Further, in step S23, after obtaining the tracked point in the starting video frame, the initial point query (f e , l e ), the initial point query of the tracked point in the starting video frame can be directly used as the initial point query of the tracked point in each video frame in the video, so as to obtain the initial point query of the tracked point in each video frame in the video. That is to say, the initial point query corresponding to each video frame of the tracked point in the video is (f e , l e ), that is, for the video frame t in the video, f e Initialize the initial semantic features f of the tracked point in video frame t t , using l e Initialize the initial position l of the tracked point in video frame t t .

[0104] In addition, in practical applications, after obtaining the tracked point in the starting video frame, the initial point query is recorded as (f e , l e ), other methods may be used to determine the initial point query of the tracked point in each video frame except the starting video frame. For example, the initial point query of the tracked point in each video frame other than the starting video frame may be determined according to the time interval between the video frame and the starting video frame. e An offset is performed to obtain the initial position corresponding to the video frame, that is, the initial point query corresponding to each video frame is not exactly the same, and then based on the initial position corresponding to the video frame, a depth feature map corresponding to the video frame is sampled to obtain the initial semantic features corresponding to the video frame, etc.

[0105] S30, based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, determine the target point query of the tracked point in each video frame through a decoding module to track the tracked point.

[0106] Specifically, when the decoding module updates the initial point query of the tracked point in the video frame, it will perform a vector update on the initial semantic features in the initial point query and the initial position in the initial point query to obtain the target semantic query. In other words, the decoding module determines that the target point query of the tracked point in each video frame is specified by updating the initial point query through the decoding module. When the decoding module updates the initial point query, it will perform a vector update on the initial semantic features in the initial point query to detect the target point query of the tracked point in the video frame. Among them, the target point query includes the target semantic feature f t ′ and target position l t ′, target semantic feature f t ′ represents the depth feature of the tracked point in the depth feature map of the video frame, and the target position l t ′ represents the position information of the tracked point in the video frame, that is, the tracked point in the video frame can be determined based on the target position.

[0107] Exemplarily, based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, determining the target point query of the tracked point in each video frame through the decoding module specifically includes:

[0108] S31, for each video frame, performing at least one position offset sampling in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point;

[0109] S32, determining the attention weight of each position offset sampled visual feature based on each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame;

[0110] S33. Based on the attention weights of the visual features sampled at each position offset, the initial point query of the tracked point in the video frame is updated through the mutual attention mechanism in the decoding module, and the target point query of the tracked point in the video frame is determined according to the updated initial point query.

[0111] Specifically, in step S31, the position offset sampling is used to sample the depth feature of the tracked point after the position offset is performed in the depth feature map, that is, the visual feature of the position offset sampling is the depth feature sampled after the tracked point is position offset. Among them, the position offset of the position offset sampling can be pre-set, for example, at least one position offset is pre-set, and then the position offset sampling is performed according to each position offset. The position offset sampling can also be determined according to the movement law of the segmented area where the tracked point is located, that is, before the position offset sampling is performed in the video frame, the movement law of the segmented area where the tracked point is located is first obtained, and then at least one position offset is set for the tracked point in combination with the movement law of the segmented area where the tracked point is located.

[0112] Exemplarily, performing at least one position offset sampling in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point specifically includes:

[0113] S311, obtaining the segmented area to which the tracked point belongs, and determining the movement pattern of the tracked point based on the segmented area;

[0114] S312. Determine at least one position offset of the tracked point based on the movement rule, and perform position offset sampling in the depth feature map of the video frame according to each position offset and the initial position of the tracked point in the video frame to obtain at least one position offset sampled visual feature corresponding to the tracked point.

[0115] Specifically, the segmented area can be determined by segmenting the video frame on which the point selection operation is performed when selecting the tracked point, or it can be determined by segmenting the starting video frame. The segmented area includes the tracked point, and the movement law of the tracked point has a certain correlation with the movement law of the segmented area to which it belongs. For example, if the segmented area where the tracked point is located moves upward, then the tracked point also moves upward with a large regularity. Therefore, at least one position offset of the tracked point can be determined based on the movement law, which can improve the accuracy of the position offset sampling, and further improve the accuracy of the acquired position offset sampling visual features, and further improve the accuracy of the subsequent target point query.

[0116] Furthermore, the position offset includes an offset in the x direction (i.e., the image width direction) and an offset in the y direction (i.e., the image height direction), that is, the position offset can be expressed as (Δx, Δy), Δx represents the offset in the x direction, and Δy represents the offset in the y direction. Correspondingly, when performing position offset sampling in the depth feature map of the video frame according to each position offset and the initial position of the tracked point in the video frame, the sampling position of the position offset sampling can be first determined based on the position offset and the initial position, and then the depth feature sampling can be performed in the depth feature map based on the sampling position, wherein the sampling position of the position offset sampling can be the sum of the initial position and the position offset. Of course, in practical applications, the sampling position of the position offset sampling can also be obtained in other ways, for example, the initial position and the position offset are weighted to obtain the sampling position of the position offset sampling, etc.

[0117] Further, in step S32, the attention weight is used to reflect the importance of each position offset sampled visual feature in updating the initial semantic feature and / or the initial position update through the mutual attention mechanism, wherein the attention weight can be determined by calculating the similarity between each position offset sampled visual feature and the initial semantic feature, that is, the attention weight of each position offset sampled visual feature is determined based on the initial semantic feature of each position offset sampled visual feature and the tracked point in the video frame as follows:

[0118] The similarity between each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame is calculated respectively to obtain the attention weight of each position offset sampled visual feature.

[0119] Specifically, the similarity between each position offset sampled visual feature and the initial semantic feature can be determined by performing a dot product of the position offset sampled visual feature and the initial semantic feature, and when the feature dimension of the position offset sampled visual feature is different from the feature dimension of the initial semantic feature, the feature dimension of the position offset sampled visual feature can be first adjusted to be consistent with the feature dimension of the initial semantic feature (for example, by MLP adjustment, etc.), and then the similarity between the two can be determined by a dot product method. Of course, in practical applications, other methods can also be used to determine the similarity, such as Euclidean distance, or cosine similarity, etc.

[0120] The embodiment of the present application determines the attention weight by calculating the similarity between the position offset sampled visual features and the initial semantic features, and then updates the initial point query based on the attention weight. In this way, only the similarity between the initial semantic features and the visual features of each position offset sampled need to be calculated, without calculating the similarity between the tracked point and each image point in each video frame, which greatly reduces the amount of similarity calculation, thereby improving the tracking efficiency of point tracking and reducing the requirements of point tracking on hardware resources.

[0121] Further, in step S23, when the initial point query of the tracked point in the video frame is updated by the mutual attention mechanism in the decoding module, the updating can be performed by the decoding layer in the decoding module. Specifically, Figure 3 As shown, the decoding module may include N decoding layers, and the structures of the N decoding layers are exactly the same, N is a positive integer, for example, the decoding layer may adopt the structure of the decoder in the Transformer, etc. Here, one decoding layer is taken as an example for explanation. The decoding layer may adopt a cross attention mechanism to update the initial semantic features and initial positions in the initial point query; it may also adopt a cross attention mechanism to update the initial semantic features, and then update the initial position based on the feedforward network in the decoding layer and update the updated initial semantic features again; it may also adopt a cross attention mechanism to update the initial semantic features and initial positions in the initial point query, and then adopt a feedforward network to update the updated initial semantic features and initial positions again.

[0122] In a first implementation, the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0123] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and initial positions of the tracked points in each video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracked points in the video frame, wherein the updated point query includes an updated semantic feature and an updated position;

[0124] The updated point query of the tracked point in the video frame is used as the target point query of the tracked point in the video frame.

[0125] In a second implementation, the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0126] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain the updated semantic features of the tracked point in the video frame;

[0127] The updated semantic features and the initial position of the tracked point in the video frame are updated through the feedforward network in the decoding module to obtain a target point query of the tracked point in the video frame.

[0128] In a third implementation, the attention weights of the visual features sampled based on each position offset, updating the initial point query of the tracked point in the video frame through the mutual attention mechanism in the decoding module, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically include:

[0129] Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and the initial position of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracking point in the video frame, wherein the updated point query includes an updated semantic feature and an updated position;

[0130] The update point query of the tracked point in the video frame is updated through the feedforward network in the decoding module to obtain the target point query of the tracked point in the video frame.

[0131] In the above three implementations, the initial semantic features are updated through the mutual attention mechanism in decoding, the first implementation and the third implementation involve updating the initial position through the mutual attention mechanism in the decoding module, and the second implementation and the third implementation involve updating the updated semantic features and positions through the feedforward network. Therefore, in order to make the description concise, the following describes respectively updating the initial semantic features through the mutual attention mechanism in the decoding module, updating the initial position through the mutual attention mechanism in the decoding module, and updating the updated semantic features and positions through the feedforward network.

[0132] 1. Update the initial semantic features through the mutual attention mechanism in the decoding module

[0133] Updating the initial semantic features through the mutual attention mechanism in the decoding module refers to interacting the initial semantic features with the deep feature map of the video frame through the mutual attention mechanism to update the initial semantic features by learning the feature information carried by the deep feature map. Therefore, when the initial semantic features are updated through the mutual attention mechanism in the decoding module, a query vector is constructed based on the initial semantic features, a value vector and a key vector are constructed based on this deep feature map, and then the updated semantic features are obtained through learning between the query vector, the value vector and the key vector.

[0134] For example, Figure 4 As shown, the attention weights of the visual features sampled based on each position offset and updating the initial semantic features of the tracked point in the video frame through the mutual attention mechanism in the decoding module specifically include:

[0135] The visual features of the offset samples at each position are weighted based on the attention weights of the visual features of the offset samples at each position to obtain the sampled semantic features;

[0136] The sampled semantic feature is added to the initial semantic feature of the tracked point in the video frame to obtain an updated semantic feature of the tracked point in the video frame.

[0137] Specifically, after obtaining the attention weights of the sampled visual features of each position offset, the sampled visual features of each position offset are weighted by the attention weights of the sampled visual features of each position offset to obtain the sampled semantic features, and then the sampled semantic features and the initial semantic features are added to obtain the updated semantic features.

[0138] 2. Update the initial position through the mutual attention mechanism in the decoding module

[0139] The initial position is updated by the mutual attention mechanism in the decoding module using the similarity determined when the initial semantic features are updated by the mutual attention mechanism in the decoding module as an attention weight, and then weighting each position offset and the initial position based on the attention weight to update the initial position.

[0140] For example, Figure 4 As shown, the updating of the initial position of the tracked point in each video frame through the mutual attention mechanism in the decoding module specifically includes:

[0141] For each video frame, performing position offset sampling of at least one orientation in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one offset sampled visual feature corresponding to the tracked point;

[0142] Calculating the similarity between each offset sampled visual feature and the initial semantic feature of the tracked point in the video frame respectively, so as to determine the attention weight of each offset sampled visual feature;

[0143] Based on the attention weight of the visual features of each offset sample, the position offset corresponding to each position offset sample is weighted to obtain the target position offset;

[0144] Determining an updated position based on the target position offset and the initial position of the tracked point in the video frame. The updating of the initial position of the tracked point in the video frame through the mutual attention mechanism in the decoding module based on the attention weights of the visual features sampled at each position offset specifically includes:

[0145] Based on the attention weight of the visual features of each position offset sample, the position offset corresponding to each position offset sample is weighted to obtain the target position offset;

[0146] The target position offset and the initial position of the tracked point in the video frame are added to obtain an updated position of the tracked point in the video frame.

[0147] Specifically, after obtaining the attention weights of the visual features sampled by each position offset, each position offset is weighted by the attention weights of the visual features sampled by each position offset to obtain a target position offset, and then the target position offset and the initial position are added to obtain an updated position.

[0148] 3. The feedforward network updates the semantic features and positions

[0149] The feedforward network is pre-set and used to update the semantic features and positions, wherein when the initial position is updated through the mutual attention mechanism, the feedforward network updates the updated position again; when the initial position is not updated through the mutual attention mechanism in the decoding module, the feedforward network updates the initial position. The feedforward network can be MPL, etc.

[0150] In addition, in the process of determining the target point query, the updated semantic features can also be processed in time and space, which can improve the receptive field of the updated semantic features on the one hand, and make the updated semantic features contain temporal features on the other hand. That is to say, after passing through the mutual attention mechanism in the decoding module, the updated semantic features can also be processed through the time attention mechanism and / or the space attention mechanism. Based on this, after passing through the mutual attention mechanism in the decoding module, the method also includes:

[0151] The updated semantic features determined by the mutual attention mechanism in the decoding module are updated through the spatiotemporal attention mechanism, or through at least one of the spatial attention mechanism and the temporal attention mechanism.

[0152] Specifically, the spatiotemporal attention mechanism learns the updated semantic features synchronously in time and space. The spatial attention mechanism is used to learn in space, and the temporal attention mechanism is used to learn in time. Among them, learning in space refers to learning the updated semantic features of different tracked points on the same video frame to improve the receptive field, and learning in time refers to learning the updated semantic features of the same tracked point in different video frames to learn the temporal features.

[0153] Based on the above description, the decoding module may include several decoding layers with the same model structure, the decoding layer includes a mutual attention layer, the mutual attention layer is used to update the initial semantic features and initial position in the initial point query; or, includes a mutual attention layer and a feedforward network, the mutual attention layer is used to update the initial semantic features in the initial point query, and / or the initial semantic features and initial position, and the feedforward network is used to perform a secondary update on the initial semantic features updated by the mutual attention layer; or, includes a mutual attention layer and a spatial attention mechanism; or, includes a mutual attention layer and a temporal attention mechanism; or, includes a mutual attention layer, a spatial attention mechanism and a temporal attention mechanism, or, includes a mutual attention layer, a spatial attention mechanism and a feedforward network; or, includes a mutual attention layer, a temporal attention mechanism and a feedforward network; or, includes a mutual attention layer, a spatial attention mechanism, a temporal attention mechanism and a feedforward network. Among them, the mutual attention layer is used to update the initial semantic features and initial position, the self-attention layer is used for spatial interactive learning, and the temporal attention layer is used for temporal interactive learning.

[0154] In a typical implementation of the embodiment of the present application, Figure 4 As shown, each decoding layer includes a mutual attention layer, a self-attention layer, a temporal attention layer and a feedforward network, wherein the mutual attention layer is used to update the initial semantic features and the initial position, the self-attention layer is used for spatial interactive learning, the temporal attention layer is used for temporal interactive learning, and the feedforward network is used to update the second semantic features determined by the temporal attention layer and the updated position determined by the mutual attention layer to obtain the target semantic features and the target position. In addition, it should be noted that when the number of decoding layers is 2 or more decoding layers, the input item of the first decoding layer is the initial point query, and the input item of the second and subsequent decoding layers is the target point query output by the previous decoding layer, that is, the target semantic feature in the target point query output by the previous decoding layer is the initial semantic feature of the next decoding layer, and the target position in the target point query output by the previous decoding layer is the initial position of the next decoding layer, and the output item of the last decoding layer is the target point query corresponding to the tracked point.

[0155] In this typical implementation, the update process of the initial point query can be achieved through a mutual attention mechanism, a spatial attention mechanism, a temporal attention mechanism, and a feedforward network. Specifically, first, the initial semantic features and the initial position are updated through the mutual attention mechanism to obtain updated semantic features and updated positions. Secondly, the updated semantic features corresponding to each video frame are spatially interactively learned through the spatial attention mechanism to obtain a first semantic feature. Thirdly, the first semantic feature and the first semantic feature of the tracked point in other video frames are temporally interactively learned through the temporal attention mechanism to obtain a second semantic feature. Finally, the second semantic feature and the updated position are updated through the feedforward network to obtain the target semantic feature and target position.

[0156] Further, if Figure 4 As shown, the mutual attention layer can be a multilayer perceptron, a sampling offset unit, a dot multiplication unit, a first weighted summation unit, a second weighted summation unit, a first adder, and a second adder, wherein the multilayer perceptron is connected to the sampling offset unit, the sampling offset unit is connected to the dot multiplication unit, the dot multiplication unit is respectively connected to the first weighted summation unit and the second weighted summation unit, the first weighted summation unit is connected to the first adder, and the second weighted summation unit is connected to the second adder. The input item of the multilayer perceptron is the initial semantic feature f t i , t represents the t-th video frame, i represents the i-th tracked point, and the input item of the dot product unit is the initial semantic feature and position offset sampling visual features The input of the first weighted summation unit is the attention weights output by the dot product unit and the visual features of each position offset sampling. The input of the second weighted sum unit is the attention weights and initial positions output by the dot product unit. The input items of the first adder are the sampled semantic features and the initial semantic features determined by the first weighted summing unit, and the input items of the second adder are the target position offset and the initial position determined by the second weighted summing unit.

[0157] The above completes the description of the process of determining the target point query of a tracked point on a video frame. The process of determining the target point query of each tracked point on each video frame is the same, so it will not be described one by one here. In addition, when there are multiple tracked points and the video includes multiple video frames, the multiple tracked points will be tracked synchronously, and each video frame will also be tracked synchronously. In other words, the target point query of each tracked point on all video frames is calculated in parallel, which places high demands on hardware resources. If there are many video frames, it may limit the application scenario of the method for tracking points in a video provided in the embodiment of the present application.

[0158] Therefore, when determining the target point query of each tracked point in each video frame, a sliding window method can be used to process the video frame to achieve video tracking through multiple operations. In each operation, only the video frames contained in the sliding window are tracked to reduce the single calculation amount required for tracking, thereby reducing the requirements for hardware resources.

[0159] Exemplarily, before the query of the target point of the tracked point in each video frame by the decoding module based on the depth feature map of each video frame in the video and the initial point of the tracked point in each video frame is determined to track the tracked point, the method further includes:

[0160] Dividing the video into a plurality of video frame groups according to time sequence, wherein each video frame in the video is included in at least one video frame group;

[0161] Each of the video frame groups is taken as a sub-video to be tracked, and a depth feature map of each video frame in the sub-video to be tracked and an initial point query of the tracked point in each video frame are performed on each sub-video to be tracked, and a target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point.

[0162] Specifically, each of the several video frame groups includes several video frames that are consecutive in time sequence, and each video frame in the video frames is included in at least one video frame group, wherein two adjacent video frame groups in the several video frame groups may not contain the same video frames, or may contain the same video frames. In addition, since each video frame group includes several video frames that are consecutive in time sequence, the several video frame groups also have a time sequence relationship, wherein the time sequence relationship of the several video frame groups can be determined according to the time sequence relationship between the first frames included in each of them. For example, the several video frame groups include video frame group a, video frame group b and video frame group c, video frame group a includes video frame 1, video frame 2, video frame 3 and video frame 4, video frame group b includes video frame 5, video frame 6, video frame 7 and video frame 8, video frame group c includes video frame 9, video frame 10, video frame 11 and video frame 12, then the first frame of video frame group a is video frame 1, the first frame of video frame group b is video frame 5, and the first frame of video frame group c is video frame 9. Accordingly, the time sequence relationship of video frame group a, video frame group b and video frame group c is video frame group a-video frame group b-video frame group c.

[0163] In one implementation, the video is divided into several video frame groups according to the time sequence, which can be achieved by a sliding window method, and the process of determining several video frame groups and the process of using the video frame groups as the sub-video to be tracked to determine the target point query of the tracked point in each video frame of the sub-video to be tracked are performed alternately. Specifically, firstly, the first video frame group is selected in the video according to the time sequence according to the preset window size, and the first video frame group is used as the sub-video to be tracked to query the target point of the tracked point in each video frame of the sub-video to be tracked; then, the sliding window is moved according to the preset number of moving frames to obtain the second video frame, and the second video frame group is used as the sub-video to be tracked to determine the target point query of the tracked point in each video frame of the sub-video to be tracked, and so on, until the target point query of the last frame of the video is obtained. In addition, in order to ensure that each video frame is included in at least one video frame group, the preset number of moving frames will be less than or equal to the preset window size.

[0164] For example: assuming that the preset window size is 8 frames and the preset number of moving frames is 4 frames, then video frame 1-video frame 8 will be taken as the first video frame group, and the first video frame will be taken as the sub-video to be tracked to determine the target point query of the tracked point in each video frame; then, the sliding window will be moved backward by 4 frames, that is, video frame 5-video frame 12 will be taken as the second video frame group, and the second video frame will be taken as the sub-video to be tracked to determine the target point query of the tracked point in each video frame, and so on, until the target point query of the last frame of the video is obtained.

[0165] It should be noted that when processing a video by a sliding window method, if the number of remaining frames cannot meet the size of the sliding window when sliding to the end of the video, the size of the sliding window can be dynamically adjusted so that the number of remaining frames meets the size of the adjusted sliding window. The preset window size can be determined according to the hardware resources used when running the algorithm so that the hardware resources can be fully used.

[0166] Of course, in practical applications, the video may be processed first to obtain a number of video frame groups, and then each video frame group may be used as a sub-video to be tracked to determine the target point query of the tracked point in each video frame. Furthermore, when each video frame group is used as a sub-video to be tracked to determine the target point query of the tracked point in each video frame, each video frame group may be used as a sub-video to be tracked to determine the target point query of the tracked point in each video frame in sequence according to the time sequence of each video frame group, or each video frame group may be used as a sub-video to be tracked to determine the target point query of the tracked point in each video frame in parallel through multiple computing resources.

[0167] Furthermore, when determining the target point query of the tracked point in each video frame of the sub-video to be tracked, the target point query can be determined directly based on the initial point query of the tracked point in the video frame of each sub-video to be tracked, or the initial point query of the tracked point in each video frame of the sub-video to be tracked can be updated based on the target point query determined in the previous sub-video to be tracked, and then the target point query is determined based on the updated initial point query.

[0168] Exemplarily, before performing a query on each sub-video to be tracked based on a depth feature map of each video frame in the sub-video to be tracked and an initial point query of the tracked point in each video frame, and determining a target point query of the tracked point in each video frame by a decoding module to track the tracked point, the method includes:

[0169] Detect whether there is an overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked;

[0170] If there is no overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked, read the first target point query of the last video frame of the tracked point in the previous sub-video to be tracked, and use the first target point query as the initial point query of each video frame of the tracked point in the current sub-video to be tracked;

[0171] If there are overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, read the overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, and update the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frame.

[0172] Specifically, the current sub-video to be tracked and its previous sub-video to be tracked are two video frame groups that are adjacent in time sequence, and the overlapping video frames refer to video frames that are simultaneously included in the current sub-video to be tracked and its previous sub-video to be tracked. When there are no overlapping video frames, it means that there is no corresponding target point query for each video frame of the tracked point in the current sub-video to be tracked, so the first target point query corresponding to the last video frame can be used as the initial point query for each video frame of the tracked point in the current sub-video to be tracked. When there are overlapping video frames, it means that there are corresponding target point queries for some video frames of the tracked point in the current sub-video to be tracked, that is, the process of determining the target point query has been executed for some video frames, and at this time, the initial point query of each video frame of the tracked point in the current sub-video to be tracked can be updated based on the second target point query of the tracked point in the overlapping video frame. In this way, the accuracy of the initial point query can be improved, and the accuracy of the target point query for each video frame of the tracked point in the current sub-video to be tracked can be further improved.

[0173] In one implementation, the updating of the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frames specifically includes:

[0174] For each overlapping video frame in the current sub-video to be tracked, querying the second target point of the tracked point in the overlapping video frame as the initial point query of the tracked point in the overlapping video frame;

[0175] For the non-overlapping video frames in the current sub-video to be tracked, the last overlapping video frame in the overlapping video frames is read in time sequence, and the second target point query of the tracked point in the last overlapping video frame is used as the initial point query of the tracked point in each non-overlapping video frame.

[0176] Specifically, for each overlapping video frame in the current sub-video to be tracked, since there is a target point query for the tracked point in each of the overlapping video frames, the second target point query for the tracked point in each of the overlapping video frames can be directly used as the initial point query to re-determine the target point query, so that the accuracy of the target point query can be improved by iterating the target point query. For non-overlapping video frames in the current sub-video to be tracked, the second target point query for the tracked point in the last overlapping video frame can be used as the initial point query for each overlapping video frame, so that the position information of the tracked point that is closest in time sequence can be used to track each overlapping video frame, which can improve the accuracy of the target point query for the tracked point in each overlapping video frame. Of course, in actual applications, when updating the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frame, other methods can also be used. For example, the second target point query of the last overlapping video frame is used as the initial point query for each video frame in the current sub-video to be tracked, or an overlapping video frame is randomly selected from the overlapping video frames, and the second target point query of the selected overlapping video frame is used as the initial point query for each video frame in the current sub-video to be tracked, etc.

[0177] In order to let the user know more clearly the situation of the tracked point in each video frame, after obtaining the target point query, the visibility of the tracked point in each video frame can be determined based on the target semantic features in the target point query. Accordingly, after the target point query of the tracked point in each video frame is determined by the decoding module based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame to track the tracked point, the method further includes:

[0178] The visibility category of the tracked point in each video frame is determined based on the target semantic features of the tracked point in each video frame, wherein the visibility category is visible or invisible.

[0179] Specifically, the visibility category includes two possibilities: visible and invisible. Visible means that the tracked point is displayed in the video frame, and invisible means that the tracked point is not displayed in the video frame, for example, the tracked point is blocked by other objects. Among them, the visibility category can be determined by a pre-set classifier, such as MLP, Transformer, that is, after obtaining the target point query of the tracked point in the video frame, the target semantic features in the target point query are input into the classifier, and the visibility category of the tracked point in the video frame is determined by the classifier. In other words, the above-mentioned decoding module can be connected to the classifier, and the target point query output by the decoding module will be input into the classifier, and the visibility category of the tracked point in the video frame will be output by the classifier. In addition, after obtaining the visibility category of the tracked point in the video frame, the visibility category and the target position are displayed in a visual manner as the tracking information of the tracked point in the video frame.

[0180] This application will determine the target point query of the tracked point in each video frame. Even if the tracked point is occluded in a certain video frame, this application should continue to detect the target point query of the tracked point in the subsequent video frames of the video frame to predict the target position of the tracked point at the occluded point and its subsequent video frames, so that the tracked point can be detected when the tracked point reappears, avoiding the impact of occlusion on point tracking.

[0181] Exemplarily, the method for tracking points in a video provided by an embodiment of the present application can be implemented by a tracking model, wherein the tracking model can include a feature extraction module, an initial semantic feature construction module, a decoding module, and a classifier, the feature extraction module is connected to the initial semantic feature construction module and the decoding module, the initial semantic feature construction module is connected to the decoding module, and the decoding module is connected to the classifier. Accordingly, the tracking model can be deployed when running the tracking point in the video provided by the embodiment of the present application, and the video and the tracked point specified by the user are received, and then the tracked point is tracked in the video through the tracking module to obtain the tracking information of the tracked point in the video.

[0182] After obtaining the tracking information of the tracked point in the video frame, the tracking information of the tracked point in the video frame can be stored and used as an input item for downstream tasks. For example, the tracking information of the tracked point in the video frame can be used as the input of the SAM model, so that the SAM model outputs the mask of the target object to which the point belongs; it can also be used as the input of the TRex model, so that the TRex model outputs the box of the target to which the point belongs, so as to achieve the tracking of the target object at the mask or box level. The tracked point can also be marked in the video based on the tracking information of the tracked point in the video frame to form a moving trajectory of the tracked point, wherein the moving trajectory refers to the moving trajectory of the tracked point relative to the absolute position of the video interface, rather than the moving trajectory of the relative position in the video.

[0183] In one implementation, after determining the visibility category of the tracked point in each video frame based on the target semantic features of the tracked point in each video frame, the method includes:

[0184] According to the visibility category of the tracked point in each video frame, the tracked point is marked in each video frame based on the target position of the tracked point in each video frame to show the movement trajectory of the tracked point.

[0185] Specifically, the movement trajectory is displayed by marking the tracked point based on the target position of the tracked point in each video frame. That is to say, after obtaining the target point query of the tracked point in each video frame, the image position of the tracked point in each video frame can be determined based on the target position in the target point query, and then the tracked point is marked at the image position. Among them, the tracked point is marked in each video frame according to the visibility category of the tracked point in each video frame, for example, only the visibility category is marked as visible, or, only the visibility category is marked as invisible, or, the visibility category is marked as visible and part of the visibility category is marked as invisible, etc.

[0186] Exemplarily, when marking the tracked point in each video frame according to the visibility category of the tracked point in each video frame, the visibility display condition set by the user can be read, and then the target video frame that meets the visibility display condition is selected according to the visibility category, and then the tracking point is marked in the selected target video frame to display the movement trajectory of the tracked point. Among them, the visibility display condition can be to only mark the visibility category as visible, only mark the visibility category as invisible, or, mark all video frames in the first video frame and the last video frame of the video frame with the visibility category as visible. Of course, in actual applications, the user can set the visibility display condition, and the tracking point can be marked according to the default condition, for example, only mark the visibility category as visible, or mark the tracked point on each video frame.

[0187] In summary, the present embodiment provides a method, apparatus, device and storage medium for tracking points in a video, wherein the method includes selecting a tracked point in a video in response to a point selection operation; constructing an initial point query of the tracked point in each video frame in the video; based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, determining the target point query of the tracked point in each video frame through a decoding module to track the tracked point. The present application constructs an initial point query including semantic features and position information for the tracked point in each video frame, and then updates the initial point query of each video frame in parallel based on the depth feature map of each video frame to obtain a target point query, so that the tracked point is detected from each video frame, so as to avoid the problem that the tracking point cannot be tracked stably when it is blocked. At the same time, the present application can group the video, and each video frame group obtained by grouping is used as a sub-video to be tracked, and then all video frames in each sub-video to be tracked are tracked in parallel in turn, so that the point tracking efficiency can be improved, and the requirements for hardware resources of the point tracking can be reduced.

[0188] Based on the above method for tracking points in a video, this embodiment provides a device for tracking points in a video, such as Figure 5 As shown, the device for tracking points in a video specifically includes:

[0189] A response module 100 is used to respond to a point selection operation and select a tracked point in the video;

[0190] A construction module 200, for constructing an initial point query of the tracked point in each video frame in the video, wherein the initial point query includes an initial semantic feature and an initial position;

[0191] The control module 300 is used to determine the target point query of the tracked point in each video frame through the decoding module based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, so as to track the tracked point, wherein the target point query includes the target semantic feature and the target position.

[0192] Based on the above-mentioned method for tracking points in a video, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method for tracking points in a video as described in the above-mentioned embodiment.

[0193] Based on the above method for tracking points in a video, the present application also provides a terminal device, such as Figure 6As shown, it includes at least one processor 20, a display screen 21, and a memory 22, and may also include a communication interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communication interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0194] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0195] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0196] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0197] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for tracking points in a video, characterized in that: The method for tracking points in a video specifically includes: In response to the point selection operation, select the tracked point in the video; Constructing an initial point query of the tracked point in each video frame in the video, wherein the initial point query includes an initial semantic feature and an initial position; Based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame, the target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point, wherein the target point query includes a target semantic feature and a target position.

2. The method for tracking points in a video according to claim 1, characterized in that: The tracked point is at least one tracked point determined by a user on at least one video frame in the video.

3. The method for tracking points in a video according to claim 1, characterized in that: The constructing of the initial point query of the tracked point in each video frame in the video specifically includes: Reading a starting video frame corresponding to the tracked point; Acquire an initial position of the tracked point in the starting video frame, determine an initial semantic feature of the tracked point in the starting video frame based on the initial position and a depth feature map of the starting video frame, and construct an initial point query of the tracked point in the starting video frame based on the initial semantic feature and the initial position; The initial point query of the tracked point in the starting video frame is used as the initial point query of the tracked point in each video frame in the video, so as to obtain the initial point query of the tracked point in each video frame in the video.

4. The method for tracking points in a video according to claim 3, characterized in that: The starting video frame is the video frame on which the point selection operation is performed, or the starting video frame is any video frame selected by the user in the video.

5. The method for tracking points in a video according to claim 3, characterized in that: The depth feature map of the starting video frame includes a plurality of depth feature maps of different image scales, and the determining of the initial semantic features of the tracked point in the starting video frame based on the initial position and the depth feature map of the starting video frame specifically includes: Sample the depth features corresponding to the initial position in the depth feature map of each image scale respectively; All the sampled deep features are fused to obtain the initial semantic features of the tracked point.

6. The method for tracking points in a video according to claim 1, characterized in that: The querying of the target point of the tracked point in each video frame based on the depth feature map of each video frame in the video and the initial point of the tracked point in each video frame, and determining the target point of the tracked point in each video frame through the decoding module specifically includes: For each video frame, performing at least one position offset sampling in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point; Determining an attention weight of each position offset sampled visual feature based on each position offset sampled visual feature and an initial semantic feature of the tracked point in the video frame; Based on the attention weights of the visual features sampled at each position offset, an initial point query of the tracked point in the video frame is updated through a mutual attention mechanism in the decoding module, and a target point query of the tracked point in the video frame is determined according to the updated initial point query.

7. The method for tracking points in a video according to claim 6, characterized in that: The performing at least one position offset sampling in the depth feature map of the video frame based on the initial position of the tracked point in the video frame to obtain at least one position offset sampling visual feature corresponding to the tracked point specifically includes: Acquire the segmented area to which the tracked point belongs, and determine the movement pattern of the tracked point based on the segmented area; Based on the movement rule, at least one position offset of the tracked point is determined, and position offset sampling is performed in the depth feature map of the video frame according to each position offset and the initial position of the tracked point in the video frame to obtain at least one position offset sampled visual feature corresponding to the tracked point.

8. The method for tracking points in a video according to claim 6, characterized in that: The attention weight of each position offset sampled visual feature is determined based on each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame as follows: The similarity between each position offset sampled visual feature and the initial semantic feature of the tracked point in the video frame is calculated respectively to obtain the attention weight of each position offset sampled visual feature.

9. The method for tracking points in a video according to claim 6, characterized in that: The method of updating the initial point query of the tracked point in the video frame by the mutual attention mechanism in the decoding module based on the attention weights of the visual features of each position offset sampling, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically includes: Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and the initial position of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracked point in the video frame, wherein the updated point query includes an updated semantic feature and an updated position; The updated point query of the tracked point in the video frame is used as the target point query of the tracked point in the video frame.

10. The method for tracking points in a video according to claim 6, characterized in that: The method of updating the initial point query of the tracked point in the video frame by the mutual attention mechanism in the decoding module based on the attention weights of the visual features of each position offset sampling, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically includes: Based on the attention weights of the visual features sampled at each position offset, the initial semantic features of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain the updated semantic features of the tracked point in the video frame; The updated semantic features and the initial position of the tracked point in the video frame are updated through the feedforward network in the decoding module to obtain a target point query of the tracked point in the video frame.

11. The method for tracking points in a video according to claim 6, characterized in that: The method of updating the initial point query of the tracked point in the video frame by the mutual attention mechanism in the decoding module based on the attention weights of the visual features of each position offset sampling, and determining the target point query of the tracked point in the video frame according to the updated initial point query specifically includes: Based on the attention weights of the visual features sampled at each position offset, the initial semantic features and the initial position of the tracked point in the video frame are updated through the mutual attention mechanism in the decoding module to obtain an updated point query of the tracking point in the video frame, wherein the updated point query includes an updated semantic feature and an updated position; The update point query of the tracked point in the video frame is updated through the feedforward network in the decoding module to obtain the target point query of the tracked point in the video frame.

12. The method for tracking a point in a video according to claims 9-11, characterized in that: The attention weights of the visual features sampled based on each position offset, and updating the initial semantic features of the tracked point in the video frame through the mutual attention mechanism in the decoding module specifically include: The visual features of the offset samples at each position are weighted based on the attention weights of the visual features of the offset samples at each position to obtain the sampled semantic features; The sampled semantic feature is added to the initial semantic feature of the tracked point in the video frame to obtain an updated semantic feature of the tracked point in the video frame.

13. The method for tracking points in a video according to claim 9 or 11, characterized in that: The updating of the initial position of the tracked point in the video frame by the mutual attention mechanism in the decoding module based on the attention weights of the visual features sampled at each position offset specifically includes: Based on the attention weight of the visual features of each position offset sample, the position offset corresponding to each position offset sample is weighted to obtain the target position offset; The target position offset and the initial position of the tracked point in the video frame are added to obtain an updated position of the tracked point in the video frame.

14. The method for tracking a point in a video according to any one of claims 9 to 11, characterized in that: After updating the initial semantic features of the tracked point in the video frame by the mutual attention mechanism in the decoding module based on the attention weights of the visual features sampled at each position offset, the method further includes: The updated semantic features are updated through a spatiotemporal attention mechanism, or through at least one of a spatial attention mechanism and a temporal attention mechanism.

15. The method for tracking points in a video according to claim 1, characterized in that: The decoding module includes several decoding layers, and the input items of each decoding layer in the several decoding layers are initial point queries, wherein the initial point query of the first decoding layer is the initial point query of the tracked point in the video frame, and the initial point queries of other decoding layers are the target point queries output by their previous decoding layers.

16. The method for tracking points in a video according to claim 1, characterized in that: Before the method of determining the target point query of the tracked point in each video frame by a decoding module based on the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame to track the tracked point, the method further comprises: Dividing the video into a plurality of video frame groups according to time sequence, wherein each video frame in the video is included in at least one video frame group; Each of the video frame groups is taken as a sub-video to be tracked, and a depth feature map of each video frame in the sub-video to be tracked and an initial point query of the tracked point in each video frame are performed on each sub-video to be tracked, and a target point query of the tracked point in each video frame is determined by a decoding module to track the tracked point.

17. The method for tracking points in a video according to claim 16, characterized in that: Before performing a query on each sub-video to be tracked based on a depth feature map of each video frame in the sub-video to be tracked and an initial point query of the tracked point in each video frame, and determining a target point query of the tracked point in each video frame by a decoding module to track the tracked point, the method includes: Detect whether there is an overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked; If there is no overlapping video frame between the current sub-video to be tracked and the previous sub-video to be tracked, read the first target point query of the last video frame of the tracked point in the previous sub-video to be tracked, and use the first target point query as the initial point query of each video frame of the tracked point in the current sub-video to be tracked; If there are overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, read the overlapping video frames between the current sub-video to be tracked and the previous sub-video to be tracked, and update the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frame.

18. The method for tracking points in a video according to claim 17, characterized in that: The updating of the initial point query of the tracked point in each video frame in the current sub-video to be tracked based on the second target point query of the tracked point in the overlapping video frames specifically includes: For each overlapping video frame in the current sub-video to be tracked, querying the second target point of the tracked point in the overlapping video frame as the initial point query of the tracked point in the overlapping video frame; For the non-overlapping video frames in the current sub-video to be tracked, the last overlapping video frame in the overlapping video frames is read in time sequence, and the second target point query of the tracked point in the last overlapping video frame is used as the initial point query of the tracked point in each non-overlapping video frame.

19. The method for tracking points in a video according to claim 1, characterized in that: After the depth feature map of each video frame in the video and the initial point query of the tracked point in each video frame are determined by a decoding module to query the target point of the tracked point in each video frame to track the tracked point, the method further includes: The visibility category of the tracked point in each video frame is determined based on the target semantic features of the tracked point in each video frame, wherein the visibility category is visible or invisible.

20. The method for tracking points in a video according to claim 19, characterized in that: After determining the visibility category of the tracked point in each video frame based on the target semantic features of the tracked point in each video frame, the method includes: According to the visibility category of the tracked point in each video frame, the tracked point is marked in each video frame based on the target position of the tracked point in each video frame to show the movement trajectory of the tracked point.

21. A device for tracking points in a video, characterized in that The device for tracking points in a video specifically includes: The response module is used to respond to the point selection operation and select the tracked point in the video; A construction module, used to construct an initial point query of the tracked point in each video frame in the video, wherein the initial point query includes an initial semantic feature and an initial position; A control module is used to determine a target point query of the tracked point in each video frame through a decoding module based on a depth feature map of each video frame in the video and an initial point query of the tracked point in each video frame to track the tracked point, wherein the target point query includes a target semantic feature and a target position.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the method for tracking points in a video as described in any one of claims 1-20.

23. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the processor implements the steps in the method for tracking points in a video as described in any one of claims 1 to 20.

Images