Trajectory tracking method

Abstract

The embodiment of the present application relates to the field of computer vision and discloses a trajectory tracking method, which includes: obtaining multiple image frames in a video sequence; performing feature extraction on the multiple image frames according to a detection module in a target trajectory tracking model to obtain feature data of the multiple image frames; processing the feature data of the multiple image frames according to a screening module in the target trajectory tracking model to screen at least two first image frames to obtain at least one second image frame; processing the at least one second image frame and the current image frame according to a processing module in the target trajectory tracking model to obtain a trajectory tracking result of the target object. The application of the technical solution of the present application can solve the problem of target object tracking errors or tracking failures caused by occlusion and other conditions during the trajectory tracking process of the target object.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and specifically to a trajectory tracking method. Background Art

[0002] Multi-target tracking is one of the important technologies in computer vision. It can identify and track multiple target objects in a scene. It has a wide range of applications in practical scenarios in life, such as video surveillance, autonomous driving, and robot navigation.

[0003] For video scenes, the shape, direction or angle of the target object (such as a person) may change at any time during movement, and occlusion may occur between different target objects. This poses a great challenge to the trajectory tracking of the target object and may lead to tracking errors or tracking failures of the target object. Summary of the Invention

[0004] In view of the above problems, an embodiment of the present application provides a trajectory tracking method for solving the problem of target object tracking error or tracking loss caused by changes in the target object or mutual occlusion between target objects during the tracking process of the target object.

[0005] According to a first aspect of an embodiment of the present application, a trajectory tracking method is provided, the method comprising: acquiring multiple image frames in a video sequence; the multiple image frames include a current image frame and at least two first image frames. According to a detection module in a target trajectory tracking model, feature extraction is performed on the multiple image frames to obtain feature data of the multiple image frames. According to a screening module in the target trajectory tracking model, the feature data of the multiple image frames are processed, and at least one second image frame is screened from at least two first image frames to obtain; wherein the similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity, and the second image frame includes a target object. According to a processing module in the target trajectory tracking model, the at least one second image frame and the current image frame are processed to obtain a trajectory tracking result of the target object.

[0006] In an optional manner, the screening module includes a graph convolutional neural network; according to the screening module in the target trajectory tracking model, the feature data of multiple image frames are processed, and at least one second image frame is obtained by screening at least two first image frames, including: processing the feature data of multiple image frames according to the graph convolutional neural network to obtain the cosine similarity between each first image frame and the current image frame; according to the size relationship between the cosine similarity between each first image frame and the current image frame and the first preset similarity, at least one second image frame is obtained from at least two first image frames.

[0007] In an optional manner, at least one second image frame is obtained by screening at least two first image frames based on a magnitude relationship between a cosine similarity between each first image frame and the current image frame and a first preset similarity, including: if the cosine similarity between the first image frame and the current image frame is greater than or equal to the first preset similarity, then the first image frame is determined as the second image frame.

[0008] In an optional manner, the processing module in the target trajectory tracking model includes a target similarity learning module and an association module; according to the processing module in the target trajectory tracking model, at least one second image frame and the current image frame are processed to obtain a trajectory tracking result of the target object, including: according to the target similarity learning module in the processing module, calculating the similarity between the target object in each second image frame and the target object in the current image frame to obtain a similarity matrix; according to the association module and the similarity matrix in the processing module, processing the similarity matrix to obtain a trajectory tracking result of the target object.

[0009] In an optional manner, according to the target similarity learning module in the processing module, the similarity between the target object in each second image frame and the target object in the current image frame is calculated to obtain a similarity matrix, including: according to the target similarity learning module, using an attention mechanism to obtain feature data of the target object in each second image frame and feature data of the target object in the current image frame; according to the feature data of the target object in each second image frame and the feature data of the target object in the current image frame, determining the similarity between the target object in each second image frame and the target object in the current image frame to obtain a similarity matrix.

[0010] In an optional manner, the similarity matrix is ​​processed according to the association module in the processing module to obtain a trajectory tracking result of the target object, including: according to the association module, using the Hungarian matching algorithm to process the similarity matrix to determine the matching result of the target object in the second image frame and the target object in the current image frame; according to the identification information of the target object, the matching results are associated in time sequence to obtain the trajectory tracking result of the target object; wherein the identification information of the target object is obtained by the detection module.

[0011] In an optional embodiment, the method further includes: obtaining a training sample set, the training sample set comprising multiple sample image frames in a sample video sequence and the type of target object in each first sample image frame; wherein the multiple sample image frames include a current sample image frame and at least two first sample images; performing feature extraction on the multiple sample image frames according to a detection module of an initial trajectory tracking model to obtain feature data for the multiple sample image frames; processing the feature data of the multiple sample image frames according to a screening module of the initial trajectory tracking model to screen at least one second sample image frame from the at least two first sample image frames; wherein the similarity between the second sample image frame and the current sample image frame is greater than or equal to a first preset similarity, and the second sample image frame includes the target object; calculating the similarity of the target object in at least one sample image frame group according to an initial similarity learning module of the initial trajectory tracking model to obtain a predicted similarity; wherein the sample image frame group includes the second sample image frame and the current image frame; using the predicted similarity as the initial training output of the initial similarity learning module, and using the type of the target object in each sample image frame group as supervisory information, iteratively training the initial similarity learning module to obtain the target similarity learning module.

[0012] In an optional manner, the predicted similarity is used as the initial training output of the initial similarity learning module, the type of the target object in each sample image frame group is used as supervision information, and the initial similarity learning module is iteratively trained to obtain a target similarity learning module, including: determining a loss value based on the predicted similarity and the type of the target object in the sample image frame group; iteratively updating the initial similarity learning module based on the loss value until the target similarity learning module that meets preset conditions is obtained; wherein the preset conditions include the loss value being less than or equal to a first preset threshold.

[0013] In an optional manner, a loss value is determined based on the predicted similarity and the type of target objects in the sample image frame group, including: if the types of target objects in the sample image frame group are the same, determining the loss value based on the predicted similarity; if the types of target objects in the sample image frame group are different, and the predicted similarity is less than or equal to a second preset similarity, determining the loss value as a second preset threshold, and the second preset threshold is less than or equal to the first preset threshold; if the types of target objects in the sample image frame group are different, and the predicted similarity is greater than the second preset similarity, updating the predicted similarity to the second preset similarity, and determining the loss value based on the second preset similarity.

[0014] In an optional way, the loss value is obtained by the following formula:

[0015]

[0016] Among them, L is the loss value, D is the predicted Euclidean distance of the target object in the sample image frame group, m is the second preset similarity, and Y indicates whether the types of the target objects in the sample image frame group are the same or different. When the types of the target objects in the sample image frame group are the same, Y is 0; when the types of the target objects in the sample image frame group are different, Y is 1.

[0017] According to the second aspect of the embodiment of the present application, a trajectory tracking device is provided. The device includes: an image acquisition module for acquiring multiple image frames in a video sequence; the multiple image frames include a current image frame and at least two first image frames. An image detection module for extracting features from the multiple image frames according to the detection module in the target trajectory tracking model to obtain feature data of the multiple image frames. An image screening module for processing the feature data of the multiple image frames according to the screening module in the target trajectory tracking model, and screening at least one second image frame from at least two first image frames; wherein the similarity between the second image frame and the current image frame is greater than or equal to the first preset similarity, and the second image frame includes the target object. An image processing module for processing the at least one second image frame and the current image frame according to the processing module in the target trajectory tracking model to obtain a trajectory tracking result of the target object.

[0018] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus; the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the trajectory tracking method as described in any one of the above-mentioned first aspects.

[0019] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores at least one executable instruction. When the executable instruction is executed on an electronic device, the electronic device performs the operation of the trajectory tracking method as described in any one of the first aspects above.

[0020] The embodiment of the present application constructs a target trajectory tracking model, which includes a block detection module, a screening module and a processing module. First, a plurality of image frames in a video sequence are obtained, wherein the plurality of image frames include at least two first image frames and a current image frame; then, according to the detection module, feature extraction is performed on the plurality of image frames to obtain feature data of the plurality of image frames; then, according to the screening module, the feature data of the plurality of image frames are processed, and at least one second image frame is screened out from at least two first image frames, wherein the second image frame includes a target object, and the similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity; therefore, after processing by the screening module, invalid frames that do not include the target object (for example, the target object is blocked) in at least two first image frames can be filtered out, and valid frames (i.e., second image frames) that include the target object are retained; finally, the at least one second image frame obtained after screening and the current image frame are processed by the processing module to obtain a trajectory tracking result of the target object. Therefore, the embodiment of the present application filters multiple first image frames adjacent to the current image frame, discards invalid frames that do not contain the target object, and only retains valid frames that contain the target object, and obtains the trajectory tracking results of the target object by further processing the filtered valid frames, thereby improving the efficiency and accuracy of target object trajectory tracking.

[0021] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0023] Figure 1 A flow chart of a trajectory tracking method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a target trajectory tracking model provided in an embodiment of the present application;

[0025] Figure 3A A schematic diagram of a graph convolution provided in an embodiment of the present application;

[0026] Figure 3B A schematic diagram of another graph convolution provided in an embodiment of the present application;

[0027] Figure 4A schematic diagram of another target trajectory tracking model provided in an embodiment of the present application;

[0028] Figure 5 A flowchart of another trajectory tracking method provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a target similarity learning module provided in an embodiment of the present application;

[0030] Figure 7 A flowchart of another trajectory tracking method provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of a trajectory tracking device provided in an embodiment of the present application;

[0032] Figure 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0034] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0035] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0036] The terms "comprises" and "comprising" and any variations thereof in this application are intended to cover but not exclude inclusion. For example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed, but may include other components not expressly listed or inherent to such products or devices.

[0037] The term "and / or" in this application simply describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0038] Multi-target tracking algorithms have important and widespread applications in various visual scenarios. In some examples, they can track the trajectory of target objects by first detecting the target and then associating the data. For example, by calculating the similarity between consecutive image frames, the degree of association between target objects can be determined. Based on this association, target objects can be matched in consecutive image frames to obtain the target object's trajectory tracking result.

[0039] In some examples, target detection can be performed on multiple consecutive upper and lower image frames in a video frame, and then data association can be performed. Specifically, in the target detection stage, the category and position information of the target object in the consecutive upper and lower image frames can be obtained; in the data association stage, a Kalman filter can be used to predict the position of the target object in the next image frame based on the position of the target object in the current image frame to obtain a prediction frame, and then the intersection over union (IOU) between the prediction frame and the detection frame of the target object in the current image frame is calculated to generate a cost matrix (assignment cost matrix). Finally, the Hungarian matching algorithm and the cost matrix are used to complete the matching of the target object and form the tracking result of the target object.

[0040] According to the above embodiments, in the process of tracking the trajectory of a target object, multiple consecutive image frames in a video sequence, including the current image frame, are typically processed, and then the trajectory tracking of the target object in the current image frame is obtained based on the processing results. However, for video scenes, the target objects in the image frames, such as pedestrians, often undergo changes in appearance, direction of travel, and angle during movement. In addition, occlusion may occur between different target objects. These factors pose significant challenges to subsequent data association to obtain the trajectory tracking results of the target object, and may result in target object tracking failure or tracking loss.

[0041] For example, among x consecutive image frames, such as frame 1, frame 2, frame 3, and so on (xth frame) (x is an integer greater than or equal to 2), frame 2 is the current image frame. If the target object in frame 3 cannot be detected due to occlusion, then using frame 3 as the input for data association may result in target object trajectory tracking failure because the target object is not included in frame 3. Furthermore, since frame 3 does not include the target object, using frame 3 as the input for data association will increase the computational complexity during target object trajectory tracking, affecting the system's operating speed.

[0042] In order to solve the problem of target object tracking errors and tracking failures caused by the randomness of the target object's activities and the possibility that the target object may be obscured during the tracking process of the above-mentioned target object (such as a pedestrian), an embodiment of the present application proposes a trajectory tracking method, which calculates the correlation between the current image frame and each other image frame other than the current image frame in multiple continuous image frames, and filters the multiple continuous image frames according to the relationship between the correlation calculation result and the preset correlation, retaining the valid image frames containing the target object (the target object is not obscured), which may be discontinuous in time sequence, and performing subsequent data association based on these filtered valid image frames to obtain the trajectory tracking result of the target object. The trajectory tracking method provided by the embodiment of the present application can improve the efficiency of target object trajectory tracking, and can also avoid the problem of target object tracking failure caused by occlusion and the like.

[0043] Figure 1 A flow chart of a trajectory tracking method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the trajectory tracking method may include steps S110 to S140 as shown below.

[0044] Step S110: Acquire multiple image frames in a video sequence.

[0045] Exemplarily, the multiple image frames include a current image frame and at least two first image frames. Specifically, the multiple image frames in the video sequence may be multiple temporally consecutive image frames, wherein the current image frame may be any image frame among the image frames, and the first image frames may be image frames other than the current image frame among the multiple image frames.

[0046] In some examples, the first image frame may include P (P is an integer greater than 1) image frames before the current image frame; or the first image frame may include Q (Q is an integer greater than 1) image frames after the current image frame; or the first image frame may include P (P is an integer greater than 1) image frames before the current image frame and Q (Q is an integer greater than 1) image frames after the current image frame. This application is not limited to this. For example, the number of first image frames can be pre-set according to needs.

[0047] For example, the multiple image frames include 7 image frames, namely: the 1st frame image, the 2nd frame image, the 3rd frame image, the 4th frame image, the 5th frame image, the 6th frame image and the 7th frame image; among them, the 1st frame image, the 2nd frame image, the 3rd frame image... and the 7th frame image are continuous in time; if the 4th frame image is the current image frame, the first image frame can be the 2 frame images before the 4th frame image and the 2 frame images after the 4th frame image; that is, the first image frame includes 4 image frames of the 2nd frame image, the 3rd frame image, the 5th frame image and the 6th frame image.

[0048] Exemplarily, the current image frame includes at least one target object, and the first image frame may or may not include the target object. In some examples, when the first image frame includes a target object, the target object may be completely identical to, partially identical to, or completely different from the target object in the current image frame. Specifically, the types of the target objects in the first image frame and the current image frame may be the same or different. The types of target objects may include pedestrians, vehicles, buildings, etc.

[0049] For example, when the 4th image frame (current image frame) includes target object a and target object b, the 2nd image frame (first image frame) may include target object a, or target object b, or target object a and target object b, or target object c (target object c is different from both target object a and target object b).

[0050] In another example, the number of target objects in the current image frame may be the same as or different from the number of target objects in the first image frame. For example, the fourth image frame (the current image frame) may include two target objects, the second image frame (the first image frame) may include three target objects, and the third image frame (another first image frame) may include one target object.

[0051] It should be noted that the target objects in each first image frame may be the same or different, and this application does not limit this.

[0052] After acquiring multiple image frames, the multiple image frames may be input into a target trajectory tracking model for further processing.

[0053] Figure 2 A schematic diagram of a target trajectory tracking model provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the target trajectory tracking model 1 may include a detection module 10, a screening module 20 and a processing module 30. Figure 2 , continue to describe steps S120 to S140 in the trajectory tracking method provided in the embodiment of the present application.

[0054] Step S120 : performing feature extraction on the plurality of first image frames according to the detection module in the target trajectory tracking model to obtain feature data of the plurality of image frames.

[0055] In some embodiments, the detection module 10 may include a Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN). PRB-FPN has a bidirectional (e.g., top-down and bottom-up) fusion function, which can detect large and small objects with high precision at one time. At the same time, through bottom-up fusion, PRB-FPN can recover lost information from feature maps of lower layers, thereby retaining richer contextual information. Therefore, PRB-FPN can more accurately extract feature data of the first image frame.

[0056] For example, the detection module 10 may obtain identification information of the target object in the first image frame. The identification information may include type information, identity document (ID) information, and location information of the target object. The ID information of the target object may uniquely identify the target object; the location information of the target object may be, for example, the location coordinate data of the target object in the image frame.

[0057] After the target detection module 10 obtains the feature data of the multiple image frames, the feature data of the multiple image frames may be sent to the screening module 20 .

[0058] Step S130 : processing feature data of a plurality of image frames according to a screening module in the target trajectory tracking model, and screening at least two first image frames to obtain at least one second image frame.

[0059] As described in the above embodiment, if continuous image frames in time sequence are used to track the trajectory of the target object, the image frame with occlusion does not include relevant information of the target object, or the image frame with occlusion and its nearby continuous frames include less effective information about the target object, and using these continuous image frames to track the trajectory of the target object will result in target object tracking failure or tracking errors.

[0060] For example, in multiple image frames, a target object a exists in the first frame image, but is blocked in the second to fifth frames images. Therefore, the second to fifth frames images cannot provide much effective information for trajectory tracking of target object a. If the second to fifth frames images are used to track the trajectory of target object a, the trajectory tracking of target object a may be misled, which not only causes the target object tracking to fail, but also increases the computational complexity of the system and reduces the efficiency of target tracking.

[0061] Exemplarily, image frames that include the target object or can provide valid information for the trajectory tracking of the target object can be screened out from multiple (i.e., at least two) first image frames as valid image frames for tracking the trajectory of the target object. The screened valid image frames may be discontinuous in time but close in time. Specifically, the screening module 20 can filter out the first image frames that are continuous in time but do not include the target object or cannot provide valid information for the trajectory tracking of the target object among the multiple first image frames, and retain the first image frames that may be discontinuous in time but contain the target object or can provide valid information for the trajectory tracking of the target object as the second image frames to improve the accuracy and efficiency of the trajectory tracking of the target object.

[0062] In some embodiments, the similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity, and the second image frame includes the target object.

[0063] The second image frame including the target object includes: there is a target object in the second image frame, or the second image frame includes information that can provide useful information for tracking the trajectory of the target object. Since the second image frame is a valid image frame obtained after screening from multiple first image frames, the second image frame is part or all of the first image frame. Specifically, the second image frame can be at least two discontinuous (adjacent) image frames among the multiple first image frames, or at least two continuous image frames among the multiple first image frames. For example, the number of second image frames can be less than or equal to the number of first image frames.

[0064] In some embodiments, the screening module 20 includes a graph convolutional neural network (GCN), wherein GCN has the characteristics of neighbor aggregation and information transmission, and the correlation between each node can be obtained through GCN.

[0065] Figure 3A A schematic diagram of a graph convolution provided in an embodiment of the present application, referring to Figure 3A , one of the nodes (i.e. Figure 3A The circle in the middle) can be the feature data of an image frame, and the correlation between the image frames can be calculated based on Figure 3A The weights assigned to the edges connecting the nodes in the .

[0066] Exemplarily, the similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity. Specifically, if the similarity between the first image frame and the current image frame is small, such as less than the first preset similarity, it indicates that the target object in the first image frame has a small correlation with the target object in the current image frame, that is, the target object in the first image frame may be obscured, or the first image frame cannot provide useful information for tracking the trajectory of the target object. In this case, the first image frame can be filtered out, that is, the first image frame cannot be used as a valid image frame to track the trajectory of the target object. If the similarity between the first image frame and the current image frame is large, such as greater than or equal to the first preset similarity, it indicates that the target object in the first image frame has a large correlation with the target object in the current image frame, that is, the first image frame may include the target object, or the first image frame can provide useful information for tracking the trajectory of the target object. In this case, the first image frame can be retained and determined as the second image frame, that is, the first image frame can be used as a valid image frame to track the trajectory of the target object.

[0067] Therefore, by retaining valid discontinuous image frames for target object trajectory tracking, the accuracy of target object trajectory tracking is improved, and the calculation of a large number of invalid continuous frames is reduced, thereby improving the operation speed and efficiency of target object trajectory tracking.

[0068] In some embodiments, according to the screening module 20 in the target trajectory tracking model, feature data of multiple image frames are processed to screen at least two first image frames to obtain at least one second image frame, specifically including:

[0069] The feature data of multiple image frames are processed according to a graph convolutional neural network to obtain the cosine similarity between each first image frame and the current image frame; and at least one second image frame is obtained by screening from at least two first image frames based on the size relationship between the cosine similarity between each first image frame and the current image frame and a first preset similarity.

[0070] In some embodiments, if the cosine similarity between the first image frame and the current image frame is greater than or equal to a first preset similarity, the first image frame is determined as the second image frame.

[0071] If the cosine similarity between the first image frame and the current image frame is less than a first preset similarity, the first image frame is discarded.

[0072] Exemplarily, the similarity between each first image frame and the current image frame may be determined by the cosine similarity, or cosine distance, between feature data of the first image frame and feature data of the current image frame.

[0073] In some examples, the cosine similarity may range from [-1, 1], where the closer the cosine similarity between the first image frame and the current image frame is to 1, the more similar the first image frame and the current image frame are. In other words, the greater the cosine similarity between the first image frame and the current image frame, the more similar the first image frame and the current image frame are.

[0074] In some examples, the first preset similarity may include a preset cosine similarity, and the preset cosine similarity may be preset according to actual needs. For example, the preset cosine similarity may be set to 0.8. When the cosine similarity between the first image frame and the current image frame is greater than or equal to 0.8, it indicates that the similarity between the first image frame and the current image frame meets the preset conditions and is relatively similar, so the first image frame can be determined as the second image frame, that is, the valid frame. When the cosine similarity between the first image frame and the current image frame is less than 0.8, it indicates that the first image is not similar to the current image frame. In this case, the first image frame may affect the trajectory tracking of the target object. Therefore, the first image frame can be determined as an invalid frame and discarded.

[0075] Figure 3B A schematic diagram of another graph convolution provided in an embodiment of the present application, referring to Figure 3B , the digital sequence in each node represents an image frame. For example, the current image frame is the 6th image frame, the first image frame is the 3 image frames before the 6th image frame (such as the 3rd image frame, the 4th image frame and the 5th image frame), and the 3 image frames after the 6th image frame (such as the 7th image frame, the 8th image frame and the 9th image frame), and the preset cosine similarity is 0.8; Figure 3B As shown, the edges between each first image frame and the current image frame may represent the cosine similarity (ie, weight) between each first image frame and the current image frame.

[0076] like Figure 3BAs shown, since the cosine similarities between the third image frame, the fifth image frame, and the ninth image frame and the sixth image frame are respectively greater than or equal to the preset cosine similarity of 0.8, the third image frame, the fifth image frame, and the ninth image frame can be determined as the second image frame. Since the cosine similarities between the fourth image frame, the seventh image frame, and the eighth image frame and the sixth image frame are all less than the preset cosine similarity of 0.8, the fourth image frame, the seventh image frame, and the eighth image frame can be discarded. Therefore, after screening, the image frames obtained include the third image frame, the fifth image frame, the ninth image frame (the first image frame), and the sixth image frame (the current image frame).

[0077] Therefore, after multiple image frames are processed by the screening module 20, at least one second image frame containing the target object can be screened out from at least two first image frames. That is to say, the screening module 20 filters out the image frames in which the target object is occluded in the first image frames, or other image frames that cannot provide useful information for the trajectory tracking of the target object. The obtained second image frames contain the target object, or can provide effective information for the trajectory tracking of the target object. Therefore, the second image frames obtained after screening can improve the efficiency and accuracy of the target object trajectory tracking.

[0078] Step S140 : Processing the at least one second image frame and the current image frame according to the processing module in the target trajectory tracking model to obtain a trajectory tracking result of the target object.

[0079] Reference Figure 2 After the screening module 20 obtains at least one second image frame and the current image frame, the at least one second image frame and the current image frame can be input into the processing module 30, and the processing module 30 processes the at least one second image frame and the current image frame to obtain the trajectory tracking result of the target object.

[0080] Figure 4 A schematic diagram of another target trajectory tracking model provided in an embodiment of the present application. In some embodiments, referring to Figure 4 The processing module 30 may include a target similarity learning module 31 and an association module 32 .

[0081] In some embodiments, according to the processing module in the target trajectory tracking model, processing the at least one second image frame and the current image frame to obtain the trajectory tracking result of the target object specifically includes:

[0082] According to the target similarity learning module 31 in the processing module 30, the similarity between the target object in each second image frame and the target object in the current image frame is calculated to obtain a similarity matrix; according to the association module 32 in the processing module 30 and the similarity matrix, the similarity matrix is ​​processed to obtain the trajectory tracking result of the target object.

[0083] Therefore, the embodiment of the present application constructs a target trajectory tracking model, which includes a block detection module, a screening module and a processing module. First, a plurality of image frames in a video sequence are obtained, wherein the plurality of image frames include at least two first image frames and a current image frame; then, according to the detection module, feature extraction is performed on the plurality of image frames to obtain feature data of the plurality of image frames; then, according to the screening module, the feature data of the plurality of image frames are processed, and at least one second image frame is screened out from at least two first image frames, wherein the second image frame includes the target object, and the similarity between the second image frame and the current image frame is greater than or equal to the first preset similarity; therefore, after processing by the screening module, invalid frames that do not include the target object (for example, the target object is blocked) in at least two first image frames can be filtered out, and valid frames (i.e., second image frames) that include the target object are retained; finally, the at least one second image frame and the current image frame obtained after screening are processed by the processing module to obtain the trajectory tracking result of the target object. Therefore, the embodiment of the present application filters multiple first image frames adjacent to the current image frame, discards invalid frames that do not contain the target object, and only retains valid frames that contain the target object, and obtains the trajectory tracking results of the target object by further processing the filtered valid frames, thereby improving the efficiency and accuracy of target object trajectory tracking.

[0084] Figure 5 A flowchart of another trajectory tracking method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes steps S510 to S550.

[0085] Step S510: Acquire multiple image frames in a video sequence.

[0086] Step S520 : performing feature extraction on the multiple image frames according to the detection module in the target trajectory tracking model to obtain feature data of the multiple image frames.

[0087] Step S530 : processing feature data of the plurality of image frames according to the screening module in the target trajectory tracking model, and screening at least two first image frames to obtain at least one second image frame.

[0088] It should be noted that steps S510 to S530 are similar to steps S110 to S130 in the above embodiment, and will not be described again here to avoid repetition.

[0089] Step S540 : Calculate the similarity between the target object in each second image frame and the target object in the current image frame according to the target similarity learning module in the processing module to obtain a similarity matrix.

[0090] It should be noted that when there are multiple second image frames, the similarity of the target objects in any two of the second image frames can also be calculated to obtain a similarity matrix. The process of calculating the similarity of the target objects in any two of the second image frames is similar to the process of calculating the similarity between the target object in the second image frame and the target object in the current image frame. To avoid repetition, the embodiment of the present application uses the calculation of the similarity between the target object in the second image frame and the target object in the current image frame as an example for schematic description.

[0091] Step S550: Process the sum similarity matrix according to the association module in the processing module to obtain the trajectory tracking result of the target object.

[0092] In the at least one second image frame acquired through step S530, due to filtering out some invalid frames in the first image frame, the retained at least one second image frame may be discontinuous in time and only adjacent in time. For example, in the third image frame, the fourth image frame, the fifth image frame, the sixth image frame (current image frame), the seventh image frame, the eighth image frame, and the ninth image frame, the fourth image frame, the seventh image frame, and the eighth image frame are filtered out as invalid frames, and the third image frame, the fifth image frame, and the eighth image frame are retained as the second image frame along with the sixth image frame. Among the second image frames, the fifth image frame and the sixth image frame are temporally continuous, while the third image frame and the eighth image frame are temporally discontinuous but temporally adjacent.

[0093] After obtaining at least one second image frame, the motion approximation matrix and the appearance approximation matrix between at least one second image frame and the current image frame, as well as the second image frames, can be calculated. Among them, the target appearance features are crucial to ensure the robustness of the tracking algorithm. In some examples, the target appearance features can be extracted by a convolutional neural network and then applied to subsequent data association. In order to further distinguish different target objects more clearly and obtain more discriminative target appearance features, such as Figure 4 As shown, the embodiment of the present application adopts the target similarity learning module 31 to obtain the similarity between the target object in the second image frame and the target object in the current image frame.

[0094] Figure 6 A schematic diagram of a target similarity learning module provided in an embodiment of the present application.

[0095] In some embodiments, as Figure 6As shown, the target similarity learning module 31 may include a transformer layer 311. Exemplarily, the transformer layer 311 includes an attention mechanism, such as a self-attention mechanism. The transformer can process temporally consecutive or adjacent image frames, providing better temporal correlation and enabling the learning of more discriminative similarities between different target objects.

[0096] In some examples, reference Figure 6 The target similarity learning module 31 may further include a fully connected (FC) layer 312. Specifically, the attention mechanism in the transformer layer 311 extracts features of the target object in the filtered valid image frames (e.g., the valid image frames include the second image frame and the current image frame); the fully connected layer 312 fuses the features of the target object extracted by the attention mechanism in the transformer layer.

[0097] In some embodiments, the target similarity learning module in the processing module calculates the similarity between the target object in each second image frame and the target object in the current image frame to obtain a similarity matrix, including:

[0098] According to the target similarity learning module, an attention mechanism is adopted to obtain the feature data of the target object in each second image frame and the feature data of the target object in the current image frame; based on the feature data of the target object in each second image frame and the feature data of the target object in the current image frame, the similarity between the target object in each second image frame and the target object in the current image frame is determined to obtain a similarity matrix.

[0099] In some examples, after the second image frame and the current image frame are input into the similarity learning module 31, a transformer is used as a decoder to extract features of the target object in the two second image frames. The parameters of the transformer are shared by the second image frame and the current image frame. After feature extraction, a fully convolutional neural network can be used to fuse the features.

[0100] Specifically, the features of the target object can be learned through the attention mechanism, and the attention mechanism can satisfy Formula 1 as shown below.

[0101]

[0102] The feature extraction process of the transformer layer 311 is described below by taking the input second image frame and the current image frame as examples.

[0103] In some examples, first, the image data of the second image frame and the image data of the current image frame are multiplied by a preset matrix W to obtain an embedding vector (embedding), wherein the preset matrix W can be an initialization matrix, and the embodiments of the present application do not limit the preset matrix W matrix. Then, the embedding enters the attention mechanism and is multiplied by three preset matrices respectively to obtain three different vectors Q, K, and V; wherein the three preset matrices can be different initialization matrices. In Formula 1, d can represent the dimension (dimention) of vector Q and vector K. In some examples, the dimensions of vectors Q, K, and V can be the same, such as d. The result of multiplying vector Q and vector K (such as the transpose of vector K) will increase as the dimension d increases. Therefore, the product of vector Q and vector K can be divided by dimension d, and the product of vector Q and vector K can be normalized. Then, the normalized result is input into the function softmax, the normalized result can be converted into a probability distribution, and the probability distribution result is further multiplied with vector V. After adding the results of the above operations on multiple vectors Q, vector K and vector V, the attention learning result can be obtained, that is, the correlation between the target object in the second image frame and the target object in the current image frame can be obtained.

[0104] In some examples, to obtain discriminative features, the cosine distance can be used to calculate the cosine similarity of two input features. A smaller cosine distance between features of the same target object (i.e., the same target object) results in a higher cosine similarity score, while a larger cosine distance between features of different target objects results in a lower similarity score.

[0105] In some examples, reference Figure 6 The target similarity learning module 31 may further include a cosine similarity layer 313. The cosine similarity layer 313 may calculate the cosine similarity between the target object in the second image frame and the target object in the current image frame.

[0106] Exemplarily, the cosine similarity may satisfy Formula 2 shown below.

[0107]

[0108] In Formula 2, vector A and vector B represent two different input feature matrices. For example, A can be the feature matrix of the target object in any second image frame, vector B is the feature matrix of the target object in the current image frame, and n can represent the sum of at least one second image frame and the current image frame.

[0109] Therefore, the target similarity learning module 31 can obtain the similarity between the features of the target object in the second image frame and the features of the target object in the current image frame, thereby obtaining a similarity matrix.

[0110] Continue to refer to Figure 4 , the similarity matrix obtained by the target learning similarity module 31 is input into the association module 32, the similarity matrix is ​​processed, and the trajectory tracking result of the target object is obtained.

[0111] In some embodiments, the similarity matrix is ​​processed according to the association module in the processing module to obtain the trajectory tracking result of the target object, including:

[0112] According to the association module, the Hungarian matching algorithm is used to process the similarity matrix to determine the matching results of the target object in the second image frame and the target object in the current image frame; according to the identification information of the target object, the matching results are associated in time sequence to obtain the trajectory tracking result of the target object; wherein, the identification information of the target object is obtained by the detection module.

[0113] Specifically, the similarity matrix obtained by the target similarity learning module 31 is used as input to the association module 32, and the target object in the second image frame is matched with the target object in the current image frame using the Hungarian matching algorithm to obtain a matching result. For example, the matching result may include the matching degree between the target object in the second image frame and the target object in the current image frame, and based on the magnitude of the matching degree, the target object in the second image frame and the current image frame are further determined to be the same.

[0114] After obtaining the same target object in the second image frame as the current image frame, the target object is associated in time sequence based on the identification information of the target object in the second image frame and the identification information of the target object in the current image frame, thereby obtaining a trajectory result of the target object. For example, if the second image frame is before the current image frame, the target object's position information in the second image frame can be associated with the target object's position information in the current image frame to predict the target object's movement trajectory.

[0115] It should be noted that the matching process of the second image frame and other target objects in the current image frame is similar to the above process, and will not be described again here to avoid repetition.

[0116] After obtaining the matching results of each target object in any two image frames between all valid image frames (including at least one second image frame and the current image frame), the target objects in all valid image frames can be associated in chronological order to obtain the trajectory tracking results of each target object.

[0117] Therefore, the target similarity learning module 31 can improve the feature determination of the target object in at least two second image frames, and further improve the accuracy of trajectory tracking of the target object.

[0118] Figure 7 A flow chart of another trajectory tracking method provided in an embodiment of the present application is as follows: Figure 7 As shown, the method includes steps S710 to S750.

[0119] Step S710: Obtain a training sample set.

[0120] Exemplarily, the training sample set includes a plurality of sample image frames in a sample video sequence and types of target objects in the plurality of sample image frames, wherein the plurality of sample image frames include a current sample image frame and at least two first sample images.

[0121] Step S720 : performing feature extraction on the plurality of sample image frames according to the detection module of the initial trajectory tracking model to obtain feature data of the plurality of sample image frames.

[0122] The target trajectory tracking model in the above embodiment is the trained initial trajectory model. The structure of the initial trajectory model is similar to that of the target trajectory tracking model. For example, the structure of the initial trajectory tracking model can be as follows: Figure 2 or Figure 4 shown.

[0123] Step S730 : Processing the feature data of the target object in the plurality of sample image frames according to the screening module in the initial trajectory tracking model, and screening at least two first sample image frames to obtain at least one second sample image frame.

[0124] Exemplarily, the similarity between the second sample image frame and the current sample image frame is greater than or equal to a first preset similarity, and the second sample image frame includes the target object.

[0125] Step S740 : calculating the similarity of the target object in at least one sample image frame group according to the initial similarity learning module in the initial trajectory tracking model to obtain a predicted similarity.

[0126] Exemplarily, each sample image frame group in at least one sample image frame includes a second sample image frame and a current image frame. In some examples, the second sample image frames included in each sample image frame group are different.

[0127] The structure of the initial similarity learning model is similar to that of the target similarity learning model. For example, the structure of the initial similarity learning model can be as follows: Figure 6 According to the initial similarity learning module, the predicted similarity between the target objects in each sample image frame group can be calculated.

[0128] It should be noted that steps S710 to S740 are similar to steps S510 to S540 in the above embodiment, and will not be described again here to avoid repetition.

[0129] Step S750 , using the predicted similarity as the initial training output of the initial similarity learning module and the type of the target object in each sample image frame group as supervision information, iteratively training the initial similarity learning module to obtain a target similarity learning module.

[0130] In some embodiments, the predicted similarity is used as the initial training output of the initial similarity learning module, the type of the target object in each sample image frame group is used as supervision information, and the initial similarity learning module is iteratively trained to obtain the target similarity learning module, specifically including:

[0131] A loss value is determined based on the predicted similarity and the type of the target object in the sample image frame group; based on the loss value, the initial similarity learning module is iteratively updated until a target similarity learning module that meets preset conditions is obtained.

[0132] Exemplarily, the preset condition includes a loss value being less than or equal to a first preset threshold. That is, the target similarity learning model is obtained by training the initial similarity learning model based on the loss value. When the loss value is less than or equal to the first preset value, the trained initial similarity learning model becomes the target similarity learning model. The first preset threshold can be pre-set based on actual needs.

[0133] In some embodiments, determining a loss value based on the predicted similarity and the type of the target object in the sample image frame group specifically includes:

[0134] If the types of target objects in the sample image frame group are the same, the loss value is determined based on the predicted similarity; if the types of target objects in the sample image frame group are different, and the predicted similarity is less than or equal to the second preset similarity, the loss value is determined to be the second preset threshold, and the second preset threshold is less than or equal to the first preset threshold; if the types of target objects in the sample image frame group are different, and the predicted similarity is greater than the second preset similarity, the predicted similarity is updated to the second preset similarity, and the loss value is determined based on the second preset similarity.

[0135] In some embodiments, the loss value can be obtained by the following formula 3:

[0136]

[0137] In Formula 3, L is the loss value; D is the predicted Euclidean distance of the target object in the sample image frame group; m is the second preset similarity; Y indicates whether the types of the target objects in the sample image frame group are the same or different. When the types of the target objects in the sample image frame group are the same, Y is 0; when the types of the target objects in the sample image frame group are different, Y is 1.

[0138] In some examples, the predicted similarity between target objects in the sample image frame group can be determined based on the predicted Euclidean distance D between the target objects in the sample image frame group. For example, the larger the predicted Euclidean distance D between the target objects in the sample image frame group, the smaller the similarity between the target objects in the sample image frame group.

[0139] According to Formula 3, when the types of target objects in the sample image frame group are the same, Y=0. In this case, the loss value L can be obtained according to the following Formula 4, that is, the loss value L can be determined by the predicted Euclidean distance D of the target object in the sample image frame group.

[0140]

[0141] According to Formula 3, when the types of the objects in the sample image frame group are different, Y=1. In this case, the loss value L can be obtained according to the following Formula 5, where m is the second preset similarity, that is, the loss value L can be determined by the predicted Euclidean distance D of the target object in the sample image frame group and the second preset similarity m.

[0142]

[0143] As shown in Formula 5, when the predicted Euclidean distance D is greater than the second preset similarity m, the loss value L can be determined to be a second preset threshold, where the second preset threshold is less than or equal to the first preset threshold. For example, the second preset threshold can be 0. When the second threshold is 0, if the predicted Euclidean distance D is greater than m, the loss value L = 0. In this case, there is no need to optimize the target trajectory tracking model, thereby saving model training time and increasing model training speed.

[0144] As shown in Formula 5, when the predicted Euclidean distance D is less than or equal to the second preset similarity m, the loss value L can be determined based on the predicted Euclidean distance D and the second preset similarity m. However, when Y = 1, since the target objects belong to different classes, no matter how great the similarity of the target objects in the sample image frame group is, there is no need to continue model training. In this case, the predicted Euclidean distance D can be updated to the second preset similarity m. When the predicted Euclidean distance D is updated to the second preset similarity m, the loss value L can be determined based on the second preset threshold. For example, when the second preset threshold is 0, if the predicted Euclidean distance D is adjusted to m, the loss value L = 0. Therefore, there is no need to train the target trajectory tracking model, further improving the speed of model training.

[0145] In some examples, the screening module in the initial trajectory tracking model can also be referred to as the initial screening module. Embodiments of the present application can also iteratively update the initial screening module in the initial trajectory tracking model based on the loss value until a screening model that meets the second preset condition is obtained. When all modules in the initial trajectory tracking model have completed training and meet the relevant preset conditions, the trained initial trajectory tracking model becomes the target trajectory tracking model.

[0146] Therefore, the embodiment of the present application can better optimize the target trajectory tracking model, reduce system computing power consumption, and improve model training speed by optimizing the loss function and introducing the contrast loss value.

[0147] Figure 8 A schematic diagram of a trajectory tracking device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the trajectory tracking device 100 includes: an image acquisition module 101 , an image detection module 102 , an image screening module 103 and a data processing module 104 .

[0148] The image acquisition module 101 is configured to acquire a plurality of first image frames in a video sequence; the plurality of image frames includes a current image frame and at least two first image frames;

[0149] An image detection module 102 is configured to extract features from the plurality of image frames according to a detection module in a target trajectory tracking model to obtain feature data in the plurality of image frames;

[0150] An image screening module 103 is configured to process feature data of the plurality of image frames according to the screening module in the target trajectory tracking model, and screen at least one or two second image frames from at least two first image frames; wherein a similarity between at least one second image frame and the current image frame is greater than or equal to a first preset similarity; and the second image frame includes the target object;

[0151] The data processing module 104 is configured to process the at least one second image frame according to the processing module in the target trajectory tracking model to obtain a trajectory tracking result of the target object.

[0152] In some embodiments, the screening module includes a graph convolutional neural network; the image screening module 103 is specifically used to: process the feature data of multiple image frames according to the graph convolutional neural network to obtain the cosine similarity between each first image frame and the current image frame; based on the size relationship between the cosine similarity between each first image frame and the current image frame and the first preset similarity, screen at least one second image frame from at least two first image frames.

[0153] In some embodiments, the image screening module 103 is specifically configured to: if the cosine similarity between the first image frame and the current image frame is greater than or equal to a first preset similarity, determine the first image frame as the second image frame.

[0154] In some embodiments, the processing module in the target trajectory tracking model includes a target similarity learning module and an association module; the image processing module 104 is specifically used to: calculate the similarity between the target object in each second image frame and the target object in the current image frame according to the target similarity learning module in the processing module to obtain a similarity matrix; and process the similarity matrix according to the association module in the processing module to obtain a trajectory tracking result of the target object.

[0155] In some embodiments, the image processing module 104 is specifically used to: according to the target similarity learning module, adopt the attention mechanism to obtain the feature data of the target object in each second image frame, and the feature data of the target object in the current image frame; according to the feature data of the target object in each second image frame, and the feature data of the target object in the current image frame, determine the similarity between the target object in each second image frame and the target object in the current image frame to obtain a similarity matrix.

[0156] In some embodiments, the image processing module 104 is specifically used to: process the similarity matrix using the Hungarian matching algorithm according to the association module to determine the matching results of the target object in the second image frame and the target object in the current image frame; according to the identification information of the target object, associate the matching results in time sequence to obtain the trajectory tracking result of the target object; wherein the identification information of the target object is obtained by the detection module.

[0157] In some embodiments, the trajectory tracking device 100 further includes a model training module, wherein the image acquisition module 101 is further configured to: acquire a training sample set, the training sample set including a plurality of sample image frames in a sample video sequence, and types of target objects in the plurality of sample image frames; wherein the plurality of sample image frames include a current sample image frame and at least two first sample images;

[0158] The image detection module 102 is further configured to: perform feature extraction on a plurality of sample image frames according to the detection module of the initial trajectory tracking model to obtain feature data of the plurality of sample image frames;

[0159] The image screening module 103 is further configured to: process feature data of the plurality of sample image frames according to the screening module in the initial trajectory tracking model, and screen at least two first sample image frames to obtain at least one second sample image frame; wherein the similarity between the second sample image frame and the current sample image frame is greater than or equal to a first preset similarity, and the second sample image includes the target object;

[0160] The image processing module 104 is further configured to calculate the similarity of the target object in the sample image frame group according to the initial similarity learning module in the initial trajectory tracking model to obtain a predicted similarity; wherein the sample image frame group includes a second sample image frame and the current sample image frame;

[0161] The model training module is used to: use the predicted similarity as the initial training output of the initial similarity learning module, use the type of target object in each sample image frame group as supervision information, and iteratively train the initial similarity learning module to obtain the target similarity learning module.

[0162] In some embodiments, the model training module is specifically used to: determine the loss value based on the similarity and the type of target object in the sample image frame group; iteratively update the initial similarity learning module based on the loss value until the target similarity learning module that meets the preset conditions is obtained; the preset conditions include that the loss value is less than or equal to a first preset threshold.

[0163] In some embodiments, the model training module is specifically further used to: if the types of target objects in the sample image frame group are the same, determine the loss value based on the predicted similarity; if the types of target objects in the sample image frame group are different, and the predicted similarity is less than or equal to the second preset similarity, determine the loss value as the second preset threshold, and the second preset threshold is less than or equal to the first preset threshold; if the types of target objects in the sample image frame group are different, and the predicted similarity is greater than the second preset similarity, update the predicted similarity to the second preset similarity, and determine the loss value based on the second preset similarity.

[0164] In some embodiments, the loss value is obtained by the following formula:

[0165]

[0166] Among them, L is the loss value, D is the predicted Euclidean distance of the target object in the sample image frame group, m is the second preset similarity, and Y indicates whether the types of the target objects in the sample image frame group are the same or different. When the types of the target objects in the sample image frame group are the same, Y is 0; when the types of the target objects in the sample image frame group are different, Y is 1.

[0167] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0168] like Figure 9 As shown, the electronic device 1000 may include: a processor (processor) 1001 and a memory 1002. Exemplarily, the electronic device 1000 may further include: a communication interface (Communications Interface) 1003 and a communication bus 1004.

[0169] The processor 1001, the memory 1002 and the communication interface 1003 communicate with each other via a communication bus 1004. The communication interface 1003 is used to communicate with other devices such as a client or other server network elements.

[0170] In some embodiments, the processor 1001 is configured to execute a program 1005, specifically, to execute the relevant steps in the above trajectory tracking method embodiment. Specifically, the program 1005 may include program code, which includes computer-executable instructions.

[0171] For example, the processor 1001 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The electronic device 1000 may include one or more processors of the same type, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

[0172] In some embodiments, the memory 1002 is used to store the program 1005. The memory 1002 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0173] Program 1005 can be specifically called by processor 1001 to enable electronic device 1000 to perform the following operations:

[0174] Acquire multiple image frames in a video sequence, where the multiple image frames include a current image frame and at least two first image frames;

[0175] According to the detection module in the target trajectory tracking model, feature extraction is performed on multiple image frames to obtain feature data of the multiple image frames;

[0176] Processing feature data in the plurality of image frames according to a screening module in the target trajectory tracking model to screen at least two first image frames to obtain at least one second image frame; wherein a similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity, and the second image frame includes the target object;

[0177] According to the processing module in the target trajectory tracking model, at least one second image frame and the current image frame are processed to obtain a trajectory tracking result of the target object.

[0178] An embodiment of the present application provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on the electronic device 1000, the electronic device 1000 executes the trajectory tracking method in the above embodiment.

[0179] The executable instructions may be specifically used to enable the electronic device 1000 to perform the following operations:

[0180] Acquire multiple image frames in a video sequence, where the multiple image frames include a current image frame and at least two first image frames;

[0181] According to the detection module in the target trajectory tracking model, feature extraction is performed on multiple image frames to obtain feature data of the multiple image frames;

[0182] Processing feature data in the plurality of image frames according to a screening module in the target trajectory tracking model to screen at least two first image frames to obtain at least one second image frame; wherein a similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity, and the second image frame includes the target object;

[0183] According to the processing module in the target trajectory tracking model, at least one second image frame and the current image frame are processed to obtain a trajectory tracking result of the target object.

[0184] For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0185] The trajectory tracking device, electronic device, and computer-readable storage medium provided in the embodiments of the present application are all used to execute the corresponding trajectory tracking method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device (such as the trajectory tracking device) is divided into different functional modules to complete all or part of the functions described above. The specific working process of the above-described system, device (such as the trajectory tracking device) and unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices (e.g., trajectory tracking devices) and methods can be implemented in other ways. For example, the device (e.g., trajectory tracking device) embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0191] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A trajectory tracking method, characterized in that: The method comprises: Acquire a plurality of image frames in a video sequence; the plurality of image frames include a current image frame and at least two first image frames; performing feature extraction on the plurality of image frames according to a detection module in a target trajectory tracking model to obtain feature data of the plurality of image frames; processing, according to a screening module in the target trajectory tracking model, feature data of the plurality of image frames, and screening at least one second image frame from the at least two first image frames; wherein a similarity between the second image frame and the current image frame is greater than or equal to a first preset similarity, and the second image frame includes a target object; The at least one second image frame and the current image frame are processed according to the processing module in the target trajectory tracking model to obtain a trajectory tracking result of the target object.

2. The method according to claim 1, characterized in that The screening module includes a graph convolutional neural network; the screening module in the target trajectory tracking model processes the feature data of the multiple image frames and screens at least one second image frame from the at least two first image frames, including: Processing feature data of the plurality of image frames according to the graph convolutional neural network to obtain cosine similarities between each of the first image frames and the current image frame; The at least one second image frame is obtained by screening from the at least two first image frames according to a magnitude relationship between the cosine similarity between each of the first image frames and the current image frame and the first preset similarity.

3. The method according to claim 2, characterized in that The step of selecting the at least one second image frame from the at least two first image frames based on a magnitude relationship between a cosine similarity between each of the first image frames and the current image frame and the first preset similarity includes: If the cosine similarity between the first image frame and the current image frame is greater than or equal to the first preset similarity, the first image frame is determined as the second image frame.

4. The method according to claim 1, wherein The processing module in the target trajectory tracking model includes a target similarity learning module and an association module; the processing module in the target trajectory tracking model processes the at least one second image frame and the current image frame to obtain a trajectory tracking result of the target object, including: Calculating, according to the target similarity learning module in the processing module, the similarity between the target object in each of the second image frames and the target object in the current image frame to obtain a similarity matrix; The similarity matrix is ​​processed according to the association module in the processing module to obtain a trajectory tracking result of the target object.

5. The method according to claim 4, characterized in that The target similarity learning module in the processing module calculates the similarity between the target object in each second image frame and the target object in the current image frame to obtain a similarity matrix, including: Acquiring, according to the target similarity learning module, feature data of the target object in each of the second image frames and feature data of the target object in the current image frame using an attention mechanism; Based on the feature data of the target object in each of the second image frames and the feature data of the target object in the current image frame, the similarity between the target object in each of the second image frames and the target object in the current image frame is determined to obtain the similarity matrix.

6. The method according to claim 4, characterized in that The step of processing the similarity matrix according to the association module in the processing module to obtain the trajectory tracking result of the target object includes: According to the association module, the similarity matrix is ​​processed using a Hungarian matching algorithm to determine a matching result between the target object in the second image frame and the target object in the current image frame; According to the identification information of the target object, the matching results are associated in time sequence to obtain a trajectory tracking result of the target object; wherein the identification information of the target object is obtained by the detection module.

7. The method according to claim 4, characterized in that The method further comprises: Acquire a training sample set, the training sample set comprising a plurality of sample image frames in a sample video sequence, and a type of a target object in each of the sample image frames; wherein the plurality of sample image frames comprises a current sample image frame and at least two first sample images; performing feature extraction on the plurality of sample image frames according to the detection module in the initial trajectory tracking model to obtain feature data of the plurality of sample image frames; processing, according to the screening module in the initial trajectory tracking model, feature data of the plurality of sample image frames, and screening at least one second sample image frame from the at least two first sample image frames; wherein a similarity between the second sample image frame and the current sample image frame is greater than or equal to a first preset similarity, and the second sample image frame includes the target object; Calculating the similarity of the target object in at least one sample image frame group according to the initial similarity learning module in the initial trajectory tracking model to obtain a predicted similarity; wherein the sample image frame group includes the second sample image frame and the current sample image frame; The predicted similarity is used as the initial training output of the initial similarity learning module, the type of the target object in each of the sample image frame groups is used as supervision information, and the initial similarity learning module is iteratively trained to obtain the target similarity learning module.

8. The method according to claim 7, characterized in that The method uses the predicted similarity as the initial training output of the initial similarity learning module, uses the type of the target object in each of the sample image frame groups as supervision information, and iteratively trains the initial similarity learning module to obtain the target similarity learning module, including: determining a loss value according to the predicted similarity and the type of the target object in the sample image frame group; According to the loss value, the initial similarity learning module is iteratively updated until the target similarity learning module that meets a preset condition is obtained; wherein the preset condition includes that the loss value is less than or equal to a first preset threshold.

9. The method according to claim 8, characterized in that The determining of the loss value according to the predicted similarity and the type of the target object in the sample image frame group includes: If the types of the target objects in the sample image frame group are the same, determining a loss value according to the predicted similarity; If the types of the target objects in the sample image frame groups are different and the predicted similarity is less than or equal to a second preset similarity, determining the loss value to be a second preset threshold, where the second preset threshold is less than or equal to the first preset threshold; If the types of the target objects in the sample image frame group are different and the predicted similarity is greater than the second preset similarity, the predicted similarity is updated to the second preset similarity, and the loss value is determined according to the second preset similarity.

10. The method according to claim 9, characterized in that The loss value is obtained by the following formula: Wherein, L is the loss value, D is the predicted Euclidean distance of the target object in the sample image frame group, m is the second preset similarity, and Y indicates whether the types of the target objects in the sample image frame group are the same or different. When the types of the target objects in the sample image frame group are the same, Y is 0; when the types of the target objects in the sample image frame group are different, Y is 1.

Images