Multi-target tracking method and related equipment

Abstract

The invention relates to the field of multi-target tracking, and discloses a multi-target tracking method and related equipment. The method comprises the following steps: acquiring target detection information of each frame of image of a video sequence, determining target tracking information of a previous preset continuous frame of image, and taking the target tracking information as reference data; determining interaction information and scene constraint information of the last frame of image in the reference data; determining prediction tracking information of a next frame of image of the reference data based on target tracking information, interaction information and scene constraint information of the last frame of image in the reference data; determining the target tracking information of the frame image based on the target detection information and the prediction tracking information of the next frame image of the reference data; and taking the target tracking information of the next frame of image of the reference data and the reference data as new reference data, and repeatedly executing the previous steps until the target tracking information of each frame of image in the video sequence is obtained so as to determine the tracking trajectory of each target in the video sequence. The method can improve the accuracy of multi-target tracking.

Description

Technical Field

[0001] The present application relates to the technical field of multi-target tracking, and in particular to a multi-target tracking method and related equipment. Background Art

[0002] As a mid-level task in the field of computer vision, multi-target tracking has broad application prospects, such as security monitoring, behavior analysis, unmanned driving, and posture estimation. The task of multi-target tracking is to input a video and output the trajectory of the target appearing in the video. Since pedestrian tracking has a wide range of research value, multi-pedestrian tracking has become the mainstream research in the field of multi-target tracking.

[0003] In recent years, with the continuous improvement of the performance of detection algorithms, multi-pedestrian tracking based on the detection and tracking framework has become the mainstream method for multi-pedestrian tracking. The principle of the detection and tracking framework is to first detect pedestrians in each frame of the video, then extract the appearance or motion features of the pedestrians for data association and form the final motion trajectory. Existing methods usually learn reliable motion patterns to match the same target between adjacent frames, and learn discriminative appearance features to re-identify targets that have been lost for a long time. However, during the tracking process, dense crowds and extreme occlusions can easily affect the reliability of motion prediction and the discriminability of appearance, making it difficult to achieve accurate association between targets, resulting in low accuracy in multi-target tracking.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a multi-target tracking method and related devices to improve the accuracy of multi-target tracking.

[0007] In some embodiments, a multi-target tracking method includes: S1, obtaining target detection information of each frame image in a video sequence to be tracked, and determining the target tracking information of previously preset continuous frame images based on each target detection information, and using the target tracking information of the previously preset continuous frame images as reference data; S2, determining the interaction information between each target in the last frame image in the reference data and determining the scene constraint information based on the reference data; S3, determining the predicted tracking information of the next frame image of the reference data based on the target tracking information of the last frame image in the reference data, the corresponding interaction information and the scene constraint information; S4, determining the target tracking information of the next frame image of the reference data based on the target detection information and the predicted tracking information of the next frame image of the reference data; S5, using the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeating S2-S5 until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

[0008] In some embodiments, a multi-target tracking system includes: an acquisition module, which is used to acquire target detection information of each frame image in a video sequence to be tracked, and determine the target tracking information of previously preset continuous frame images based on each of the target detection information, and use the target tracking information of the previously preset continuous frame images as reference data; a determination module, which is used to determine the interaction information between each target in the last frame image in the reference data and determine the scene constraint information based on the reference data; a prediction module, which is used to determine the predicted tracking information of the next frame image of the reference data based on the target tracking information of the last frame image in the reference data, the corresponding interaction information and the scene constraint information; a tracking module, which is used to determine the target tracking information of the next frame image of the reference data based on the target detection information and the predicted tracking information of the next frame image of the reference data; a trajectory module, which is used to use the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeatedly execute the steps in the acquisition module, the determination module, the prediction module, the tracking module and the trajectory module until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

[0009] In some embodiments, a computing device includes a processor and a memory storing program instructions, wherein the processor is configured to execute the multi-target tracking method as described above when running the program instructions.

[0010] In some embodiments, a computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the multi-target tracking method as described above.

[0011] The multi-target tracking method and related devices provided by the embodiments of the present disclosure can achieve the following technical effects:

[0012] By obtaining the target detection information of each frame image in the video sequence to be tracked and using it to determine the target tracking information of the previously preset continuous frame images, the interaction information and scene constraint information between the various targets in the last frame image of the reference data can be determined according to the target tracking information of each frame image in the reference data. In this way, the information association and decision-making capabilities in the tracking process can be enhanced through the interaction information and scene constraints between targets, and the accurate association between targets can be achieved, so that the target tracking information of the next frame image of the reference data can be predicted more accurately. When the target tracking information of the next frame of the reference data is determined based on the predicted tracking information and target detection information of the next frame of the reference data, the accuracy of the target tracking information can be improved, the accuracy and robustness of multi-target tracking can be significantly improved, and the accurate association between targets can be achieved to improve the accuracy of multi-target tracking.

[0013] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0015] Figure 1 is a flowchart of a multi-target tracking method provided by an embodiment of the present disclosure;

[0016] Figure 2 is a structural diagram of a multi-target tracking system provided by an embodiment of the present disclosure;

[0017] Figure 3 It is a structural diagram of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0019] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0020] Unless otherwise stated, the term "plurality" means two or more.

[0021] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0022] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0023] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0024] The embodiment of the present disclosure provides a multi-target tracking method and related devices, including a multi-target tracking method, a system, a computing device and a computer storage medium. In this embodiment, a multi-target tracking method is applied to an electronic device. The electronic device is, for example, a computer.

[0025] Combination Figure 1 As shown, the embodiment of the present disclosure provides a multi-target tracking method, including:

[0026] Step S1, obtaining target detection information of each frame image in the video sequence to be tracked, and determining target tracking information of previously preset continuous frame images according to each target detection information, and using the target tracking information of previously preset continuous frame images as reference data.

[0027] It can be understood that the video sequence can be a surveillance video, and there are multiple targets in the video sequence, and the targets are pedestrians. Video sequence S = {I1, I2, ..., I n}, I n is the nth frame image. The number of the previous preset continuous frame images is less than the number of image frames in the video sequence. For example, if the video sequence has 20 frames of images, the number of the previous preset frame images is less than 20.

[0028] In some embodiments, each frame of the video sequence is identified by a target detection algorithm to identify the target (i.e., pedestrian) in each frame, and there is at least one target in each frame. The target detection algorithm is a YOLOX detector, and the target detection information represents the detection information of each target in the frame image. The target detection information includes the coordinates of the detection frame corresponding to each target (i.e., the position of the target), the confidence score corresponding to each detection frame, and the category label corresponding to each detection frame.

[0029] In some embodiments, target tracking information of previously preset continuous frame images is determined based on each target detection information, including tracking processing based on the target detection information of previously preset continuous frame images using Kalman filtering or SORT (Simple Online and Realtime Tracking) algorithm to determine the target tracking information of the previous continuous frame images. That is, a motion trajectory is constructed based on the position of the target in different frame images using Kalman filtering or SORT algorithm to determine the target tracking information of the previous continuous frame images. The target tracking information represents the tracking information of each target in the frame image. The target tracking information includes the coordinates of the tracking frame corresponding to each target (i.e., the position of the target) and the ID corresponding to each tracking frame (i.e., the ID of the target).

[0030] In some embodiments, the number of the preset continuous frames is greater than or equal to 2. Assuming that the number of the preset continuous frames is 2, the first preset continuous frames are the first two frames of the video sequence, that is, the first frame and the second frame of the video sequence.

[0031] Step S2: Determine, based on the reference data, the interaction information between the various targets in the last frame image in the reference data and the scene constraint information.

[0032] It can be understood that the last frame image in the reference data represents the last frame image in the previous preset continuous frame image corresponding to the reference data. The scene constraint information represents the set of scene constraint conditions between various targets in the same frame image, and the scene constraint conditions represent the constraints that affect the mutual movement between targets, such as obstacles.

[0033] S3. Determine predicted tracking information of a next frame of the reference data based on the target tracking information of the last frame of the reference data, the corresponding interaction information and the scene constraint information.

[0034] It can be understood that the next frame image of the reference data represents the next frame image of the last frame image in the reference data, that is, represents the next frame image of the last frame image in the previous preset continuous frame images corresponding to the reference data.

[0035] S4. Determine target tracking information of a next frame image of the reference data based on the target detection information and predicted tracking information of the next frame image of the reference data.

[0036] S5. Use the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeat S2-S5 until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

[0037] It can be understood that this embodiment starts from the preset continuous frame image (i.e., the first reference data), repeatedly uses the obtained target tracking information to predict the target tracking information of the next frame image (i.e., the predicted tracking information), and determines the actual target tracking information of the next frame image based on the target detection information of the next frame image and the predicted target tracking information. Until the target tracking information of the last frame image of the video sequence is predicted, and the actual target tracking information of the last frame image of the video sequence is determined based on the target detection information of the last frame image of the video sequence and the predicted target tracking information. Finally, based on the actual target tracking information of each frame image of the video sequence, the trajectory of each target in the video sequence is updated to obtain the tracking trajectory of each target in the video sequence. For example, the motion trajectory of target j is defined as: T j ={(x j,t ,y j,t )}|t∈[t0,t f ]. Among them, T j is the motion trajectory of target j, including the position of the target at different times. (x j,t ,y j,t ) The position coordinates of target j at time t, x j,t Indicates the horizontal position, y j,t Indicates the vertical position. t0 and t f The start and end time of tracking represents the movement trajectory of the target during this period.

[0038] The multi-target tracking method provided by the embodiment of the present disclosure is adopted, by obtaining the target detection information of each frame image in the video sequence to be tracked and determining the target tracking information of the previously preset continuous frame images, so as to determine the interaction information and scene constraint information between each target in the last frame image in the reference data according to the target tracking information of each frame image in the reference data. In this way, the information association and decision-making ability in the tracking process can be enhanced through the interaction information and scene constraints between the targets, and the accurate association between the targets can be achieved, so that the target tracking information of the next frame image of the reference data can be predicted more accurately. When the target tracking information of the next frame of the reference data is determined based on the predicted tracking information and target detection information of the next frame of the reference data, the accuracy of the target tracking information can be improved, the accuracy and robustness of multi-target tracking can be significantly improved, and the accurate association between the targets can be achieved to improve the accuracy of multi-target tracking.

[0039] Optionally, based on the reference data, determining the interaction information between each target in the last frame image of the reference data, including: extracting distance feature information between each target in each frame image of the reference data; based on the target tracking information of each frame image in the reference data and the distance feature information between each target, determining the interaction information between each target in the last frame image of the reference data.

[0040] In this way, based on the target tracking information in each frame image and the distance feature information between each target in the reference data, the motion changes of each target with respect to other targets in different frame images can be determined, thereby determining the interaction information between each target.

[0041] In some embodiments, the distance feature information represents relative distance and relative speed, etc. The relative distance is the Euclidean distance. The relative speed is obtained by the speed vector of each target in the current frame. For example, there are two targets in the second frame image, target i and target j, and the speed vector of target i is V i , the velocity vector of target j is V j , then in the second frame image, the relative speed between target i and target j is V ij =V i -V j In some other embodiments, the distance characteristic information also includes relative acceleration and the like.

[0042] Furthermore, when there are many frame images in the reference data, a preset number of continuous frame images can be selected from the reference data to determine the interaction information between the various targets in the last frame image in the reference data. Moreover, the selected preset number of continuous frame images need to include the last frame image in the reference data. For example, if there are 20 frames of images in the reference data, and the last frame image in the reference data is the 20th frame, the 19th and 20th frames of images can be selected to determine the interaction information between the various targets in the last frame image (i.e., the 20th frame image) in the reference data.

[0043] Optionally, based on the target tracking information of each frame image in the reference data and the distance feature information between each target, determining the interaction information between each target in the last frame image in the reference data includes: determining the nodes of the graph model based on the target tracking information of each frame image in the reference data; constructing the edges of the graph model based on the distance feature information between each target in each frame image in the reference data; determining the weights of the edges between each node based on the nodes and edges of the graph model, and determining the interaction information between each target in the last frame image in the reference data based on the weights of each edge. Wherein, the graph model represents a graph neural network (GNN).

[0044] In this way, the message passing mechanism is implemented using the graph model so that each node can transmit information to each other and update its own feature representation, which helps to capture the interactive influence between targets in order to obtain the interactive information between each target in the last frame image in the reference.

[0045] Specifically, determining the nodes of the graph model based on the target tracking information of each frame image in the reference data includes: determining the feature vector of each target in each frame image based on the target tracking information of each frame image in the reference data, and using each feature vector as a node of the corresponding frame image in the graph model. The feature vector includes position (x, y), speed (v x , v y ) and acceleration information.

[0046] Specifically, the edges of the graph model are constructed based on the distance feature information between each target in each frame of the reference data, including: if the distance (i.e., relative distance) between two targets (i.e., two nodes) is within a preset threshold range, an edge is established between the two targets, indicating that there is a mutual relationship between the two targets. The Euclidean distance can be used as a distance metric, and the speed difference can be used as a metric of mutual movement to form weighted edges.

[0047] In some embodiments, the setting of edge weights includes: distance weight and speed difference (i.e., relative speed difference). The distance weight can be the reciprocal of the Euclidean distance between two targets, i.e., the smaller the distance, the greater the weight, indicating a stronger interaction. The reciprocal of the speed difference can also be used for edge weights to indicate the consistency of the target movement trends. Targets with a small speed difference are assigned a larger weight, indicating that there may be a stronger interaction between the two targets.

[0048] In some embodiments, the message passing mechanism formula in the graph model is:

[0049]

[0050] in, is the feature representation of target i in the kth layer of the graph neural network (i.e., graph model), and the σ activation function is used to increase the nonlinear expression ability of the model, usually ReLU or Sigmoid. (k) The weight matrix of the kth layer of the graph neural network represents the weight parameter of the feature transformation. N(i) is the set of neighbor nodes of target i, which represents other targets that have a mutual relationship with target i. M is the message function used to transmit information from neighbor node j to target i, which is usually the weighted sum of the features of the neighbor nodes. The weight ω of the edge between target i and target j output by GNN (i.e., graph model) ij , reflecting the degree of interaction between targets.

[0051] Further, when there are many frame images in the reference data, continuous frame images of a preset number of frames can be selected to determine the interaction information between each target in the last frame image in the reference data. Moreover, the selected continuous frame images of the preset number of frames need to include the last frame image in the reference data. For example, if there are 20 frames of images in the reference data, and the last frame image in the reference data is the 20th frame, the 19th and 20th frames of images can be selected to determine the interaction information between each target in the last frame image (i.e., the 20th frame image) in the reference data. That is, the nodes of the graph model are determined based on the target tracking information of the 19th and 20th frames of images in the reference data; the edges of the graph model are constructed based on the distance feature information between each target in the 19th and 20th frames of images in the reference data; the weights of the edges between each node are determined based on the nodes and edges of the graph model, and the interaction information between each target in the 20th frame image is determined based on the weights of each edge. In this way, the interaction information between each target in the last frame image in the reference data can be obtained, and the amount of calculation can be reduced, thereby improving the efficiency of multi-target tracking.

[0052] Specifically, a multi-target tracking method further includes: determining the interaction behavior between the targets based on the interaction information between the targets in the last frame image in the reference data, such as whether there is collision, mutual attraction or repulsion, etc. This is conducive to predicting the target tracking information of the next frame image of the reference data.

[0053] Optionally, determining scene constraint information includes: determining obstacle information of the last frame image in the reference data; determining an obstacle area and a walking area based on the obstacle information; and determining scene constraint information based on the obstacle area and the walking area.

[0054] In this way, by determining the obstacle information of the last frame of the reference data, it is possible to know the area where the target can walk freely, so as to determine the scene constraints when the targets move between each other, which is conducive to more accurate prediction of target tracking information.

[0055] Specifically, a background subtraction algorithm (such as a Gaussian mixture model) is applied to detect static objects or obstacles in the scene. For example, by comparing the background and foreground of consecutive frames, the obstacle area is extracted to obtain obstacle information. An image segmentation algorithm (such as K-means clustering or watershed algorithm) is used to segment the scene and extract the walking area and obstacle area.

[0056] Specifically, determining the scene constraint information based on the obstacle area and the walking area includes: determining the distance between each target based on the target tracking information of the obstacle area, the walking area and the last frame image in the reference data. C obstacleAs the scene constraint conditions between the various objects in the last frame image in the reference data, the set of scene constraint conditions between the various objects in the last frame image in the reference data is used as the scene constraint information. obstacle It is the constraint condition of obstacles in the scene, indicating the relationship between the target and the obstacle. When it is 1, the target is in the free space, that is, the target is in the walking area; when it is 0, the target should avoid moving to this area, which is the obstacle area.

[0057] In some embodiments, if the scene in the video sequence to be tracked is a static scene, that is, the scene of the video sequence has not changed, and the video sequence is a video shot in a fixed scene, then the corresponding obstacle information can be obtained by identifying any frame in the reference data, and the corresponding obstacle information is used as the obstacle information of the last frame image in the reference data.

[0058] In other embodiments, if the scene in the video sequence to be tracked is a dynamic scene, that is, the scene of the video sequence has changed, and the video sequence is a video shot in a dynamic scene. In order to adapt to the changes in the dynamic scene, the parameters of the Gaussian mixture model can be adjusted using incremental learning and adaptive update mechanisms. For example, historical data is used to update the model parameters, and the learning rate is dynamically adjusted according to the changes in the loss function. Using historical data to update the model parameters:

[0059]

[0060] Among them, θ t represents the parameter vector of the model at time t, which is continuously updated over time. η represents the learning rate, which controls the step size of parameter update. Represents the gradient of the loss function with respect to the parameters, which is used to guide the update direction of the model parameters.

[0061] Optionally, based on the target tracking information of the last frame image in the reference data, the corresponding interaction information and the scene constraint information, the predicted tracking information of the next frame image of the reference data is determined, including: constructing a total loss function of the joint optimization algorithm based on the interaction information and the scene constraint information corresponding to the last frame image in the reference data to obtain a target joint optimization algorithm; based on the target tracking information of the last frame image in the reference data and the target joint optimization algorithm, determining the predicted tracking information of the next frame image of the reference data.

[0062] In this way, the two losses of "motion consistency" and "scene constraint" are balanced through a joint optimization algorithm, thereby optimizing the trajectory tracking result of the target, so that the last frame image in the reference data can be accurately predicted and the predicted tracking information of the next frame image of the reference data can be obtained.

[0063] Specifically, a total loss function of the joint optimization algorithm is constructed based on the interaction information and scene constraint information corresponding to the last frame of the reference data to obtain a target joint optimization algorithm, including: determining motion consistency loss based on the interaction information corresponding to the last frame of the reference data; determining scene constraint loss based on the scene constraint information corresponding to the last frame of the reference data. A total loss function of the joint optimization algorithm is constructed based on the motion consistency loss and the scene constraint loss.

[0064] Specifically, determining the motion consistency loss based on the interaction information corresponding to the last frame image in the reference data includes: determining the motion consistency loss by the following formula: Among them, Γ motion is the motion consistency loss, which indicates the degree of motion matching between targets, ω ij D is the weight of the edge between each target (i.e., node) in the last frame of the reference data output by the graph model. i 2 j The sum of the squared Euclidean distances between target i and target j is used to maintain the consistency of the target's trajectory; the square of the Euclidean distance amplifies the large distance difference, making the optimization pay more attention to those targets that are farther away to ensure that their trajectories tend to be consistent.

[0065] In addition, since the weight of the edge corresponding to each frame image and the Euclidean distance between each target are different, the expression The results of interactive behavior analysis can be effectively incorporated into the loss, making motion consistency optimization more in line with the interactive needs in actual scenarios.

[0066] Specifically, determining the scene constraint loss based on the scene constraint information corresponding to the last frame image in the reference data includes: determining the scene constraint loss by the following formula:

[0067] Γ scene =∑ i,j C ij (D ij -D max ) 2 Among them, Γ scene is the scene constraint loss, which indicates the degree of adaptation between the target and the scene constraint (such as obstacles), and is used to calculate the deviation between the distance between the target and the scene constraint; C ij is the scene constraint between target i and target j, that is, whether there will be overlap with obstacles when target i and target j move.

[0068] C ij for When C ijWhen C is 1, it indicates that when the target i and target j move, the target i and target j will not overlap with obstacles. ij When it is 0, it indicates that when the target i and target j move, the target i and / or target j overlaps with the obstacle. max The maximum distance threshold allowed is used to control the maximum distance between targets.

[0069] Specifically, the total loss function of the joint optimization algorithm is constructed based on the motion consistency loss and the scene constraint loss, including: constructing the total loss function of the joint optimization algorithm through the following formula: Γ = α·Γ motion +β·Γ scene Among them, Γ is the total loss function of the joint optimization algorithm, which is used to optimize the tracking process, and α and β are weight parameters of the loss function, which are used to control the balance between motion consistency and scene constraints.

[0070] Optionally, based on the target detection information and predicted tracking information of the next frame image of the reference data, the target tracking information of the next frame image of the reference data is determined, including: constructing an IOU global matrix based on the target detection information of the next frame image of the reference data and the predicted tracking information of the next frame image of the reference data, and solving the global optimal solution; determining the target tracking information of the next frame image of the reference data based on the global optimal solution.

[0071] In this way, by constructing the IOU global cost matrix and solving the global optimal solution, in order to determine the target in the predicted tracking information corresponding to the target in the target detection information, the target tracking information of the next frame image of the reference data can be determined more accurately.

[0072] Specifically, an IOU global cost matrix is ​​constructed based on each detection frame in the target detection information of the next frame image of the reference data and each tracking frame in the target tracking information, and the Hungarian algorithm is used to solve it to obtain the global optimal solution, that is, a tracking frame matching each detection frame is obtained, and then the id of the tracking frame is given to the detection frame matching it to determine the id of each detection frame in the target detection information of the next frame image of the reference data, and the target detection information assigned with the tracking frame id is determined as the target tracking information.

[0073] Optionally, a multi-target tracking method also includes: if a new target is identified in the target tracking information of any frame image except the first frame image, determining the tracking trajectory of the new target in the video sequence; and managing the tracking trajectory of each target in the video sequence based on the tracking trajectory of the new target in the video sequence.

[0074] Since many targets are moving in the video sequence, it is easy for targets to be blocked or new targets to appear. Therefore, by determining the tracking trajectory of the new target in the video sequence and managing the tracking trajectory of each target in the video sequence, the consistency of each target ID can be ensured, so as to continuously track and accurately identify the target.

[0075] Specifically, the tracking trajectories of each target in the video sequence are managed based on the tracking trajectory of the new target in the video sequence, including: comparing the trajectory of the new target with each historical trajectory for trajectory similarity to obtain a comparison result. Based on the comparison result, it is determined whether there is a similar historical trajectory. If so, the trajectory of the new target is merged with the similar historical trajectory. That is, the id of the target corresponding to the similar historical trajectory is assigned to the new target. The historical trajectory represents the trajectory of the target that is earlier than the new target.

[0076] It is understandable that if similar historical trajectories are determined based on the comparison results, it means that the new target is a target that reappears after the existing target is blocked. At this time, the trajectory of the new target is merged with the corresponding historical trajectory, so as to ensure the consistency of the target ID and facilitate the management of the target's trajectory information.

[0077] Specifically, the trajectory of the new target is compared with each historical trajectory for trajectory similarity, and the comparison results are obtained, including: ReID (T current , T history )=Sim(T current , T history ). Among them, ReID(T current , T history ) is the re-identification metric of the trajectory of the new target and the historical trajectory, and Sim is the comparison result.

[0078] Furthermore, a multi-target tracking method further includes: if there is a target (i.e., a detection frame) in the target detection information of the next frame image of the reference data that cannot be matched with the target tracking information, and the target that cannot be matched cannot find a matching tracking frame in the target tracking information of the frame image corresponding to the preset time length, then the trajectory of the target is terminated or deleted. If there is a target (i.e., a detection frame) in the target detection information of the next frame image of the reference data that cannot be matched with the target tracking information, and the target that cannot be matched can find a matching tracking frame in the target tracking information of the frame image corresponding to the preset time length, then the trajectory of the target is initialized, that is, a new id is assigned to the target, and a new trajectory is created.

[0079] In this way, by merging, terminating or initializing the targets that failed to match, the target accumulation management is achieved, and the continuous tracking and accurate identification of the targets can be maintained.

[0080] The disclosed embodiment enhances the ability to capture the target trajectory by combining short-term and long-term motion information, predicts the future position of the target using the long-term motion mode (i.e., the mutual information between each target in each frame image), and incorporates the scene geometric features (i.e., scene constraint information) into the constraint conditions, thereby improving the robustness of the target motion trajectory. At the same time, the joint optimization algorithm of the disclosed embodiment can comprehensively consider the mutual motion between targets and scene constraints, so that the multi-target tracking method improved by the disclosed embodiment can adaptively adjust the tracking strategy according to the dynamic scene, further reducing the possibility of target loss and identity confusion. This long-term motion learning mechanism is particularly suitable for dealing with scenes with large-scale occlusion or short-term disappearance of targets. In addition, although the complex multi-pedestrian tracking (i.e., multi-target tracking) task requires processing a large amount of data association and calculation, the disclosed embodiment fully considers the control of computational complexity while improving the tracking performance. By introducing incremental learning and adaptive update mechanisms, the disclosed embodiment can dynamically update the target's motion model (i.e., graph model) and scene constraints, avoiding frequent global optimization operations and reducing computational costs. This incremental learning mechanism only needs to process small-scale local updates, which not only maintains the real-time nature of the tracking process, but also reduces the consumption of memory and computing resources. In addition, the target's motion range is limited by scene constraints, which reduces unnecessary motion prediction and trajectory matching operations, thereby improving the overall efficiency of the multi-target tracking method improved by the embodiment of the present disclosure. It can significantly improve the real-time response capability of the system while maintaining a high tracking accuracy, and is particularly suitable for use in scenarios with high computing resource requirements, such as security monitoring and unmanned driving.

[0081] Combination Figure 2As shown, the embodiment of the present disclosure provides a multi-target tracking system, including: an acquisition module, a determination module, a prediction module, a tracking module and a trajectory module. Among them, the acquisition module is used to acquire the target detection information of each frame image in the video sequence to be tracked, and determine the target tracking information of the previous preset continuous frame images according to each target detection information, and use the target tracking information of the previous preset continuous frame images as reference data. The determination module is used to determine the interaction information between each target in the last frame image in the reference data and determine the scene constraint information according to the reference data. The prediction module is used to determine the predicted tracking information of the next frame image of the reference data based on the target tracking information of the last frame image in the reference data, the corresponding interaction information and the scene constraint information. The tracking module is used to determine the target tracking information of the next frame image of the reference data based on the target detection information and predicted tracking information of the next frame image of the reference data. The trajectory module is used to use the target tracking information and reference data of the next frame image of the reference data as new reference data, and repeatedly execute the steps in the acquisition module, the determination module, the prediction module, the tracking module and the trajectory module until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

[0082] The multi-target tracking system provided by the embodiment of the present disclosure obtains the target detection information of each frame image in the video sequence to be tracked and determines the target tracking information of the previously preset continuous frame images based on this information, so as to determine the interaction information and scene constraint information between each target in the last frame image in the reference data according to the target tracking information of each frame image in the reference data. In this way, the information association and decision-making ability in the tracking process can be enhanced through the interaction information and scene constraints between targets, and the accurate association between targets can be achieved, so that the target tracking information of the next frame image of the reference data can be predicted more accurately. When the target tracking information of the next frame of the reference data is determined based on the predicted tracking information and target detection information of the next frame of the reference data, the accuracy of the target tracking information can be improved, the accuracy and robustness of multi-target tracking can be significantly improved, and the accurate association between targets can be achieved to improve the accuracy of multi-target tracking.

[0083] Optionally, the determination module is specifically used to extract distance feature information between each target in each frame image in the reference data; based on the target tracking information of each frame image in the reference data and the distance feature information between each target, determine the interaction information between each target in the last frame image in the reference data.

[0084] Optionally, a determination module is specifically used to determine the nodes of the graph model based on the target tracking information of each frame image in the reference data; construct the edges of the graph model based on the distance feature information between each target in each frame image in the reference data; determine the weights of the edges between each node based on the nodes and edges of the graph model, and determine the interaction information between each target in the last frame image in the reference data based on the weights of each edge.

[0085] Optionally, the determination module is specifically used to determine the obstacle information of the last frame image in the reference data; determine the obstacle area and the walking area based on the obstacle information; and determine the scene constraint information based on the obstacle area and the walking area.

[0086] Optionally, the prediction module is specifically used to construct a total loss function of the joint optimization algorithm based on the interaction information and scene constraint information corresponding to the last frame image in the reference data to obtain a target joint optimization algorithm; based on the target tracking information of the last frame image in the reference data and the target joint optimization algorithm, determine the predicted tracking information of the next frame image of the reference data.

[0087] Optionally, the tracking module is specifically used to construct an IOU global matrix based on the target detection information and predicted tracking information of the next frame image of the reference data, and solve the global optimal solution; determine the target tracking information of the next frame image of the reference data based on the global optimal solution.

[0088] Optionally, a multi-target tracking system further includes a management module. The management module is used to determine the tracking trajectory of the new target in the video sequence if a new target is identified in the target tracking information of any frame image except the first frame image; and manage the tracking trajectory of each target in the video sequence based on the tracking trajectory of the new target in the video sequence.

[0089] Combination Figure 3 As shown, an embodiment of the present disclosure provides a computing device, including a processor and a memory. Optionally, the computing device may also include a communication interface and a bus. The processor, the communication interface, and the memory may communicate with each other through the bus. The communication interface may be used for information transmission. The processor may call the logic instructions in the memory to execute the multi-target tracking method of the above embodiment.

[0090] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0091] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes the function application and data processing by running the program instructions / modules stored in the memory, that is, implementing the multi-target tracking method in the above embodiment.

[0092] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory.

[0093] In an embodiment of the present invention, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium. When the instructions are executed, the multi-target tracking method in the above embodiment is executed.

[0094] The computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0095] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0096] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method or device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0097] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0098] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units can be only a logical function division, and there can be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0099] The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A multi-target tracking method, characterized in that: include: S1, obtaining target detection information of each frame image in a video sequence to be tracked, and determining target tracking information of previously preset continuous frame images according to each of the target detection information, and using the target tracking information of the previously preset continuous frame images as reference data; S2. Determine, according to the reference data, interaction information between various targets in the last frame image in the reference data and scene constraint information; S3, determining predicted tracking information of a next frame of the reference data based on the target tracking information of the last frame of the reference data, the corresponding interaction information and the scene constraint information; S4, determining target tracking information of a next frame image of the reference data based on the target detection information and predicted tracking information of the next frame image of the reference data; S5. Use the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeat S2-S5 until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

2. The method according to claim 1, characterized in that Determining, according to the reference data, the interaction information between the respective targets in the last frame image in the reference data includes: Extracting distance feature information between each target in each frame of the reference data; Based on the target tracking information of each frame image in the reference data and the distance feature information between the targets, the interaction information between the targets in the last frame image in the reference data is determined.

3. The method according to claim 2, characterized in that The step of determining the interaction information between the targets in the last frame of the reference data based on the target tracking information of each frame of the reference data and the distance feature information between the targets includes: Determine the nodes of the graph model based on the target tracking information of each frame of the reference data; Constructing edges of the graph model based on distance feature information between each target in each frame of the reference data; The weights of the edges between the nodes are determined based on the nodes and edges of the graph model, and the interaction information between the targets in the last frame image in the reference data is determined based on the weights of the edges.

4. The method according to claim 1, characterized in that The determining of scene constraint information includes: Determine obstacle information of the last frame of image in the reference data; Determining an obstacle area and a walking area based on the obstacle information; Scene constraint information is determined based on the obstacle area and the walking area.

5. The method according to claim 1, characterized in that The step of determining predicted tracking information of a next frame of the reference data based on target tracking information of a last frame of the reference data, corresponding interaction information, and scene constraint information includes: Constructing a total loss function of a joint optimization algorithm based on the interaction information and scene constraint information corresponding to the last frame of the image in the reference data to obtain a target joint optimization algorithm; Based on the target tracking information of the last frame image in the reference data and the target joint optimization algorithm, the predicted tracking information of the next frame image of the reference data is determined.

6. The method according to claim 1, characterized in that The determining the target tracking information of the next frame image of the reference data based on the target detection information and the predicted tracking information of the next frame image of the reference data comprises: Construct an IOU global matrix based on the target detection information and predicted tracking information of the next frame image of the reference data, and solve the global optimal solution; The target tracking information of the next frame image of the reference data is determined based on the global optimal solution.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: If a new target is identified in the target tracking information of any frame of image except the first frame of image, determining the tracking trajectory of the new target in the video sequence; The tracking trajectory of each target in the video sequence is managed based on the tracking trajectory of the new target in the video sequence.

8. A multi-target tracking system, characterized in that: include: An acquisition module, used to acquire target detection information of each frame image in the video sequence to be tracked, and determine target tracking information of previously preset continuous frame images according to each of the target detection information, and use the target tracking information of the previously preset continuous frame images as reference data; A determination module, used to determine, based on the reference data, interaction information between various targets in the last frame image of the reference data and scene constraint information; A prediction module, configured to determine predicted tracking information of a next frame of the reference data based on target tracking information of a last frame of the reference data, corresponding interaction information, and scene constraint information; A tracking module, configured to determine target tracking information of a next frame image of the reference data based on target detection information and predicted tracking information of a next frame image of the reference data; The trajectory module is used to use the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeatedly execute the steps in the acquisition module, the determination module, the prediction module, the tracking module and the trajectory module until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

9. A computing device, characterized in that The invention comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the multi-target tracking method according to any one of claims 1 to 7 when running the program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the multi-target tracking method according to any one of claims 1 to 7.

Images