Multi-target tracking method and related device

Abstract

The application relates to the field of multi-target tracking, and discloses a multi-target tracking method and related equipment. The method comprises the following steps: acquiring target detection information of each frame of image in a video sequence and determining target tracking information of a preset continuous frame of image in front, taking the target tracking information as reference data; determining interaction information and scene constraint information of the last frame of image in the reference data; determining predicted tracking information of the next frame of image of the reference data based on the target tracking information of the last frame of image in the reference data, the interaction information and the scene constraint information; determining target tracking information of the next frame of image of the reference data based on target detection information and predicted tracking information of the next frame of image of the reference data; taking the target tracking information of the next frame of image of the reference data and the reference data as new reference data, repeatedly executing the foregoing steps until target tracking information of each frame of image in the video sequence is obtained, so as to determine a tracking track of each target in the video sequence. The method can improve the accuracy of multi-target tracking.

Description

Technical Field

[0001] This application relates to the field of multi-target tracking technology, and in particular to a multi-target tracking method and related equipment. Background Technology

[0002] Multi-object tracking, as a mid-level task in computer vision, has broad application prospects, such as security monitoring, behavior analysis, autonomous driving, and pose estimation. The task of multi-object tracking is to take a video as input and output the trajectories of the objects appearing in the video. Due to the extensive research value of pedestrian tracking, multi-pedestrian tracking has become the mainstream research area in multi-object tracking.

[0003] In recent years, with the continuous improvement of detection algorithm performance, multi-pedestrian tracking based on detection-tracking frameworks has become the mainstream method for multi-pedestrian tracking. The principle of detection-tracking frameworks is to first detect pedestrians in each frame of the video, then extract the appearance or motion features of the pedestrians, perform data association, and form the final motion trajectory. Existing methods typically learn reliable motion patterns to match the same targets between adjacent frames, and learn discriminative appearance features to re-identify targets lost for a long time. However, during tracking, dense crowds and extreme occlusion can easily affect the reliability of motion prediction and the discriminability of appearance, making it difficult to achieve accurate association between targets, thus resulting in low accuracy in multi-target tracking.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a multi-target tracking method and related equipment to improve the accuracy of multi-target tracking.

[0007] In some embodiments, a multi-target tracking method includes: S1, acquiring target detection information for each frame of a video sequence to be tracked, determining target tracking information for a previous preset consecutive frame based on the target detection information, and using the target tracking information for the previous preset consecutive frame as reference data; S2, determining interaction information between targets in the last frame of the reference data and determining scene constraint information based on the reference data; S3, determining predicted tracking information for the next frame of the reference data based on the target tracking information, corresponding interaction information, and scene constraint information of the last frame of the reference data; S4, determining target tracking information for the next frame of the reference data based on the target detection information and predicted tracking information of the next frame of the reference data; S5, using the target tracking information of the next frame of the reference data and the reference data as new reference data, repeating S2-S5 until target tracking information for each frame of the video sequence is obtained, so as to determine the tracking trajectory of each target in the video sequence.

[0008] In some embodiments, a multi-target tracking system includes: an acquisition module, configured to acquire target detection information for each frame of a video sequence to be tracked, determine target tracking information for a previous preset consecutive frame based on the target detection information, and use the target tracking information for the previous preset consecutive frame as reference data; a determination module, configured to determine interaction information between targets in the last frame of the reference data and determine scene constraint information based on the reference data; a prediction module, configured to determine predicted tracking information for the next frame of the reference data based on the target tracking information, corresponding interaction information, and scene constraint information of the last frame of the reference data; a tracking module, configured to determine target tracking information for the next frame of the reference data based on the target detection information and predicted tracking information of the next frame of the reference data; and a trajectory module, configured to repeatedly execute the steps of the acquisition module, determination module, prediction module, tracking module, and trajectory module using the target tracking information of the next frame of the reference data and the reference data as new reference data, until target tracking information for each frame of the video sequence is obtained, so as to determine the tracking trajectory of each target in the video sequence.

[0009] In some embodiments, a computing device includes a processor and a memory storing program instructions, the processor being configured to execute the multi-target tracking method as described above when running the program instructions.

[0010] In some embodiments, a computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the multi-target tracking method as described above.

[0011] The multi-target tracking method and related equipment provided in this disclosure can achieve the following technical effects:

[0012] By acquiring target detection information from each frame of the video sequence to be tracked and using this information to determine target tracking information for the preceding consecutive frames, the interaction information and scene constraints between targets in the last frame of the reference data can be determined based on the target tracking information from each frame of the reference data. This enhances information association and decision-making capabilities during the tracking process through target interaction information and scene constraints, enabling accurate target association and thus allowing for more accurate prediction of target tracking information for the next frame of the reference data. Furthermore, by determining the target tracking information for the next frame of the reference data based on the predicted tracking information and target detection information of that next frame, the accuracy of target tracking information determination is improved, significantly enhancing the accuracy and robustness of multi-target tracking, achieving accurate target association, and improving the accuracy of multi-target tracking.

[0013] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0014] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0015] Figure 1 This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this disclosure;

[0016] Figure 2 This is a schematic diagram of the structure of a multi-target tracking system provided in an embodiment of this disclosure;

[0017] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0018] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0019] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0020] Unless otherwise stated, the term "multiple" means two or more.

[0021] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0022] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0023] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0024] This disclosure provides a multi-target tracking method and related equipment, including a multi-target tracking method, a system, a computing device, and a computer storage medium. In this embodiment, the multi-target tracking method is applied to an electronic device, such as a computer.

[0025] Combination Figure 1 As shown, this disclosure provides a multi-target tracking method, including:

[0026] Step S1: Obtain the target detection information of each frame in the video sequence to be tracked, determine the target tracking information of the previous preset consecutive frames based on each target detection information, and use the target tracking information of the previous preset consecutive frames as reference data.

[0027] Understandably, a video sequence can be a piece of surveillance video containing multiple targets, such as pedestrians. The video sequence S = {I1, I2, ..., I...} n}, I n This refers to the nth frame. The number of preceding consecutive frames is less than the total number of frames in the video sequence. For example, if the video sequence has 20 frames, then the number of preceding frames is less than 20.

[0028] In some embodiments, an object detection algorithm is used to identify each frame of the video sequence to identify the target (i.e., pedestrian) in each frame, with at least one target in each frame. The object detection algorithm is a YOLOX detector, and the object detection information represents the detection information of each target within the frame. The object detection information includes the coordinates of the detection box corresponding to each target (i.e., the position of the target), the confidence score corresponding to each detection box, and the class label corresponding to each detection box.

[0029] In some embodiments, target tracking information for previous consecutive frames is determined based on target detection information. This includes using Kalman filtering or SORT (Simple Online and Realtime Tracking) algorithms to perform tracking processing based on the target detection information of the previous consecutive frames, thereby determining the target tracking information for the previous consecutive frames. Specifically, Kalman filtering or SORT algorithms are used to construct motion trajectories based on the target's position in different frame images to determine the target tracking information for the previous consecutive frames. Target tracking information represents the tracking information of each target within a frame image. Target tracking information includes the coordinates of the tracking box corresponding to each target (i.e., the target's position) and the ID corresponding to each tracking box (i.e., the target's ID).

[0030] In some embodiments, the number of preset consecutive frames is greater than or equal to 2. Assuming the number of preset consecutive frames is 2, the first two preset consecutive frames are the first two frames of the video sequence, i.e., the first and second frames of the video sequence.

[0031] Step S2: Based on the reference data, determine the interaction information between each target in the last frame of the reference data and determine the scene constraint information.

[0032] It is understandable that the last frame in the reference data represents the last frame in the preceding pre-defined consecutive frame images corresponding to the reference data. Scene constraint information represents the set of scene constraints between various targets within the same frame image. Scene constraints represent the constraints that affect the mutual motion between targets, such as obstacles.

[0033] S3. Based on the target tracking information, corresponding interaction information, and scene constraint information of the last frame image in the reference data, determine the prediction tracking information of the next frame image of the reference data.

[0034] It is understandable that the next frame image of the reference data represents the next frame image of the last frame image in the reference data, that is, the next frame image of the last frame image in the previous preset consecutive frame images corresponding to the reference data.

[0035] S4. Based on the target detection information and prediction tracking information of the next frame image of the reference data, determine the target tracking information of the next frame image of the reference data.

[0036] S5. Using the target tracking information of the next frame of the reference data and the reference data as new reference data, repeat S2-S5 until the target tracking information of each frame of the video sequence is obtained, so as to determine the tracking trajectory of each target in the video sequence.

[0037] It is understood that this embodiment starts from a previously preset consecutive frame image (i.e., the first reference data), repeatedly uses the obtained target tracking information to predict the target tracking information of the next frame image (i.e., predicted tracking information), and determines the actual target tracking information of the next frame image based on the target detection information and predicted target tracking information of the next frame image. This continues until the target tracking information of the last frame image of the video sequence is predicted, and the actual target tracking information of the last frame image of the video sequence is determined based on the target detection information and predicted target tracking information of the last frame image of the video sequence. Finally, the trajectory of each target in the video sequence is updated based on the actual target tracking information of each frame image of the video sequence to obtain the tracking trajectory of each target in the video sequence. For example, the motion trajectory of target j is defined as: T j ={(x j,t ,y j,t )}|t∈[t0,t f ]. Among them, T j Let x be the trajectory of target j, including its position at different times. j,t ,y j,t The position coordinates of target j at time t, x j,t Indicates the horizontal position, y j,t Indicates the position in the vertical direction. t0 and t f Track the start and end times to represent the target's trajectory during that time period.

[0038] The multi-target tracking method provided in this disclosure acquires target detection information for each frame of a video sequence to be tracked and uses this information to determine target tracking information for previously preset consecutive frames. This allows for the determination of interaction information and scene constraints between targets in the last frame of the reference data based on the target tracking information of each frame. By leveraging the interaction information and scene constraints between targets, the information association and decision-making capabilities during tracking are enhanced, enabling accurate correlation between targets and allowing for more accurate prediction of target tracking information for the next frame of the reference data. Furthermore, by determining the target tracking information for the next frame of the reference data based on the predicted tracking information and target detection information of that next frame, the accuracy of target tracking information determination is improved, significantly enhancing the accuracy and robustness of multi-target tracking and achieving accurate correlation between targets to improve the accuracy of multi-target tracking.

[0039] Optionally, based on the reference data, the interaction information between each target in the last frame of the reference data is determined, including: extracting distance feature information between each target in each frame of the reference data; and determining the interaction information between each target in the last frame of the reference data based on the target tracking information and the distance feature information between each target in each frame of the reference data.

[0040] In this way, based on the target tracking information and distance feature information between each target in each frame of the reference data, it is possible to determine the motion changes of each target with other targets in different frames of the image, thereby determining the interaction information between each target.

[0041] In some embodiments, distance feature information represents relative distance and relative velocity, etc. The relative distance is Euclidean distance. Relative velocity is obtained from the velocity vectors of each target in the current frame. For example, in the second frame image, there are two targets, target i and target j, and the velocity vector of target i is V. i The velocity vector of target j is V j Then, in the second frame image, the relative velocity between target i and target j is V. ij =V i -V j In other embodiments, the distance feature information also includes relative acceleration, etc.

[0042] Furthermore, when there are many frames in the reference data, a preset number of consecutive frames can be selected from the reference data to determine the interaction information between targets within the last frame of the reference data. The selected preset number of consecutive frames must include the last frame of the reference data. For example, if the reference data contains 20 frames, and the last frame is frame 20, then frames 19 and 20 can be selected to determine the interaction information between targets within the last frame (i.e., frame 20) of the reference data.

[0043] Optionally, based on the target tracking information and distance feature information between each target in each frame of the reference data, the interaction information between each target in the last frame of the reference data is determined, including: determining the nodes of the graph model based on the target tracking information of each frame of the reference data; constructing the edges of the graph model based on the distance feature information between each target in each frame of the reference data; determining the weights of the edges between each node based on the nodes and edges of the graph model; and determining the interaction information between each target in the last frame of the reference data based on the weights of each edge. Here, the graph model represents a graph neural network (GNN).

[0044] In this way, by implementing a message passing mechanism using a graph model, each node can exchange information and update its own feature representation, which helps to capture the interaction between targets and obtain the interaction information between each target in the last frame of the reference image.

[0045] Specifically, determining the nodes of the graph model based on the target tracking information of each frame of the reference data includes: determining the feature vector of each target in each frame of the reference data based on the target tracking information of each frame of the reference data, and using each feature vector as a node in the graph model for the corresponding frame of the image. The feature vector includes position (x, y) and velocity (v). x v y Information such as acceleration.

[0046] Specifically, the edges of the graph model are constructed based on the distance feature information between each target in each frame of the reference data. This includes: if the distance (i.e., the relative distance) between two targets (i.e., two nodes) is within a preset threshold range, then an edge is established between the two targets, indicating that there is a relationship between them. Euclidean distance can be used as the distance metric, and velocity difference can be used as the metric for mutual motion to form weighted edges.

[0047] In some embodiments, the weighting of edges includes: distance weighting and velocity difference (i.e., relative velocity difference). The distance weighting can be the reciprocal of the Euclidean distance between two targets; that is, the smaller the distance, the larger the weighting, indicating a stronger interaction. The reciprocal of the velocity difference can also be used for edge weighting to represent the consistency of the target's movement trends. Targets with smaller velocity differences are assigned larger weights, indicating a potentially stronger interaction between the two targets.

[0048] In some embodiments, the message passing mechanism formula in the graph model is as follows:

[0049]

[0050] in, Let w be the feature representation of target i in the k-th layer of the graph neural network (i.e., graph model). σ is the activation function used to increase the non-linear expressive power of the model, typically ReLU or Sigmoid. (k) The weight matrix of the k-th layer of the graph neural network represents the weight parameters of the feature transformation. N(i) is the set of neighbor nodes of target i, representing other targets that have a relationship with target i. M is the message function, used to pass information from neighbor node j to target i, usually a weighted sum of the features of neighbor nodes. ω represents the weight of the edge between target i and target j output by the GNN (i.e., graph model). ij This reflects the degree of interaction between targets.

[0051] Furthermore, when there are many frames in the reference data, a preset number of consecutive frames can be selected to determine the interaction information between targets within the last frame of the reference data. The selected preset number of consecutive frames must include the last frame of the reference data. For example, if the reference data contains 20 frames, and the last frame is frame 20, then frames 19 and 20 can be selected to determine the interaction information between targets within the last frame (i.e., frame 20). Specifically, the nodes of the graph model are determined based on the target tracking information from frames 19 and 20; the edges of the graph model are constructed based on the distance features between targets within frames 19 and 20; the weights of the edges between nodes are determined based on the nodes and edges of the graph model; and the interaction information between targets within frame 20 is determined based on the weights of the edges. This approach not only obtains the interaction information between targets within the last frame of the reference data but also reduces computational load and improves the efficiency of multi-target tracking.

[0052] Specifically, a multi-target tracking method further includes: determining the interaction behavior between targets based on the interaction information between each target in the last frame of the reference data. This includes determining whether there are behaviors such as collision, attraction, or repulsion. This is helpful in predicting target tracking information in the next frame of the reference data.

[0053] Optionally, scene constraint information is determined, including: determining obstacle information of the last frame image in the reference data; determining obstacle region and walking region based on obstacle information; and determining scene constraint information based on obstacle region and walking region.

[0054] In this way, by determining the obstacle information in the last frame of the reference data, we can know the area where the target can move freely, so as to determine the scene constraints when the target moves, which is conducive to more accurate prediction of target tracking information.

[0055] Specifically, background subtraction algorithms (such as Gaussian mixture models) are applied to detect static objects or obstacles in a scene. For example, obstacle regions are extracted by comparing the background and foreground of consecutive frames to obtain obstacle information. Image segmentation algorithms (such as K-means clustering or watershed algorithms) are used to segment the scene and extract walking areas and obstacle regions.

[0056] Specifically, scene constraint information is determined based on obstacle and walking regions, including: determining the relationships between each target based on target tracking information from the obstacle region, walking region, and the last frame of the reference data. C obstacleAs scene constraints between targets in the last frame of the reference data, the set of scene constraints between targets in the last frame of the reference data is used as scene constraint information. Wherein, C obstacle This represents the constraint conditions for obstacles in the scene, indicating the relationship between the target and the obstacles. When it is 1, the target is in free space, that is, the target is in the walking area; when it is 0, the target should avoid moving to this area, which is the obstacle area.

[0057] In some embodiments, if the scene in the video sequence to be tracked is a static scene, that is, the scene of the video sequence does not change, the video sequence is a video shot under a fixed scene. In this case, the corresponding obstacle information can be obtained by identifying any frame in the reference data, and the corresponding obstacle information is used as the obstacle information of the last frame image in the reference data.

[0058] In other embodiments, if the scene in the video sequence to be tracked is a dynamic scene, meaning the scene of the video sequence changes, and the video sequence is a video shot in a dynamic scene, then to adapt to the changes in the dynamic scene, incremental learning and adaptive update mechanisms can be used to adjust the parameters of the Gaussian mixture model. For example, historical data can be used to update the model parameters, and the learning rate can be dynamically adjusted according to changes in the loss function. Updating model parameters using historical data:

[0059]

[0060] Where, θ t This represents the model's parameter vector at time t, which is continuously updated over time. η represents the learning rate, controlling the step size for parameter updates. This represents the gradient of the loss function with respect to the parameters, which guides the direction of model parameter updates.

[0061] Optionally, based on the target tracking information, corresponding interaction information, and scene constraint information of the last frame image in the reference data, the predicted tracking information of the next frame image of the reference data is determined, including: constructing the total loss function of the joint optimization algorithm based on the interaction information and scene constraint information corresponding to the last frame image in the reference data, and obtaining the target joint optimization algorithm; and determining the predicted tracking information of the next frame image of the reference data based on the target tracking information and the target joint optimization algorithm of the last frame image in the reference data.

[0062] In this way, by using a joint optimization algorithm to balance the losses of "motion consistency" and "scene constraints", the trajectory tracking results of the target are optimized, so as to accurately predict the last frame of the reference data and obtain the prediction tracking information of the next frame of the reference data.

[0063] Specifically, the overall loss function of the joint optimization algorithm is constructed based on the interaction information and scene constraint information corresponding to the last frame of the reference data, and the target joint optimization algorithm is obtained. This includes: determining the motion consistency loss based on the interaction information corresponding to the last frame of the reference data; determining the scene constraint loss based on the scene constraint information corresponding to the last frame of the reference data; and constructing the overall loss function of the joint optimization algorithm based on the motion consistency loss and the scene constraint loss.

[0064] Specifically, the motion consistency loss is determined based on the interaction information corresponding to the last frame of the reference data, including: determining the motion consistency loss using the following formula: Among them, Γ motion The motion consistency loss represents the degree of motion matching between targets, ω. ij The weights of the edges between each target (i.e., node) in the last frame of the reference data output by the graph model. i 2 j Let be the sum of squared Euclidean distances between target i and target j, used to maintain the consistency of the target trajectories; the square of the Euclidean distance amplifies the differences in distance, making the optimization focus more on those targets that are farther away, so as to ensure that their trajectories tend to be consistent.

[0065] Furthermore, since the weights of the edges corresponding to each frame and the Euclidean distances between each target differ, the expression... It can effectively incorporate the results of interactive behavior analysis into the loss, making motion consistency optimization more in line with the interactive needs in real-world scenarios.

[0066] Specifically, the scene constraint loss is determined based on the scene constraint information corresponding to the last frame image in the reference data, including: determining the scene constraint loss using the following formula:

[0067] Γ scene =∑ i,j C ij (D ij -D max ) 2 Among them, Γ scene The scene constraint loss represents the degree of fit between the target and scene constraints (such as obstacles), and is used to calculate the deviation between the distance between targets and scene limitations; C ij The scene constraints between target i and target j are: whether target i and target j will overlap with obstacles during their movement.

[0068] C ij for When C ijWhen C is 1, representing the motion between target i and target j, target i and target j will not overlap with obstacles. ij When the value is 0, it indicates that during the motion between target i and target j, target i and / or target j overlap with the obstacle. max The maximum allowed distance threshold is used to control the maximum distance between targets.

[0069] Specifically, the total loss function of the joint optimization algorithm is constructed based on motion consistency loss and scene constraint loss, including: the total loss function of the joint optimization algorithm is constructed using the following formula: Γ=α·Γ motion +β·Γ scene Where Γ is the total loss function of the joint optimization algorithm, used to optimize the tracking process, and α and β are weight parameters of the loss function, used to control the balance between motion consistency and scene constraints.

[0070] Optionally, based on the target detection information and prediction tracking information of the next frame image of the reference data, the target tracking information of the next frame image of the reference data is determined, including: constructing an IOU global matrix based on the target detection information and prediction tracking information of the next frame image of the reference data, and solving for the global optimal solution; and determining the target tracking information of the next frame image of the reference data based on the global optimal solution.

[0071] In this way, by constructing the IOU global cost matrix and solving for the global optimal solution, the target in the predicted tracking information can be determined to correspond to the target in the target detection information, thereby enabling a more efficient determination of the target tracking information of the next frame of the reference data.

[0072] Specifically, an IOU global cost matrix is ​​constructed based on each detection box in the target detection information of the next frame image of the reference data and each tracking box in the target tracking information. The Hungarian algorithm is then used to solve the matrix to obtain the global optimal solution, that is, to obtain the tracking box that matches each detection box. The ID of the tracking box is then assigned to the detection box that matches it to determine the ID of each detection box in the target detection information of the next frame image of the reference data. The target detection information with the assigned tracking box ID is then determined as the target tracking information.

[0073] Optionally, a multi-target tracking method further includes: if a new target is identified in the target tracking information of any frame other than the first frame, then the tracking trajectory of the new target in the video sequence is determined; and the tracking trajectories of each target in the video sequence are managed based on the tracking trajectory of the new target in the video sequence.

[0074] Because multiple targets are moving in a video sequence, situations can easily arise where targets are occluded or new targets appear. Therefore, by determining the tracking trajectory of new targets in the video sequence and managing the tracking trajectories of each target within the video sequence accordingly, the consistency of target IDs can be ensured, enabling continuous target tracking and accurate identification.

[0075] Specifically, the tracking trajectories of each target within the video sequence are managed based on the tracking trajectory of the new target. This includes: comparing the trajectory of the new target with various historical trajectories to obtain comparison results; determining whether similar historical trajectories exist based on the comparison results; and if so, merging the trajectory of the new target with the similar historical trajectories. That is, assigning the ID of the target corresponding to the similar historical trajectory to the new target. Historical trajectories represent trajectories whose trajectories are earlier than the new target's.

[0076] Understandably, if the comparison results indicate the existence of similar historical trajectories, it suggests that the new target is a re-emergence of an existing target that was occluded. In this case, merging the new target's trajectory with its corresponding historical trajectory ensures the consistency of the target ID and facilitates the management of the target's trajectory information.

[0077] Specifically, the trajectory of the new target is compared with the trajectory similarity of various historical trajectories to obtain the comparison results, including: ReID(T current T history ) = Sim(T current T history Among them, ReID(T) current T history ) is the measure of re-identification of the trajectory of the new target with the historical trajectory, and Sim is the comparison result.

[0078] Furthermore, a multi-target tracking method further includes: if the target detection information of the next frame of the reference data contains a target (i.e., a detection box) that cannot be matched with the target tracking information, and the unmatched target also fails to find a matching tracking box in the target tracking information of the frame image corresponding to a preset duration, then the trajectory of the target is terminated or deleted. If the target detection information of the next frame of the reference data contains a target (i.e., a detection box) that cannot be matched with the target tracking information, and the unmatched target can find a matching tracking box in the target tracking information of the frame image corresponding to a preset duration, then the trajectory of the target is initialized, that is, a new ID is assigned to the target, and a new trajectory is created.

[0079] In this way, by merging, terminating, or initializing targets that fail to match, the accumulation of targets can be managed, thus maintaining continuous tracking and accurate identification of targets.

[0080] This disclosure enhances the ability to capture target trajectories by combining short-term and long-term motion information. It predicts the future position of targets using long-term motion patterns (i.e., mutual information between targets within each frame) and incorporates scene geometric features (i.e., scene constraint information) into the constraints, thereby improving the robustness of the target's motion trajectory. Simultaneously, the joint optimization algorithm of this disclosure comprehensively considers the mutual motion between targets and scene constraints, enabling the improved multi-target tracking method to adaptively adjust the tracking strategy according to the dynamic scene, further reducing the possibility of target loss and identity confusion. This long-term motion learning mechanism is particularly suitable for dealing with scenarios involving large-scale occlusion or temporary target disappearance. Furthermore, although complex multi-pedestrian tracking (i.e., multi-target tracking) tasks require handling a large amount of data association and computation, this disclosure, while improving tracking performance, fully considers the control of computational complexity. By introducing incremental learning and adaptive update mechanisms, this disclosure can dynamically update the target's motion model (i.e., graph model) and scene constraints, avoiding frequent global optimization operations and reducing computational costs. This incremental learning mechanism only needs to handle small-scale local updates, maintaining the real-time nature of the tracking process and reducing memory and computational resource consumption. Furthermore, by limiting the target's motion range through scene constraints, unnecessary motion prediction and trajectory matching operations are reduced, thereby improving the overall efficiency of the multi-target tracking method improved in this embodiment. It significantly enhances the system's real-time response capability while maintaining high tracking accuracy, making it particularly suitable for scenarios with high computational resource requirements, such as security monitoring and autonomous driving.

[0081] Combination Figure 2As shown in the figure, this disclosure provides a multi-target tracking system, including: an acquisition module, a determination module, a prediction module, a tracking module, and a trajectory module. The acquisition module is used to acquire target detection information for each frame of the video sequence to be tracked, determine target tracking information for previous preset consecutive frames based on the target detection information, and use the target tracking information of the previous preset consecutive frames as reference data. The determination module is used to determine the interaction information between targets in the last frame of the reference data and determine scene constraint information based on the reference data. The prediction module is used to determine the predicted tracking information for the next frame of the reference data based on the target tracking information, corresponding interaction information, and scene constraint information of the last frame of the reference data. The tracking module is used to determine the target tracking information for the next frame of the reference data based on the target detection information and predicted tracking information of the next frame of the reference data. The trajectory module is used to repeatedly execute the steps in the acquisition module, determination module, prediction module, tracking module, and trajectory module, using the target tracking information of the next frame of the reference data and the reference data as new reference data, until the target tracking information for each frame of the video sequence is obtained, so as to determine the tracking trajectory of each target in the video sequence.

[0082] The multi-target tracking system provided in this disclosure acquires target detection information for each frame of a video sequence to be tracked and uses this information to determine target tracking information for previously preset consecutive frames. This allows for the determination of interaction information and scene constraints between targets in the last frame of the reference data based on the target tracking information of each frame. By leveraging the interaction information and scene constraints between targets, the system enhances information association and decision-making capabilities during tracking, enabling accurate target association and allowing for more accurate prediction of target tracking information for the next frame of the reference data. Furthermore, by determining the target tracking information for the next frame of the reference data based on the predicted tracking information and target detection information, the system improves the accuracy of target tracking information determination, significantly enhancing the accuracy and robustness of multi-target tracking, and achieving accurate target association to improve the accuracy of multi-target tracking.

[0083] Optionally, the determining module is specifically used to extract the distance feature information between each target in each frame of the reference data; based on the target tracking information and the distance feature information between each target in each frame of the reference data, the interaction information between each target in the last frame of the reference data is determined.

[0084] Optionally, the determining module is specifically used to determine the nodes of the graph model based on the target tracking information of each frame of the reference data; construct the edges of the graph model based on the distance feature information between each target in each frame of the reference data; determine the weights of the edges between each node based on the nodes and edges of the graph model; and determine the interaction information between each target in the last frame of the reference data based on the weights of each edge.

[0085] Optionally, the determining module is specifically used to determine obstacle information of the last frame image in the reference data; determine the obstacle region and the walking region based on the obstacle information; and determine scene constraint information based on the obstacle region and the walking region.

[0086] Optionally, the prediction module is specifically used to construct the total loss function of the joint optimization algorithm based on the interaction information and scene constraint information corresponding to the last frame image in the reference data, and obtain the target joint optimization algorithm; based on the target tracking information of the last frame image in the reference data and the target joint optimization algorithm, determine the prediction tracking information of the next frame image in the reference data.

[0087] Optionally, the tracking module is specifically used to construct an IOU global matrix based on the target detection information and predicted tracking information of the next frame image of the reference data, and solve for the global optimal solution; and determine the target tracking information of the next frame image of the reference data based on the global optimal solution.

[0088] Optionally, a multi-target tracking system further includes a management module. The management module is used to determine the tracking trajectory of the new target within the video sequence if a new target is identified in the target tracking information of any frame other than the first frame; and to manage the tracking trajectories of each target within the video sequence based on the tracking trajectory of the new target within the video sequence.

[0089] Combination Figure 3 As shown, this disclosure provides a computing device including a processor and a memory. Optionally, the computing device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can invoke logical instructions in the memory to execute the multi-target tracking method of the above embodiments.

[0090] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0091] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes the program instructions / modules stored in the memory to perform functional applications and data processing, thereby implementing the multi-target tracking method described in the above embodiments.

[0092] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0093] This invention provides a computer-readable storage medium storing instructions that, when executed, perform the multi-target tracking method described in the above embodiments.

[0094] The computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0095] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0096] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A multi-target tracking method characterized by, The method comprises the following steps: S1, obtaining target detection information of each frame image in a video sequence to be tracked, and determining target tracking information of a preset number of continuous frame images in advance according to each target detection information, and taking the target tracking information of the preset number of continuous frame images in advance as reference data; S2, determining interaction information between each target in the last frame image in the reference data and determining scene constraint information according to the reference data; wherein the scene constraint information is an overlap constraint between the target and the obstacle; S3, constructing a total loss function of a joint optimization algorithm based on the interaction information and the scene constraint information corresponding to the last frame image in the reference data, and obtaining a target joint optimization algorithm; determining the predicted tracking information of the next frame image of the reference data based on the target tracking information of the last frame image of the reference data and the target joint optimization algorithm; The total loss function of the joint optimization algorithm is constructed based on the interaction information and the scene constraint information corresponding to the last frame image in the reference data, and the target joint optimization algorithm is obtained, which comprises: determining a motion consistency loss based on the interaction information corresponding to the last frame image in the reference data; determining a scene constraint loss based on the scene constraint information corresponding to the last frame image in the reference data; and constructing a total loss function of a joint optimization algorithm based on the motion consistency loss and the scene constraint loss; The calculation formula corresponding to the total loss function is as follows: ； ； ； wherein, is a motion consistency loss, is a weight of an edge between each target in the last frame of the reference data output by a graph model, the graph model being constructed based on target tracking information of each frame of the reference data and distance feature information between each target within each frame; is a sum of squared Euclidean distances between target i and target j; is a maximum distance threshold value; is a scene constraint loss, is a scene constraint condition between target i and target j; is a total loss function of a target joint optimization algorithm, are weight parameters; S4, determining the target tracking information of the next frame image of the reference data based on the target detection information and the predicted tracking information of the next frame image of the reference data; S5, taking the target tracking information of the next frame image of the reference data and the reference data as new reference data, and repeating S2-S5 until the target tracking information of each frame image in the video sequence is obtained to determine the tracking trajectory of each target in the video sequence.

2. The method of claim 1, wherein, The interaction information between each target in the last frame image in the reference data is determined according to the reference data, which comprises: Extracting distance feature information between each target in each frame image in the reference data; Determining the interaction information between each target in the last frame image in the reference data based on the target tracking information of each frame image in the reference data and the distance feature information between each target.

3. The method of claim 2, wherein, The interaction information between each target in the last frame image in the reference data is determined based on the target tracking information of each frame image in the reference data and the distance feature information between each target, which comprises: Determining the nodes of a graph model based on the target tracking information of each frame image in the reference data; Constructing the edges of the graph model based on the distance feature information between each target in each frame image in the reference data; Determining the weight of each edge between the nodes based on the nodes and edges of the graph model, and determining the interaction information between each target in the last frame image in the reference data based on the weight of each edge.

4. The method of claim 1, wherein, The scene constraint information is determined, which comprises: Determining the obstacle information of the last frame image in the reference data; Determining the obstacle region and the walking region based on the obstacle information; Determine scene constraint information based on the obstacle region and the walking region.

5. The method of claim 1, wherein, The target detection information and the predicted tracking information of the next frame of the reference data are used to determine target tracking information of the next frame of the reference data, including: An IOU global matrix is constructed based on the target detection information and the predicted tracking information of the next frame of the reference data, and a global optimal solution is solved; The global optimal solution is used to determine the target tracking information of the next frame of the reference data.

6. The method according to any one of claims 1 to 5, characterized in that, Further comprising: If a new target is identified in the target tracking information of any frame of image except the first frame of image, determine the tracking trajectory of the new target in the video sequence; The tracking trajectories of each target in the video sequence are managed based on the tracking trajectory of the new target in the video sequence.

7. A multi-target tracking system, characterized by Comprise: An acquisition module is configured to acquire target detection information of each frame of image in a video sequence to be tracked, and determine target tracking information of a preset number of continuous frames of image in advance according to each target detection information, and use the target tracking information of the preset number of continuous frames of image in advance as reference data; A determination module is configured to determine interaction information between targets in a last frame of image in the reference data and determine scene constraint information according to the reference data; wherein the scene constraint information is an overlap constraint between a target and an obstacle; A prediction module is configured to construct a total loss function of a joint optimization algorithm based on the interaction information and the scene constraint information corresponding to the last frame of image in the reference data, and obtain a target joint optimization algorithm; and determine predicted tracking information of a next frame of image of the reference data based on target tracking information of the last frame of image of the reference data and the target joint optimization algorithm; The prediction module is specifically configured to determine a motion consistency loss based on the interaction information corresponding to the last frame of image in the reference data; determine a scene constraint loss based on the scene constraint information corresponding to the last frame of image in the reference data; and construct a total loss function of a joint optimization algorithm based on the motion consistency loss and the scene constraint loss; The total loss function corresponds to a calculation formula as follows: ； ； ； wherein, is a motion consistency loss, is a weight of an edge between each target in the last frame of reference data output by a graph model, the graph model being constructed based on target tracking information of each frame of the reference data and distance feature information between each target within each frame; is a sum of squared Euclidean distances between target i and target j; is a maximum distance threshold value; is a scene constraint loss, is a scene constraint condition between target i and target j; is a total loss function of a target joint optimization algorithm, are weight parameters; A tracking module is configured to determine target tracking information of a next frame of image of the reference data based on target detection information and predicted tracking information of the next frame of image of the reference data; A trajectory module is configured to repeatedly execute steps in the acquisition module, the determination module, the prediction module, the tracking module and the trajectory module by taking the target tracking information of the next frame of image of the reference data and the reference data as new reference data, until target tracking information of each frame of image in the video sequence is obtained, so as to determine a tracking trajectory of each target in the video sequence.

8. A computing device, comprising: The computer readable storage medium stores instructions, when the instructions are run on a terminal device, the terminal device executes the multi-target tracking method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions are run on a terminal device, the terminal device executes the multi-target tracking method as claimed in any one of claims 1 to 6.

Images