Multi-target tracking method and related equipment

Abstract

The invention discloses a multi-target tracking method and related equipment, and the method comprises the steps: carrying out the target detection of a current frame, outputting a corresponding target detection frame result, carrying out the state transition operation through a target motion dynamics model, and generating a motion prediction frame corresponding to each potential target object in the current frame; based on the confidence of a target detection frame contained in the detection frame information, according to the target detection frame and the motion detection frame, generating a corresponding cost matrix, the cost matrix being a generalized symmetric intersection-to-union ratio or a symmetric overlapping score; and inputting the cost matrix into a Hungary matching system, generating an identity label corresponding to the target detection frame, and carrying out association matching on the identity label and the current activation track. According to the embodiment of the invention, the multi-target tracking processing efficiency can be improved, and the robustness, precision and application value of a multi-target tracking system in a complex dynamic environment can be remarkably improved. The method can be widely applied to the technical field of computer vision tracking.

Description

Technical Field

[0001] This application relates to the field of computer vision tracking technology, and in particular to a multi-target tracking method and related equipment. Background Technology

[0002] In the field of computer vision, multi-target tracking is a key technology widely used in video surveillance, intelligent transportation, and behavior analysis. Its core objective is to detect and continuously track multiple targets (such as pedestrians and vehicles) from a continuous video sequence, assigning a stable identity identifier (ID) to each target. Current mainstream multi-target tracking methods can be broadly categorized into two types: detection-based tracking methods and end-to-end tracking methods based on joint modeling.

[0003] Detection-based tracking methods have the following main drawbacks: 1) Detection, prediction, and association are performed in separate steps, which can easily lead to error accumulation; 2) They rely on manually designed matching rules (such as IoU threshold and ReID cosine distance threshold), which have limited adaptability to occluded scenarios and can easily lead to ID switching; 3) The trajectory management logic is complex.

[0004] End-to-end tracking methods based on joint modeling (such as MOTR and TransTrack) have the following main drawbacks: 1) They rely on large-scale training data, have insufficient model generalization, and need to be retrained across different scenarios; 2) They use neural networks to replace Kalman filters to predict motion, resulting in high computational complexity and significantly slower inference speed than traditional methods, making it difficult to meet real-time requirements; 3) The tracking performance is highly sensitive to the performance of the feature extraction network.

[0005] In summary, current multi-target tracking methods suffer from problems such as unstable association, inaccurate motion prediction, and discontinuous identity management in practical applications. Especially in complex scenarios, such as when targets are frequently occluded, move rapidly, or re-enter the frame, existing algorithms often encounter problems such as matching errors, target drift, or accidental deletion, which seriously affect the robustness and practicality of the system. Summary of the Invention

[0006] The main objective of this application is to propose a multi-target tracking method and related equipment.

[0007] To achieve the above objectives, one aspect of this application proposes a multi-target tracking method, the method comprising the following steps:

[0008] Obtain the current frame; the current frame is one of the consecutive input frames of the video.

[0009] A lightweight target detection model is used to perform target detection on the current frame and output the corresponding target detection box results; the target detection box results include the target detection boxes and detection box information of one or more potential target objects in the current frame;

[0010] For each potential target object, based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, a state transition operation is performed through the target motion dynamics model to generate the motion prediction box corresponding to the potential target object.

[0011] Based on the confidence level of the target detection box contained in the detection box information, a corresponding cost matrix is ​​generated according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap score.

[0012] The cost matrix is ​​input into the Hungarian matching system to generate the identity identifier corresponding to the target detection box, and the identity identifier is associated and matched with the current active trajectory.

[0013] In some embodiments, before generating the motion prediction box corresponding to each potential target object by performing a state transition operation through a target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, the method further includes:

[0014] Based on a preset trend judgment threshold;

[0015] Based on the trend judgment threshold, multiple corresponding trend judgment frames are determined from the continuous input frames of the video.

[0016] Determine whether the current frame is the trend judgment frame;

[0017] If the current frame is the trend judgment frame, obtain the previous trend judgment frame corresponding to the current frame, determine the global motion trend information corresponding to the current frame based on the current frame and the previous trend judgment frame, and determine whether to perform motion compensation operation on all the target detection boxes of the current frame based on the global motion trend information.

[0018] In some embodiments, the step of generating a motion prediction box corresponding to each potential target object by performing a state transition operation through a target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame includes:

[0019] Based on the posterior state vector and covariance matrix of the previous time frame, the sampling point set corresponding to the potential target object is constructed through square root decomposition, and the weighting factor is calculated in combination with the preset scaling parameters.

[0020] The sampling point set is input into the target dynamic model, nonlinear mapping is performed, and the mapping result is output.

[0021] Based on the mapping result and the weighting factor, a weighted fusion is performed to generate the motion prediction box.

[0022] In some embodiments, based on the confidence level of the target detection box included in the detection box information, a corresponding cost matrix is ​​generated according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap score, including:

[0023] When the detection box information corresponding to the potential target object shows that the confidence of the target detection box exceeds a preset first confidence threshold, the generalized symmetric intersection-union ratio is calculated based on the target detection box and the motion prediction box and used as the cost matrix.

[0024] When the detection box information corresponding to the potential target object shows that the confidence of the target detection box is less than the preset first confidence threshold but greater than the preset second confidence threshold, the symmetric overlap score is calculated based on the target detection box and the motion prediction box and used as the cost matrix.

[0025] In some embodiments, determining the global motion trend information corresponding to the current frame based on the current frame and the previous trend judgment frame, and determining whether to perform motion compensation operation on all the target detection boxes in the current frame based on the global motion trend information, includes:

[0026] Based on the current frame and the previous trend judgment frame, the mean square error corresponding to the grayscale image between the two frames is calculated, and the sampled corner matching point set is extracted based on the sparse optical flow method. The average Euclidean displacement between all matching points is calculated based on the corner matching point set, and the average Euclidean displacement is used as the sparse optical flow velocity. The global motion trend information includes the mean square error and the sparse optical flow velocity.

[0027] If the mean square error is greater than a preset average error threshold and the sparse optical flow velocity is greater than a preset optical flow velocity threshold, it is determined that the current frame has a motion trend; otherwise, it is determined that the current frame does not have a motion trend.

[0028] When it is determined that there is a motion trend in the current frame, the current affine transformation matrix corresponding to the current frame is calculated based on all matching point pairs contained in the corner matching point set and the random sampling consensus method.

[0029] Get the previous historical compensation matrix;

[0030] Calculate the Euclidean distance between the previous historical compensation matrix and the current affine transformation matrix. If the Euclidean distance is less than a preset distance threshold, use the previous historical compensation matrix as the current compensation matrix; otherwise, use the inverse of the current affine transformation matrix as the current compensation matrix.

[0031] Geometric compensation is performed on each of the target detection boxes using the current compensation matrix.

[0032] In some embodiments, the method further includes:

[0033] Determine whether the current frame is the starting frame in the continuous input frames of the video;

[0034] If the current frame is the starting frame, for each target detection box, when the confidence of the target detection box is greater than a preset confidence threshold, an initial trajectory and its corresponding trajectory state vector are generated according to the detection box information, and the initial trajectory is marked as a live state.

[0035] Based on the preset position scale term and velocity scale term, a corresponding initial covariance matrix is ​​generated, wherein the value of the position scale term is less than that of the velocity scale term, and the initial covariance matrix is ​​set as a diagonal matrix;

[0036] Based on the trajectory state vector and the initial covariance matrix, an unscented Kalman filter model for the initial trajectory is established.

[0037] In some embodiments, the step of inputting the cost matrix into the Hungarian matching system to generate an identity identifier corresponding to the target detection box, and associating the identity identifier with the current activation trajectory, includes:

[0038] The identity identifier is associated and matched with all the currently active trajectories; the currently active trajectory is a sequence of target motion states with a unique identity identifier.

[0039] When the identity identifier successfully matches any of the currently active trajectories, the unscented Kalman filter state vector of the successfully matched current active trajectory is updated according to the target detection box. The unscented Kalman filter state vector includes position, velocity and covariance matrix.

[0040] When the identity identifier fails to match all of the currently active trajectories, the appearance features of the target detection box are extracted, and all unresolved motion trajectory features stored in the memory pool are traversed.

[0041] Calculate the cosine similarity between the apparent features and each of the pending state motion trajectory features. When there is a pending state motion trajectory feature whose cosine similarity is greater than a preset similarity threshold, mark the pending state motion trajectory corresponding to the pending state motion trajectory feature as a live state and associate it with the target detection box. Update the unscented Kalman filter state vector of the pending state motion trajectory according to the target detection box.

[0042] To achieve the above objectives, another aspect of this application provides a multi-target tracking device, the device comprising:

[0043] The first module is used to acquire the current frame; the current frame is one of the consecutive input frames of the video.

[0044] The second module is used to perform target detection on the current frame using a lightweight target detection model and output the corresponding target detection box results; the target detection box results include target detection boxes and detection box information of one or more potential target objects in the current frame;

[0045] The third module is used to generate a motion prediction box corresponding to each potential target object by performing a state transition operation through a target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame.

[0046] The fourth module is used to generate a corresponding cost matrix based on the confidence level of the target detection box contained in the detection box information, and according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or a symmetric overlap score.

[0047] The fifth module is used to input the cost matrix into the Hungarian matching system, generate the identity identifier corresponding to the target detection box, and associate and match the identity identifier with the current active trajectory.

[0048] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0049] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0050] The embodiments of this application include at least the following beneficial effects: This application provides a multi-target tracking method, apparatus, electronic device, and program product. This solution performs target detection on the current frame, outputs the corresponding target detection box results, performs state transition operations through a target motion dynamics model, and generates motion prediction boxes corresponding to each potential target object in the current frame, thereby achieving motion prediction and improving the accuracy of motion prediction. Based on the confidence level of the target detection boxes contained in the detection box information, a corresponding cost matrix is ​​generated according to the target detection boxes and motion detection boxes. The cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap fraction, realizing the establishment of data association between target detection boxes and motion detection boxes based on the confidence level of the target detection boxes to generate different types of cost matrices, solving the problem of unstable target matching. The cost matrix is ​​input into a Hungarian matching system to generate an identity identifier corresponding to the target detection box, and the identity identifier is associated and matched with the currently active trajectory, realizing identity tracking matching through the Hungarian matching system. This application can improve the processing efficiency of multi-target tracking and significantly enhance the robustness, accuracy, and application value of multi-target tracking systems in complex dynamic environments. Attached Figure Description

[0051] Figure 1 This is a flowchart of the multi-target tracking method provided in the embodiments of this application;

[0052] Figure 2 This is another flowchart of the multi-target tracking method provided in the embodiments of this application;

[0053] Figure 3 yes Figure 2 The flowchart of step S204 in the process;

[0054] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0055] Figure 5 This is a schematic diagram illustrating the calculation of GS IOU in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram illustrating the calculation of SOS in an embodiment of this application;

[0057] Figure 7 yes Figure 1 The flowchart of step S105 in the process;

[0058] Figure 8 This is another flowchart of the multi-target tracking method provided in the embodiments of this application;

[0059] Figure 9 This is a schematic diagram of the structure of the multi-target tracking device provided in the embodiments of this application;

[0060] Figure 10This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0062] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0063] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0066] 1) YOLO (You Only Look Once, a real-time object detection algorithm): This is an end-to-end, single-stage object detection algorithm that transforms the object detection problem into a regression problem, predicting the object's location and category simultaneously through a single forward propagation. YOLO is known for its speed and high detection efficiency and is widely used in real-time video analytics.

[0067] 2) DETR (Detection Transformer, a Transformer-based object detection algorithm): It is an end-to-end detection model proposed by Facebook AI. It introduces the Transformer architecture to replace the traditional manual design and post-processing (such as NMS). It predicts objects and their bounding boxes through a query mechanism, realizing full-sequence modeling of the detection task.

[0068] 3) ReID (Person Re-Identification): This is a cross-camera human identity matching technology. Its goal is to find images belonging to the same person as a given target image from a large-scale image / video library at different times and from different perspectives.

[0069] 4) Kalman Filter (KF): A recursive filter used for state estimation of linear systems, capable of optimally predicting the state (such as position and velocity) of a target in the presence of noise. In target tracking, KF is often used to smooth the target trajectory and predict the target's next position, and is suitable for uniform linear motion models.

[0070] 5) UKF (Unscented Kalman Filter): It is a nonlinear extension of the Kalman filter, which uses the unscented transform to process nonlinear systems and does not depend on the linearization model.

[0071] 6) IoU (Intersection over Union): This is a metric used in object detection and tracking to evaluate the degree of overlap between two bounding boxes (such as a predicted box and a ground truth box). It is calculated by dividing the area of ​​the intersection of the two boxes by the area of ​​their union, and its value ranges from [0,1]. A higher IoU indicates a greater overlap between the two boxes.

[0072] 7) ID Switch: In multi-target tracking, ID Switch refers to the phenomenon where the tracking system incorrectly assigns different IDs to the same target in consecutive frames, or incorrectly assigns the same ID to different targets. It usually occurs when targets are occluded, overlapped, their appearance changes, or their motion trajectories are close together, and is an important indicator for measuring tracking stability and correlation accuracy.

[0073] In related technologies, the current mainstream multi-target tracking methods can be broadly divided into two categories: detection-based tracking methods and end-to-end tracking methods based on joint modeling.

[0074] The main steps of detection-based tracking methods are as follows: First, use a target detector (such as YOLO or DETR) to detect all targets in each frame; then, associate targets between different frames through target state prediction (such as using KF) and feature matching (such as ReID or IOU calculation); finally, create, update, and terminate target trajectories based on the association information. In specific implementations, commonly used tracking frameworks include DeepSORT, ByteTrack, and BOT-SORT. DeepSORT extracts features through a ReID network, predicts the target's position in the next frame using a Kalman filter, then calculates the cosine distance between the ReID features of the detected boxes and the ReID features of the predicted boxes, and finally uses the Hungarian algorithm for matching. ByteTrack uses a Kalman filter to predict the position in the next frame, and performs a two-stage association of all detected boxes (including low-confidence boxes) by calculating the IOU distance between the detected boxes and the predicted boxes. BOT-SORT, based on ByteTrack, first uses motion camera compensation to correct the KF, and then uses the minimum value of the IOU distance and the newly designed cosine distance as the value of the cost matrix in the Hungarian algorithm for the first-stage association.

[0075] End-to-end tracking methods based on joint modeling learn target detection and cross-frame association simultaneously during the training phase using a single network architecture, thus directly outputting target trajectories with identification identifiers during the inference phase. For example, TransTrack, based on Transformer, uses two decoders: one outputting detection boxes and the other outputting tracking boxes, then matches them using the Kernel-Mount Merger (KM) algorithm. MOTR, based on Detection-Based Tracking (DETR), proposes a track query set. This track query set is input into the decoder network to generate tracking predictions and updated track queries for the current frame. This process is then repeated frame by frame throughout the entire video sequence, achieving true end-to-end multi-target tracking. Compared to detection-based tracking, TransTrack and MOTR both use neural networks to remember targets and replace Kernel-Mount Merger (KM) to predict target motion, mitigating the ID switching phenomenon caused by occlusion during tracking. However, the tracking performance of these methods heavily relies on the feature extraction network. In practical applications, on the one hand, insufficient generalization necessitates network retraining; on the other hand, the inference speed is much slower than KM, making them unsuitable for real-time tracking scenarios.

[0076] In summary, the aforementioned mainstream multi-target tracking methods suffer from problems such as unstable association, inaccurate motion prediction, and discontinuous identity management in practical applications. Especially in complex scenarios, such as when targets are frequently occluded, move rapidly, or re-enter the frame, existing algorithms often encounter problems such as matching errors, target drift, or accidental deletion, which seriously affect the robustness and practicality of the system.

[0077] In view of this, this application provides a multi-target tracking method and related equipment. This solution is based on a multi-target tracking method that enhances association with ReID and nonlinear motion prediction. It systematically innovates and improves the accuracy of data association, the rationality of target state modeling, and the mechanism for maintaining identity continuity from three aspects. It aims to solve the core problems that are common in current multi-target tracking systems in practical applications, such as unstable association, inaccurate motion prediction, and discontinuous identity management. Especially in complex scenarios, such as when the target is frequently occluded, moves quickly, or re-enters the frame, existing algorithms often have problems such as matching errors (ID Switch), target drift, or accidental deletion, which seriously affect the robustness and practicality of the system.

[0078] The multi-target tracking method provided in this application relates to the field of information technology. This multi-target tracking method can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the multi-target tracking method, but is not limited to the above forms.

[0079] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0081] Figure 1 This is an optional flowchart of the multi-target tracking method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0082] Step S101: Obtain the current frame; the current frame is one of the consecutive input frames of the video.

[0083] Step S102: Perform target detection on the current frame using a lightweight target detection model and output the corresponding target detection box results; the target detection box results include the target detection boxes and detection box information of one or more potential target objects in the current frame;

[0084] Step S103: For each potential target object, based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, a state transition operation is performed through the target motion dynamics model to generate the motion prediction box corresponding to the potential target object.

[0085] Step S104: Based on the confidence of the target detection box contained in the detection box information, generate the corresponding cost matrix according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap score.

[0086] Step S105: Input the cost matrix into the Hungarian matching system to generate the identity identifier corresponding to the target detection box, and associate and match the identity identifier with the current active trajectory.

[0087] Steps S101 to S105 as shown in the embodiments of this application improve the efficiency of multi-target tracking by organically integrating and innovating key links such as target detection, motion prediction, and identity tracking based on data association. This significantly enhances the robustness, accuracy, and application value of the multi-target tracking system in complex dynamic environments and effectively solves core problems such as unstable data association, inaccurate motion prediction, and discontinuous identity management in multi-target tracking.

[0088] In step S102 of some embodiments, optionally, a preloaded lightweight detection model Yolov8 is used to detect the current frame I. t Perform inference and output the target bounding box detection result B. t ={b t,1 ,b t,2 ,…,b t,n}, where b t,n Indicates the current frame I t The corresponding target detection box and detection box information for the nth potential target object. The detection box information includes the target object's position coordinates, size, category label, and confidence score.

[0089] Before step S103 in some embodiments, it is determined whether there is a motion trend in the current frame based on the preprocessed global motion trend. If there is, motion compensation is performed on the target box detection result. It is determined whether the current frame is the starting frame of the continuous video input frames. If so, the first frame trajectory is initialized for the current frame.

[0090] In some embodiments, refer to Figure 2 Before performing step S103, the method further includes steps S301 to S201 to S204.

[0091] Step S201: Based on a preset trend judgment threshold;

[0092] Step S202: Based on the trend judgment threshold, determine multiple corresponding trend judgment frames from the continuous input frames of the video;

[0093] Step S203: Determine whether the current frame is a trend judgment frame;

[0094] Step S204: If the current frame is a trend judgment frame, obtain the previous trend judgment frame corresponding to the current frame, determine the global motion trend information corresponding to the current frame based on the current frame and the previous trend judgment frame, and determine whether to perform motion compensation operation on all target detection boxes in the current frame based on the global motion trend information.

[0095] In some embodiments, refer to Figure 3 Step S204 may include, but is not limited to, steps S301 to S306.

[0096] Step S301: Based on the current frame and the previous trend judgment frame, calculate the mean square error corresponding to the grayscale image between the two frames, and extract the sampled corner matching point set based on the sparse optical flow method. Calculate the average Euclidean displacement between all matching points based on the corner matching point set, and use the average Euclidean displacement as the sparse optical flow velocity. The global motion trend information includes the average error and the sparse optical flow velocity.

[0097] Step S302: If the mean square error is greater than the preset mean square error threshold and the sparse optical flow velocity is greater than the preset optical flow velocity threshold, it is determined that the current frame has a motion trend; otherwise, it is determined that the current frame does not have a motion trend.

[0098] Step S303: When it is determined that there is a motion trend in the current frame, the current affine transformation matrix corresponding to the current frame is calculated based on all matching point pairs contained in the corner matching point set and the random sampling consensus method.

[0099] Step S304: Obtain the previous historical compensation matrix;

[0100] Step S305: Calculate the Euclidean distance between the previous historical compensation matrix and the current affine transformation matrix. If the Euclidean distance is less than a preset distance threshold, use the previous historical compensation matrix as the current compensation matrix and pass through the current compensation matrix. Otherwise, use the inverse matrix of the current affine transformation matrix as the current compensation matrix.

[0101] Step S306: Perform geometric compensation on each target detection box using the current compensation matrix.

[0102] In some embodiments, assuming the trend judgment threshold is k, if the current frame I t For the corresponding trend judgment frame, obtain the current frame I. t The corresponding previous trend judgment frame I t-k , will the current frame I t Compared with the previous trend judgment frame I t-k Convert each to grayscale image G t and G t-k Calculate the mean square error (MSE) between grayscale images. t G t-k Then, the inter-frame optical flow is calculated using the sparse optical flow method Lucas-Kanade, i.e., for the grayscale image G. t and G t-k A fast corner detection method is used to extract a set of matching corner points, which includes the grayscale image G. t Corresponding matching point set and grayscale image G t-k Corresponding matching point set in, and Forming matching point pairs, Then calculate the Euclidean distance for all matching point pairs, and then calculate the average Euclidean displacement. m is the number of matched point pairs. If MSE > δ mse (as per the preset average error threshold mentioned above), and (As mentioned above, the preset optical flow velocity threshold) is used to determine the current frame I. t There is a trend of movement.

[0103] According to each matching pair The current frame I is generated using RANSAC (Random Sample Consensus). t The corresponding current affine transformation matrix F t Then obtain the compensation matrix used last time, i.e., the previous historical compensation matrix F. t-1 If the current affine transformation matrix F t Compared with the previous historical compensation matrix F t-1 When the Euclidean distance between them is less than a preset distance threshold, the previous historical compensation matrix F is used. t-1 (Without inversion) is used as the current compensation matrix, i.e., inter-frame compensation reuse optimization is adopted to reduce redundant calculations; otherwise, the current affine transformation matrix F is used. t The inverse matrix F t -1 This serves as the current compensation matrix.

[0104] Assume F t -1 For the current compensation matrix, the current frame I t Positions of each target detection bounding box after inverse transform compensation:

[0105] In some embodiments, refer to Figure 4 Step S103 may include, but is not limited to, steps S401 to S403.

[0106] Step S401: Based on the posterior state vector and covariance matrix of the previous time frame, construct the sampling point set corresponding to the potential target object through square root decomposition, and calculate the weighting factor in combination with the preset scaling parameters.

[0107] Step S402: Input the sampling point set into the target dynamic model, perform nonlinear mapping, and output the mapping result;

[0108] Step S403: Based on the mapping results and weighting factors, perform weighted fusion to generate motion prediction boxes.

[0109] In step S104 of some embodiments, starting from the second frame of the continuous input video frames, the state of the surviving trajectory is predicted using UKF. The state transition function of UKF uses the target dynamics model f(x) = [cx + vx · Δt, cy + vy · Δt, w + vw · Δt, h + vh · Δt, vx, vy, vw, vh].

[0110] Specifically, firstly, according to the pre-set scaling parameters, for example, α = 10 -3 With β = 2 and κ = 0, we obtain the weighting factor: λ = α2 (n+κ)-n and

[0111] The current frame I is decomposed using Cholesky Decomposition (square root decomposition). t Previous time frame I t-Δt The covariance matrix, i.e., S = Cholesky(P) t-Δt ), where S is the lower triangular matrix after Cholesky decomposition, P t-Δt For the previous time frame I t-Δt The covariance matrix, where Δt is the inter-frame time interval, is given. The specific details of the 2n+1 sigma points are as follows:

[0112] χ (0) =I t-Δt ;

[0113] χ (i) =I t-Δt +γ·S[:,i],i=1,2,…n;

[0114] χ (i+n) =I t-Δt -γ·S[:,i],i=1,2,…n;

[0115] Where, χ (i) This represents the i-th sigma point, and the value of n can be set by the user, such as n=8;

[0116] Sigma points are a specially selected set of sampling points used for state estimation of nonlinear systems (such as variants of the Kalman filter), particularly widely used in the Unscented Kalman Filter (UKF). They accurately capture the mean and covariance of random variables, avoid linearization errors, and are suitable for nonlinear transformation problems.

[0117] Then, apply the above state transition function to each sigma point. in, χ represents the i-th sigma point after the state transition. (i) f(·) is the state transition function mentioned above; calculate the current frame I. t The predicted state vector is: in, Let the mean weight of the i-th sigma point be . The remainder is 1 / (2n+2λ); finally, through the current frame I t Predicted state vector Motion compensation transformation is performed on each target detection box to obtain the corresponding motion prediction box.

[0118] In some embodiments, the data association stage is entered: when the detection box information corresponding to the potential target object shows that the confidence of the target detection box exceeds a preset first confidence threshold, the GSIOU (Generalized Symmetric Intersection over Union) between the two boxes is calculated based on the target detection box and the motion prediction box and used as the cost matrix.

[0119] When the detection box information corresponding to the potential target object shows that the confidence of the target detection box is less than the preset first confidence threshold but greater than the preset second confidence threshold, the SOS (Symmetric Overlap Score) between the two boxes is calculated based on the target detection box and the motion prediction box and used as the cost matrix.

[0120] For example, suppose the target detection box is z t Target detection bounding box z t When the confidence level is higher than the preset first confidence threshold δ1, the target detection box z is considered to be... t It can perform identity matching and calculate the target detection box z. t Its motion prediction box The generalized symmetric intersection-union ratio (GSIOU) between the two is specifically calculated using the following formula:

[0121]

[0122] Where C is and z t The minimum closure of the two corresponding rectangles, such as Figure 5 As shown, A is The corresponding rectangle, B is z t The corresponding rectangular boxes, GSIOU9A and B), are the target detection boxes z. t Its motion prediction box The generalized symmetric intersection-union ratio GSIOU between them is used, and then GSIOU(A,B) is input as the cost matrix into the Hungarian matching system.

[0123] Assume the target detection box z t If the confidence level is less than the preset first confidence threshold δ1, but greater than the preset second confidence threshold δ2, the target detection box z is considered to be... t It can perform identity matching and calculate the target detection box z. t Its motion prediction box The SOS between the two is calculated using the following formula:

[0124]

[0125] Among them, such as Figure 6 As shown, A is The corresponding rectangle, B is the target detection box z. t The corresponding bounding box, SOS(A, B), is the target detection box z. t Its motion prediction box The symmetric overlap fractions between them are then used as the cost matrix input to the Hungarian matching system.

[0126] In the data association stage, for high-confidence target detection boxes, we draw on the penalty of GIOU for non-overlapping boxes and, based on SOS, innovatively propose to use GSIOU as the basis for association between detection results and prediction results in multi-target tracking. For low-confidence target detection boxes, we use SOS as the basis for association.

[0127] First, SOS is introduced into multi-target tracking, with a value range of [0,1]. The larger the value, the higher the correlation, and the smaller the value, the lower the correlation. Compared with the traditional IOU (Intersection over Union), SOS does not favor large bounding boxes, so it is less sensitive to occlusion, friendly to small targets, and does not increase computational complexity. Compared with GSIOU as the correlation basis, SOS is more tolerant.

[0128] Furthermore, in the event of trajectory association failure during the data association stage, a subsequent identity reactivation mechanism is used as a fallback, employing the cosine similarity of ReID features as the basis for trajectory association to maintain the continuity of the identity of the target in the field of view.

[0129] In some embodiments, the Hungarian matching system uses the Hungarian matching algorithm to perform identity matching based on the input cost matrix, generating identity identifiers for target detection boxes, specifically including:

[0130] 1) Successfully matched target detection boxes inherit the identity ID of the corresponding predicted boxes;

[0131] 2) Unmatched target detection boxes are assigned a new identity ID;

[0132] 3) Unmatched motion prediction boxes are marked as potentially missing targets.

[0133] In some embodiments, refer to Figure 7 Step S105 may include, but is not limited to, steps S501 to S504.

[0134] Step S501: Associate and match the identity identifier with all currently active trajectories; the currently active trajectory is a sequence of target motion states with a unique identity identifier;

[0135] Step S502: When the identity identifier successfully matches any currently active trajectory, the unscented Kalman filter state vector of the successfully matched current active trajectory is updated according to the target detection box. The unscented Kalman filter state vector includes position, velocity and covariance matrix.

[0136] Step S503: When the identity identifier fails to match all currently active trajectories, extract the appearance features of the target detection box and traverse all unresolved motion trajectory features stored in the memory pool.

[0137] Step S504: Calculate the cosine similarity between the apparent features and the motion trajectory features of each unresolved state. When there is an unresolved state motion trajectory feature with a cosine similarity greater than a preset similarity threshold, mark the unresolved state motion trajectory corresponding to the unresolved state motion trajectory feature as a live state and associate it with the target detection box. Based on the target detection box, update the unscented Kalman filter state vector of the unresolved state motion trajectory.

[0138] In some embodiments, lifecycle management of motion trajectories is performed, including operations such as initialization, updating, temporary retention after loss, and reactivation through appearance features, to achieve the maintenance of identity continuity and avoidance of accidental deletion for multiple targets in complex scenarios. Optionally, the lifecycle states of the motion trajectory include:

[0139] Active status: The motion trajectory successfully matches the target detection box in the current frame;

[0140] Ghost state: The number of consecutive frames with mismatched motion trajectories has not exceeded the threshold δ. ghost Marked as a ghost trail, stored in the memory pool awaiting reactivation;

[0141] Lost state: The number of consecutive unmatched motion trajectories exceeds the maximum ghost hold frame count N. ghost Stored in the memory pool for reactivation.

[0142] Optionally, each motion trajectory includes at least the following information: target identity identifier (ID), unscented Kalman filter (UKF) state vector, current lifecycle state label (such as live, ghost, or lost state), and historical appearance feature (ReID) set (if ReID matching has been activated).

[0143] If a certain trajectory is within δ ghost (e.g., frames 3–5) within a consecutive number of frames, the motion trajectory fails to be associated with any target detection box, meaning the number of consecutive unmatched frames does not exceed the threshold δ. ghostThe motion trajectory is then transferred to a ghost state. In the ghost state, the trajectory is no longer filtered and updated, but its state is retained for subsequent potential association attempts. This strategy improves the robustness of the multi-object tracking system to short-term occlusion or inter-frame detection failures.

[0144] For all pending motion trajectories (including ghost motion trajectories and lost motion trajectories), the ReID features of the last frame of the pending motion trajectory are extracted and stored in the memory pool. Subsequently, through the identity reactivation mechanism, an attempt is made to restore its live state. This strategy significantly reduces the ID switch phenomenon caused by short-term occlusion or cross-scene reproduction and enhances the long-term correlation capability of the multi-target tracking system.

[0145] In some embodiments, the currently active trajectory is a motion trajectory in a living state. Using the Hungarian matching system, based on the identity of the target detection box, all currently active trajectories are traversed, trajectory association is performed, and it is determined whether there is a currently active trajectory that matches the identity.

[0146] When an identity identifier successfully matches any currently active trajectory, the UKF update is performed on the successfully matched current active trajectory using the detection box information of the target detection box. Specifically, the current frame I is obtained. t Predictive covariance matrix Specifically as follows:

[0147]

[0148] Among them, W c (i) The covariance weighting factor for the i-th Sigma point.

[0149] Let Σ be the weighted mean of the Sigma points, and Q be the process noise, initialized to a small but non-zero value.

[0150] To enhance adaptability to sudden motion or occlusion jumps, a sliding residual mechanism is introduced. For each frame's updated motion trajectory, the residual between the motion prediction box and the target detection box is recorded. The residuals of the k nearest frames before the current frame are used to form a sliding window, and the average residual is calculated. like Greater than the set residual threshold δ r This increases the process noise Q, improving the model's response to sudden changes. Less than the residual threshold δ r This reduces process noise Q, improves filtering stability, and suppresses jitter. This is achieved by using proportional multiplication or addition to reduce and increase process noise.

[0151] Furthermore, considering that some motion abrupt changes may not be directly reflected in the position error, the "target scale change rate" is introduced as an auxiliary indicator, which is the aspect ratio 'a' of the previous time frame. t-Δt and the aspect ratio a of the current time frame. t According to a t-Δt and a t The rate of change of the scale is calculated. If the rate of change of scale Δ a Higher than the set threshold δ a The process noise Q also increases; based on the average residual and scale change rate Δ a The joint judgment can effectively avoid prediction lag caused by scale abrupt changes. Based on UKF, combined with sliding residual analysis and target scale threshold judgment, it can achieve accurate prediction of nonlinear motion of the target, significantly improving the robustness and stability of multi-target tracking.

[0152] Furthermore, the Kalman gain K is calculated. t Then, based on the Kalman gain K t Perform state updates and covariance updates as follows:

[0153]

[0154] Among them, S t For motion prediction boxes With the target detection box z t The covariance matrix, i.e., the innovation covariance matrix, x t K is the updated state vector of the current frame, i.e., the posterior state estimate. t For Kalman gain, Let z be the state-observation cross-covariance matrix. (i) Let z be the value mapped from the i-th Sigma point to the observation space, which is the target detection box z. t The corresponding parameter space, P t Let be the posterior covariance matrix.

[0155] Finally, the successfully matched current activation trajectory is related to the x mentioned above. t and P t Bind it and update its corresponding UKF parameters.

[0156] To improve the overall throughput efficiency of the system, each motion trajectory is bound to a UKF instance, a thread pool is used to support concurrent prediction and updates, and the UKF and covariance of all currently active trajectories are stored in a shared array structure, and a multi-core CPU platform is used to meet the real-time prediction requirements.

[0157] In some embodiments, when the identity fails to match all currently activated trajectories, an identity reactivation process is initiated: The ReID feature vector (as described above) of the target detection box is extracted using a pre-trained ReID model (such as Clip-ReID, ResNet-50, OSNet, etc.). Then, all unresolved motion trajectories in the memory pool are traversed. Unresolved motion trajectories include ghost state motion trajectories and lost state motion trajectories. The trajectory ReID features (as described above) corresponding to each unresolved motion trajectory are extracted. For each unresolved motion trajectory, the cosine similarity between the ReID feature vector of the target detection box and the trajectory ReID features corresponding to the unresolved motion trajectory is calculated. If the cosine similarity is greater than a preset similarity threshold δ, the reactivation process proceeds. reid The trajectory ReID of the unresolved motion trajectory is reactivated using the Hungarian matching system, restoring its liveness. Based on the target detection box information, the UKF of the unresolved motion trajectory is initialized. Simultaneously, during the traversal of the memory pool, frames exceeding the maximum ghost retention frame number N are maintained. ghost Unresolved motion trajectories are directly eliminated to save resources and reduce false matches.

[0158] This identity reactivation mechanism avoids redundant calculations caused by introducing ReID comparison during the data association stage, while retaining a high recovery capability for the target's identity after occlusion. It can effectively solve the problem of identity breakage caused by temporary occlusion or cross-camera reproduction, and significantly improve the identity continuity of target tracking.

[0159] In some embodiments, refer to Figure 8 The above method also includes steps S601 to S604.

[0160] Step S601: Determine whether the current frame is the starting frame in a continuous video input frame;

[0161] Step S602: If the current frame is the starting frame, for each target detection box, when the confidence of the target detection box is greater than the preset confidence threshold, an initial trajectory and its corresponding trajectory state vector are generated according to the detection box information, and the initial trajectory is marked as a live state.

[0162] Step S603: Generate the corresponding initial covariance matrix based on the preset position scale term and velocity scale term, wherein the value of the position scale term is less than that of the velocity scale term, and the initial covariance matrix is ​​set as a diagonal matrix.

[0163] Step S604: Based on the trajectory state vector and the initial covariance matrix, establish an unscented Kalman filter model for the initial trajectory.

[0164] In some embodiments, if the current frame is determined to be the starting frame in a series of consecutive video input frames, the first frame trajectory is initialized for the current frame. Optionally, for each target detection box b, If the confidence of the target detection box b is greater than the preset confidence threshold, the target detection box b is considered a target worth tracking. First, based on the position information [cx,cy,w,h] of the target detection box b, an initial trajectory and its trajectory state vector x0 = [cx,cy,w,h,vx,vy,vw,vh] are generated, where (cx,cy) are the center coordinates of the target detection box b, w is the width of the target detection box b, h is the height of the target detection box b, vx is the velocity of the potential target object corresponding to the target detection box b in the x-direction, vy is the velocity of the potential target object corresponding to the target detection box b in the y-direction, vw is the rate of change of the width of the potential target object corresponding to the target detection box b (usually initially 0), and vw is the rate of change of the height of the potential target object corresponding to the target detection box b (usually initially 0).

[0165] The initialized trajectory is designated as the live trajectory (trajectory state is active). Since there is no historical velocity information in the first frame, the velocity terms (vx, vy, vw, vh) are all 0. Then, based on the trajectory state vector and the initialized covariance matrix, a UKF (as described in the unscented Kalman filter model above) is established for the target detection box b. The initial covariance matrix P0 is set as a diagonal matrix. in, Represents the variance of the positional components. The variance of the dimensional components is represented. The variance of the velocity component and the position scale term are represented by... and Less than the velocity scale term This indicates that the initial velocity estimate is uncertain, for example... as well as

[0166] Please see Figure 9 This application also provides a multi-target tracking device that can implement the above-described method. The device includes:

[0167] The first module is used to acquire the current frame; the current frame is one of the consecutive input frames of the video.

[0168] The second module is used to perform target detection on the current frame using a lightweight target detection model and output the corresponding target detection box results; the target detection box results include the target detection boxes and detection box information of one or more potential target objects in the current frame;

[0169] The third module is used to generate motion prediction boxes corresponding to potential target objects by performing state transition operations through the target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame.

[0170] The fourth module is used to generate a corresponding cost matrix based on the confidence of the target detection box contained in the detection box information, and based on the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap score.

[0171] The fifth module is used to input the cost matrix into the Hungarian matching system, generate the identity identifier corresponding to the target detection box, and associate the identity identifier with the current active trajectory.

[0172] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0173] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0174] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0175] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0176] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0177] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0178] The input / output interface 903 is used to implement information input and output;

[0179] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0180] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0181] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0182] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0183] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0184] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0185] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0186] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The multi-target tracking method, apparatus, electronic device, storage medium, and program product provided in this application improve the accuracy of motion prediction by detecting targets in the current frame, outputting corresponding target detection boxes, performing state transition operations through a target motion dynamics model, and generating motion prediction boxes corresponding to each potential target object in the current frame. Based on the confidence level of the target detection boxes contained in the detection box information, a corresponding cost matrix is ​​generated according to the target detection boxes and motion detection boxes. The cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap fraction, establishing data association between target detection boxes and motion detection boxes based on the confidence level of the target detection boxes to generate different types of cost matrices, thus solving the problem of unstable target matching. The cost matrix is ​​input into a Hungarian matching system to generate an identity identifier corresponding to the target detection box, and the identity identifier is associated and matched with the currently active trajectory, achieving identity tracking matching through the Hungarian matching system. This application can improve the processing efficiency of multi-target tracking and significantly enhance the robustness, accuracy, and application value of multi-target tracking systems in complex dynamic environments.

[0188] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0189] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0192] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0193] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multi-target tracking method, characterized in that, The method includes the following steps: Obtain the current frame; the current frame is one of the consecutive input frames of the video. A lightweight target detection model is used to perform target detection on the current frame and output the corresponding target detection box results; the target detection box results include the target detection boxes and detection box information of one or more potential target objects in the current frame; For each potential target object, based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, a state transition operation is performed through the target motion dynamics model to generate the motion prediction box corresponding to the potential target object. Based on the confidence level of the target detection box contained in the detection box information, a corresponding cost matrix is ​​generated according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap score. The cost matrix is ​​input into the Hungarian matching system to generate the identity identifier corresponding to the target detection box, and the identity identifier is associated and matched with the current active trajectory.

2. The method according to claim 1, characterized in that, Before generating the motion prediction box corresponding to each potential target object by performing a state transition operation through a target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, the method further includes: Based on a preset trend judgment threshold; Based on the trend judgment threshold, multiple corresponding trend judgment frames are determined from the continuous input frames of the video. Determine whether the current frame is the trend judgment frame; If the current frame is the trend judgment frame, obtain the previous trend judgment frame corresponding to the current frame, determine the global motion trend information corresponding to the current frame based on the current frame and the previous trend judgment frame, and determine whether to perform motion compensation operation on all the target detection boxes of the current frame based on the global motion trend information.

3. The method according to claim 1, characterized in that, For each potential target object, based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame, a state transition operation is performed through a target motion dynamics model to generate a motion prediction box corresponding to the potential target object, including: Based on the posterior state vector and covariance matrix of the previous time frame, the sampling point set corresponding to the potential target object is constructed through square root decomposition, and the weighting factor is calculated in combination with the preset scaling parameters. The sampling point set is input into the target dynamic model, nonlinear mapping is performed, and the mapping result is output. Based on the mapping result and the weighting factor, a weighted fusion is performed to generate the motion prediction box.

4. The method according to claim 1, characterized in that, Based on the confidence level of the target detection box contained in the detection box information, a corresponding cost matrix is ​​generated according to the target detection box and the motion detection box; The cost matrix is ​​a generalized symmetric intersection-union ratio or symmetric overlap fraction, including: When the detection box information corresponding to the potential target object shows that the confidence of the target detection box exceeds a preset first confidence threshold, the generalized symmetric intersection-union ratio is calculated based on the target detection box and the motion prediction box and used as the cost matrix. When the detection box information corresponding to the potential target object shows that the confidence of the target detection box is less than the preset first confidence threshold but greater than the preset second confidence threshold, the symmetric overlap score is calculated based on the target detection box and the motion prediction box and used as the cost matrix.

5. The method according to claim 2, characterized in that, The step of determining the global motion trend information corresponding to the current frame based on the current frame and the previous trend judgment frame, and determining whether to perform motion compensation operation on all the target detection boxes in the current frame based on the global motion trend information, includes: Based on the current frame and the previous trend judgment frame, the mean square error corresponding to the grayscale image between the two frames is calculated, and the sampled corner matching point set is extracted based on the sparse optical flow method. The average Euclidean displacement between all matching points is calculated based on the corner matching point set, and the average Euclidean displacement is used as the sparse optical flow velocity. The global motion trend information includes the mean square error and the sparse optical flow velocity. If the mean square error is greater than a preset average error threshold and the sparse optical flow velocity is greater than a preset optical flow velocity threshold, it is determined that the current frame has a motion trend; otherwise, it is determined that the current frame does not have a motion trend. When it is determined that there is a motion trend in the current frame, the current affine transformation matrix corresponding to the current frame is calculated based on all matching point pairs contained in the corner matching point set and the random sampling consensus method. Get the previous historical compensation matrix; Calculate the Euclidean distance between the previous historical compensation matrix and the current affine transformation matrix. If the Euclidean distance is less than a preset distance threshold, use the previous historical compensation matrix as the current compensation matrix; otherwise, use the inverse of the current affine transformation matrix as the current compensation matrix. Geometric compensation is performed on each of the target detection boxes using the current compensation matrix.

6. The method according to claim 1, characterized in that, The method further includes: Determine whether the current frame is the starting frame in the continuous input frames of the video; If the current frame is the starting frame, for each target detection box, when the confidence of the target detection box is greater than a preset confidence threshold, an initial trajectory and its corresponding trajectory state vector are generated according to the detection box information, and the initial trajectory is marked as a live state. Based on the preset position scale term and velocity scale term, a corresponding initial covariance matrix is ​​generated, wherein the value of the position scale term is less than that of the velocity scale term, and the initial covariance matrix is ​​set as a diagonal matrix; Based on the trajectory state vector and the initial covariance matrix, an unscented Kalman filter model for the initial trajectory is established.

7. The method according to claim 1, characterized in that, The step of inputting the cost matrix into the Hungarian matching system to generate the identity identifier corresponding to the target detection box, and associating and matching the identity identifier with the current activation trajectory, includes: The identity identifier is associated and matched with all the currently active trajectories; the currently active trajectory is a sequence of target motion states with a unique identity identifier. When the identity identifier successfully matches any of the currently active trajectories, the unscented Kalman filter state vector of the successfully matched current active trajectory is updated according to the target detection box. The unscented Kalman filter state vector includes position, velocity and covariance matrix. When the identity identifier fails to match all of the currently active trajectories, the appearance features of the target detection box are extracted, and all unresolved motion trajectory features stored in the memory pool are traversed. Calculate the cosine similarity between the apparent features and each of the pending state motion trajectory features. When there is a pending state motion trajectory feature whose cosine similarity is greater than a preset similarity threshold, mark the pending state motion trajectory corresponding to the pending state motion trajectory feature as a live state and associate it with the target detection box. Update the unscented Kalman filter state vector of the pending state motion trajectory according to the target detection box.

8. A multi-target tracking device, characterized in that, The device includes: The first module is used to acquire the current frame; the current frame is one of the consecutive input frames of the video. The second module is used to perform target detection on the current frame using a lightweight target detection model and output the corresponding target detection box results; the target detection box results include target detection boxes and detection box information of one or more potential target objects in the current frame; The third module is used to generate a motion prediction box corresponding to each potential target object by performing a state transition operation through a target motion dynamics model based on the posterior state vector and covariance matrix of the potential target object in the previous time frame of the current frame. The fourth module is used to generate a corresponding cost matrix based on the confidence level of the target detection box contained in the detection box information, and according to the target detection box and the motion detection box; the cost matrix is ​​a generalized symmetric intersection-union ratio or a symmetric overlap score. The fifth module is used to input the cost matrix into the Hungarian matching system, generate the identity identifier corresponding to the target detection box, and associate and match the identity identifier with the current active trajectory.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images