A detection-adaptive visual multi-object tracking method

Abstract

This invention proposes a detection-adaptive visual multi-target tracking method. It constructs a similarity matrix between the detection results of different quality levels and the target trajectory to achieve cascaded matching, and uses an adaptive observation noise Kalman filter method to update the posterior state of the trajectory, achieving robust multi-target tracking in various complex environments. This method implicitly considers the center point distance and shape similarity between target boxes by calculating a corner distance penalty term, and adaptively adjusts the observation noise according to different quality detection results, greatly improving the robustness and accuracy of the multi-target tracking system in complex environments and reducing identity switching.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a detection-adaptive visual multi-target tracking method that can adapt to detection quality under different complex environments and achieve robust tracking of multiple targets. Background Technology

[0002] Multi-object tracking aims to detect and estimate the spatiotemporal trajectories of multiple objects in a dynamic scene within each frame of a given video. To accomplish this task, the objects must first be located within a frame, and then each object is assigned a unique ID. Trajectories are formed for each object with the same ID in consecutive frames. Multi-object tracking is a crucial task in computer vision, with wide applications in autonomous driving, video surveillance, and intelligent transportation.

[0003] The main frameworks for solving multi-target tracking tasks are Tracking By Detection (TBD) and Joint Detection and Tracking (JDT). The former divides the problem into two successive sub-tasks. The first task detects targets appearing in each frame using a pre-built detector. The second task associates the detection results with existing trajectories to track the same targets. The latter aims to solve both detection and tracking simultaneously. Through an end-to-end deep neural network, it not only provides the detection results for the current frame but also uses regression to replace data association between consecutive frames for identity transfer. While the shared network approach in the JDT paradigm can effectively improve inference speed, it suffers from difficulties in co-training the modules. In contrast, TBD, with its clear multi-stage structure, is easier to optimize, and its superior performance has made it a popular solution for multi-target tracking tasks.

[0004] The baseline of the TBD paradigm mainly consists of three parts: object detectors, motion models, and data association. Benefiting from the rapid development of deep learning, powerful object detectors can more effectively acquire targets of interest in a scene, thereby reducing the need for complex components in the tracker. Generally, TBD-based tracking frameworks focus more on how to extract instance-level identification features (including spatial and appearance information) of each potential surviving target from the detection results, and utilize these explicit or implicit cues to achieve better data association.

[0005] Motion models are frequently used for target state prediction. Typically, these models employ Bayesian estimation to estimate the target's state at the next time step by maximizing the posterior probability. The Kalman filter, one of the most classic Bayesian filters, is a recursive Bayesian filter that follows a typical prediction-update cycle. It assumes the true state is an unobserved Markov process, and measurements are observations from a Hidden Markov Model. Given that the linear motion assumption limits the application of the Kalman filter, algorithms such as the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF) were subsequently proposed, employing first- and third-order Taylor approximations to handle nonlinear motion. However, they still rely on the Gaussian prior of the approximate Kalman filter assumption and require motion pattern assumptions. On the other hand, particle filtering addresses nonlinear motion through sampling-based posterior estimation, but requires exponential computational costs. Besides classic filters, some algorithms choose a learning-based approach for motion prediction, such as CenterTrack and TraDes. These methods often employ a neural network to learn the spatiotemporal displacement of the same target between adjacent frames. Considering the high dependence of online tracking on real-time computation, the Kalman filter is commonly chosen as the motion model in visual multi-target tracking. Although the target motion can be approximated as linear between consecutive frames, the constant setting of observation noise further limits the Kalman filter. Therefore, in this invention, the Kalman filter is further improved to map the observation noise to a more reasonable space.

[0006] Spatial similarity is a crucial part of the data association computation process. The SORT algorithm proposes a distance calculation method based on location information. This method first uses a Kalman filter to predict the possible locations of trajectories in the current frame, and then calculates the IoU distance between these potential trajectory locations and the detection boxes to construct a cost matrix. However, the IoU distance ignores more information between candidate boxes, leading to association confusion in low frame rates or highly dynamic scenes. In this invention, the IoU distance is further improved by implicitly considering the center point distance and shape similarity between target boxes by using the distance between corresponding corner points as a penalty term. Summary of the Invention

[0007] The first objective of this invention is to address the shortcomings of existing technologies by proposing a visual multi-target tracking method based on adaptive observation noise, at least partially solving the problem of poor robustness in scenarios with varying levels of complexity. This invention constructs a similarity matrix between the detection results of different quality levels and the target trajectory to achieve cascaded matching, and uses a Kalman filter method with adaptive observation noise to update the posterior state of the trajectory, thus achieving robust multi-target tracking in various complex environments. This method implicitly considers the center point distance and shape similarity between target boxes by calculating a corner distance penalty term, and adaptively adjusts the observation noise according to different quality detection results, significantly improving the robustness and accuracy of the multi-target tracking system in complex environments and reducing identity switching.

[0008] This invention provides a detection-adaptive visual multi-target tracking method, comprising the following steps:

[0009] Step S1, Target Detection

[0010] Perform object detection on each frame of the video sequence, search for targets of interest, and construct a set of target detection boxes;

[0011] Each observation component in the set of bounding boxes for objects of interest is defined as a quintuple, denoted as... ;in, Represents the two-dimensional coordinates of the target center in the image, ( The scale representing the target. Indicates the detection confidence level of the target;

[0012] Step S2, Data Association

[0013] S2-1 Constructing high-resolution and low-resolution detection boxes:

[0014] Traverse the set of target-of-interest bounding boxes and classify them one by one according to the detection confidence to obtain high-scoring and low-scoring bounding boxes;

[0015] S2-2 initializes the first frame of the video image, which is recorded as the first frame of the survival trajectory;

[0016] S2-3 Determine whether the current frame image is the first frame image of the current trajectory. If not, proceed to step S2-4. If yes, create a corresponding trajectory set for the high-resolution detection box, initialize the state variables of the current trajectory using Kalman filtering, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory in the first frame. Repeat the current step S2-3.

[0017] S2-4 is a one-stage association to achieve matching between high-resolution detection frames and survival trajectories;

[0018] S2-5 two-stage association, which performs two-stage association matching between low-scoring detection boxes and mismatched trajectories;

[0019] Step S3: Perform lifecycle management on the data-associated trajectories to handle mismatched trajectories and detections in steps S2-4 and S2-5;

[0020] Step S4: Obtain all trajectories whose trajectory tracking status is alive in Step S3, and complete multi-target tracking of the entire video sequence;

[0021] Preferably, the target detection in step S1 uses the YOLOX frameless detector.

[0022] Preferably, step S2-1 specifically involves: defining detection boxes with a detection confidence score greater than a as high-scoring detection boxes, where 0 < a < 1; defining detection boxes with a detection confidence score greater than b and less than a as low-scoring detection boxes, where 0 < b < a < 1; and defining detection boxes with a detection confidence score less than b as false positives, which do not participate in matching.

[0023] Preferably, the state variable initialization in step S2-3 is defined as follows:

[0024]

[0025] in, This is represented by the two-dimensional coordinates and size of the target center in the image. This is represented as the velocity of the corresponding component;

[0026] Preferably, step S2-4 specifically involves:

[0027] S2-4-1 will use the posterior state of the current trajectory in the previous frame. Predicting the prior state estimate in the current frame using Kalman filtering. ;

[0028] S2-4-2 uses the prior state estimate of the current frame and the position similarity of the high-resolution detection box as the cost matrix, and solves it using Hungarian matching to obtain the matching result;

[0029] If the matching result is that the trajectory and the detection box are successfully matched, the posterior state of the corresponding trajectory is updated by Kalman filtering of the observation noise.

[0030] If the matching result is a trajectory mismatch, that is, a mismatch in the prior state estimation of the current frame, then step S2-3 is executed to perform a two-stage association between the mismatched trajectory and the low-scoring detection box.

[0031] If the matching result is a detection mismatch, i.e. a mismatch of high-scoring detection boxes, then step S3 is executed to send the mismatched high-scoring detection boxes to the trajectory management module.

[0032] Preferably, step S2-4-2 specifically involves:

[0033] The prior state estimate of the current frame And in the high-resolution detection box Calculate the coordinates of the top left and bottom right vertices of the bounding box according to equation (2), and define the results as the bounding box. and Their minimum bounding rectangle is ,in( , )express The coordinates of the top left vertex, ( , )express The coordinates of the lower right vertex;

[0034]

[0035] The IoU distance between the two bounding boxes is calculated according to equation (3), and then the relative distance between the corner points can be calculated according to equation (4). :

[0036]

[0037]

[0038] Relative distance of corner points Through weighting coefficients Harmonizing the standard IoU distance, a new distance metric, PMIoU, is introduced, as shown in equation (5); weight ratio It is a hyperparameter that measures IoU and The degree of importance;

[0039]

[0040] Using 1-PMIOU as the first-stage cost matrix, the matching results are obtained through the Hungarian algorithm.

[0041] Preferably, the step S2-4-2, which involves updating the posterior state of the corresponding trajectory using adaptive observation noise Kalman filtering, is as follows:

[0042] Update observation noise:

[0043]

[0044] in, This represents the confidence level of the k-th high-resolution detection box. This represents the i-th component of the unupdated observation noise, where i = u, v, w, h. This represents the i-th component of the observation noise after updating the confidence level. , Hyperparameters for controlling noise scale. > >1;

[0045] Next, determine whether the current trajectory tracking status is a newly generated trajectory or a surviving trajectory. If so, use the updated observation noise. Update the posterior state of the current frame trajectory, as shown in Equation (7); otherwise, consider the current trajectory tracking state as a lost trajectory and do not perform an update operation.

[0046]

[0047] in, For the test results, To observe the noise, For prior covariance, For posterior covariance, Kalman gain;

[0048] Preferably, step S2-5 specifically includes:

[0049] The prior state estimate of the current frame and the positional similarity of the high-resolution detection boxes are used as the cost matrix, which is then solved using Hungarian matching to obtain the matching results; specifically:

[0050] First, the PMIOU between the low-scoring detection box and the prior state estimate of the mismatched trajectory is calculated as the positional similarity, and a two-stage cost matrix is ​​constructed; then the bounding boxes are... and Redefining the prior state estimation of mismatched trajectories The low-scoring detection boxes are then used to calculate the PMIOU according to equation (5), and finally the matching results are obtained through the Hungarian algorithm.

[0051] If the matching result is that the trajectory and the detection box are successfully matched, it means that the tracking is successful. The corresponding detection is updated with the posterior state of the corresponding trajectory by Kalman filtering of the adaptive observation noise. In the second stage, since the low-scoring detection box represents the low-quality detection result, the detection result may have fuzziness, positional offset, false detection, etc. Consider using confidence to amplify the observation noise. The specific operation is as shown in equation (8).

[0052]

[0053] in, The detection threshold used in step 1 to distinguish between high-scoring and low-scoring detection boxes;

[0054] Use updated observation noise Update the posterior state of the current frame trajectory, as shown in equation (7);

[0055] If the matching result is a trajectory mismatch, then proceed to step 3 to send the mismatched trajectory frame to the trajectory management module;

[0056] If the matching result is a detection mismatch, then proceed to step 3 and send the mismatched detection box to the trajectory management module;

[0057] Preferably, step 3 is as follows:

[0058] The trajectory tracking status that is successfully matched in steps S2-4 and S2-5 is defined as a live trajectory.

[0059] The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the trajectory to be newly generated; if the mismatch is detected in the current frame and a match can be successfully generated in c consecutive frames, the trajectory tracking state of the current frame is considered to be updated to a new trajectory, and the state variable of Kalman filtering is initialized; otherwise, the mismatch detection in the current frame is considered to be an error detection, and the corresponding trajectory is directly deleted; c>1;

[0060] In steps S2-5, the trajectory tracking state corresponding to the mismatched trajectory is defined as the lost trajectory. If at least one frame in the subsequent d frames successfully associates with a low-resolution detection box or a high-resolution detection box, the posterior state of the Kalman filter of the mismatched trajectory is updated using the detection result, and the trajectory tracking state corresponding to the mismatched trajectory is updated to a live trajectory. If matching fails in all d consecutive frames, the trajectory tracking state corresponding to the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set. d > 1.

[0061] In steps S2-5, if a mismatch is detected, the mismatch detection in the current frame is considered an error, and the corresponding trajectory is directly deleted without participating in any operations in subsequent frames.

[0062] A second object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described thereon.

[0063] A third object of the present invention is to provide a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method.

[0064] The beneficial effects of this invention are:

[0065] This invention proposes a complete multi-target tracking scheme. To address issues such as identity swapping and trajectory fragmentation that easily occur in complex scenes, the detection set is decomposed, and a multi-stage association strategy based on the cascading approach is adopted. Considering the limitations of IoU as location information, a more robust PMIoU is proposed by using harmonic relative corner distance as a penalty term for IoU. Simultaneously, considering the inconsistency of observation noise, a Kalman filtering method based on adaptive observation noise is proposed, which exhibits stronger robustness to detection results of varying quality. Experiments on public datasets demonstrate that this invention maintains the robustness of the multi-target tracking algorithm under different detection quality results, while effectively improving tracking accuracy (IDF1 and HOTA) and reducing ID switching. Attached Figure Description

[0066] Figure 1 This is a flowchart of the present invention.

[0067] Figure 2 The results of multi-target tracking in pedestrian monitoring scenarios are shown in (a) frame 28, (b) frame 30, (c) frame 45, and (d) frame 50. Detailed Implementation

[0068] The present invention will be further analyzed below with reference to specific embodiments;

[0069] This embodiment provides a detection-adaptive visual multi-target tracking method, which can be applied to multi-target tracking processes in scenarios such as military, security monitoring, traffic management, biological science, and sports video analysis.

[0070] See Figure 1 The visual multi-target tracking method in this embodiment mainly includes the following steps:

[0071] Step S1, Target Detection

[0072] Perform object detection on each frame of the video sequence, search for targets of interest, and construct a set of target detection boxes;

[0073] Specifically, the target detection employs the YOLOX object detector without anchor boxes; the YOLOX object detector performs target detection on the input image sequence, completes the identification of targets of interest in the scene, and obtains the target's location information and confidence score.

[0074] The YOLOX algorithm consists of a backbone network, a neck network, and a head network. The original image size of the YOLOX network is 640×640×3. After processing by the Focus network structure and a CBS module, the output scale is 320×320×64 (the CBS module includes one convolution, batch normalization, and SiLu activation function processing). After one 3×3 convolution downsampling, the output scale is 160×160×128. Then, the stride of the first CBS component in each subsequent CSPlayer component is 2, so three 2x downsampling operations are performed. The three feature layer dimensions of the input neck network are 80×80×256, 40×40×512, and 20×20×1024, respectively. The extracted feature maps are fed into the neck network, and feature extraction is enhanced by the path aggregation network and the feature pyramid network. The shallow feature map details and the deep feature map semantic information are combined to achieve multi-scale feature fusion. Finally, the detection results are output by three decoupled heads: P3 is responsible for detecting small targets, P4 is responsible for detecting medium targets, and P5 is responsible for detecting large targets.

[0075] Meanwhile, in terms of data processing, the first half of the MOT Challenge video sequences were selected as the training set, and CrowdHuman, CityScapes, and VisDrone2019 were additionally mixed as supplementary training data, while the second half of the MOT Challenge video sequences were used as the test set.

[0076] Inputting the current frame image into the preceding YOLOX detector yields a set of bounding boxes for objects of interest. Each observation component in the bounding box set is defined as a quintuple, denoted as... ;in, Represents the two-dimensional coordinates of the target center in the image, ( The scale representing the target. This represents the detection confidence level of the target.

[0077] Step S2, Data Association

[0078] S2-1 Constructing high-resolution and low-resolution detection boxes:

[0079] The set of target-of-interest (ROI) detection boxes is traversed, and each ROI detection box is classified according to its detection confidence score to obtain high-scoring and low-scoring detection boxes. Detection boxes with a detection confidence score greater than threshold a (0.6 in this embodiment) are defined as high-scoring detection boxes; detection boxes with a detection confidence score greater than b (0.1 in this embodiment) and less than a (0.6 in this embodiment) are defined as low-scoring detection boxes; detection boxes with a detection confidence score less than b (0.1 in this embodiment) are defined as false positives and do not participate in matching.

[0080] S2-2 initializes the first frame of the video image, which is recorded as the first frame of the survival trajectory;

[0081] S2-3 Determine whether the current frame image is the first frame image of the current trajectory. If not, proceed to step S2-4. If yes, create a corresponding trajectory set for the high-resolution detection box, initialize the state variables of the current trajectory using Kalman filtering, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory in the first frame. Repeat the current step S2-3.

[0082] The state variables are initialized as follows:

[0083]

[0084] in, This is represented by the two-dimensional coordinates and size of the target center in the image. This is represented as the velocity of the corresponding component;

[0085] S2-4 is a one-stage association method to achieve matching between high-resolution detection bounding boxes and survival trajectories; specifically:

[0086] S2-4-1 will use the posterior state of the current trajectory in the previous frame. Predicting the prior state estimate in the current frame using Kalman filtering. :

[0087] Given a state transition model (motion model) Observation model Process noise and observation noise The following formula describes the process:

[0088]

[0089] in This represents the prior error covariance of the current frame. This represents the posterior error covariance of the previous frame. This represents the state transition model for the current frame t. This represents the process noise in the current frame t;

[0090] S2-4-2 uses the prior state estimate of the current frame and the positional similarity of the high-resolution detection boxes as the cost matrix, solves it using Hungarian matching, and obtains the matching results; specifically:

[0091] To better evaluate the difference between the detection box and the trajectory, a harmonic IoU distance based on corner distance is proposed (PMIoU). Specifically, it incorporates the prior state estimation of the current frame into the IoU distance. And in the high-resolution detection box Calculate the coordinates of the top left and bottom right vertices of the bounding box according to equation (2), and define the results as the bounding box. and Their minimum bounding rectangle is ,in( , )express The coordinates of the top left vertex, ( , )express The coordinates of the bottom right vertex;

[0092]

[0093] The IoU distance between the two bounding boxes is calculated according to equation (3), and then the relative distance between the corner points can be calculated according to equation (4). :

[0094]

[0095]

[0096] Relative distance of corner points Through weighting coefficients Harmonizing the standard IoU distance, a new distance metric, PMIoU, is introduced, as shown in equation (5); weight ratio It is a hyperparameter that measures IoU and The degree of importance;

[0097]

[0098] Using 1-PMIOU as the first-stage cost matrix, the matching results are obtained through the Hungarian algorithm. The idea behind the Hungarian algorithm is to find as many matches as possible, that is, to make each detected target match some historical trajectory. Its core approach is to start from an initial match and expand the match by finding augmenting paths until it can no longer be expanded.

[0099] The matching results of the first-stage association include three types:

[0100] (1) If the matching result is that the trajectory and the detection box are successfully matched, that is, the prior state estimate of the trajectory in the current frame and the high-resolution detection box are successfully matched, it means that the previous frame and the current frame are successfully tracked. The corresponding detection is updated by Kalman filtering of the adaptive observation noise. In the first stage, since the high-resolution detection box represents a high-quality detection result, confidence is considered to reduce the observation noise. The specific operation is as shown in equation (6).

[0101]

[0102] in, This represents the confidence level of the k-th high-resolution detection box. This represents the i-th component of the unupdated observation noise, where i = u, v, w, h. This represents the i-th component of the observation noise after updating the confidence level. , This represents the hyperparameters that control the noise scale. =10, =50;

[0103] Next, determine whether the current trajectory tracking status is a newly generated trajectory or a surviving trajectory. If so, use the updated observation noise. Update the posterior state of the current frame trajectory, as shown in Equation (7); otherwise, consider the current trajectory tracking state as a lost trajectory and do not perform an update operation.

[0104]

[0105] in, For the test results, To observe the noise, For prior covariance, For posterior covariance, Kalman gain;

[0106] (2) If the matching result is trajectory mismatch, that is, the prior state estimation of the current frame is mismatched, then execute step S2-4 to perform two-stage association between the mismatched trajectory and the low-scoring detection box;

[0107] (3) If the matching result is a detection mismatch, that is, a high-scoring detection box mismatch, then execute step 3 to send the mismatched high-scoring detection box to the trajectory management module;

[0108] S2-4 two-stage association involves performing a two-stage association matching between low-scoring detection boxes and mismatched trajectories; specifically:

[0109] First, calculate the PMIOU between the low-scoring detection box and the prior state estimate of the mismatched trajectory as the position similarity, and then construct a two-stage cost matrix.

[0110] bounding box and Redefining the prior state estimation of mismatched trajectories The low-scoring detection boxes are then used to calculate the PMIOU according to equation (5), and finally the matching results are obtained through the Hungarian algorithm.

[0111] The matching results of two-stage association include three types:

[0112] (1) If the matching result is that the trajectory and the detection box are successfully matched, it means that the tracking is successful. The corresponding detection is updated with the posterior state of the corresponding trajectory by Kalman filtering of the adaptive observation noise. In the second stage, since the low-scoring detection box represents the low-quality detection result, the detection result may have fuzziness, position offset, false positives, etc. Consider using confidence to amplify the observation noise. The specific operation is as shown in equation (8).

[0113]

[0114] in, The detection threshold for distinguishing between high-scoring and low-scoring detection boxes in step 1 is set to 0.6 in this embodiment;

[0115] Use updated observation noise Update the posterior state of the current frame trajectory, as shown in equation (7);

[0116] (2) If the matching result is a trajectory mismatch, then execute step 3 to send the mismatched trajectory frame to the trajectory management module;

[0117] (3) If the matching result is a detection mismatch, then execute step 3 and send the mismatched detection box to the trajectory management module;

[0118] Step 3: Trajectory lifecycle management, used to handle mismatched trajectories and detections from steps S2-4 and S2-5; details are as follows:

[0119] The trajectory tracking status that is successfully matched in steps S2-4 and S2-5 is defined as a live trajectory.

[0120] The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the trajectory to be newly generated; if the mismatch is detected in the current frame and a matching is successfully achieved in c consecutive frames (c=3 in this embodiment), the trajectory tracking state of the current frame is updated to a new trajectory and the state variable of the Kalman filter is initialized; otherwise, the mismatch detection in the current frame is considered an error detection, the corresponding trajectory is directly deleted, and the trajectory tracking state is not initialized.

[0121] In steps S2-5, the trajectory tracking state corresponding to the mismatched trajectory is defined as the lost trajectory. If at least one frame in the subsequent d frames (d=30 in this embodiment) successfully associates with a low-resolution detection box or a high-resolution detection box, the posterior state of the Kalman filter of the mismatched trajectory is updated using the detection result, and the trajectory tracking state corresponding to the mismatched trajectory is updated to a live trajectory. If the matching fails in all d consecutive frames, the trajectory tracking state corresponding to the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set.

[0122] In steps S2-5, if a mismatch is detected, the mismatch detection in the current frame is determined to be an error, and the corresponding trajectory is directly deleted without participating in any operations in subsequent frames.

[0123] Step 4: Based on Step 3, obtain all trajectories with a track tracking status of "surviving" to complete multi-target tracking of the entire video sequence.

[0124] Table 1 compares the performance metrics of the proposed method with those of classic multi-object tracking algorithms DeepSORT, JDE, and FairMOT on the MOT17 validation set. HOTA, MOTA, and IDF1 represent tracking performance, IDs represent the number of ID switching operations, and FPS represents inference speed. Comparative experiments show that the proposed method can effectively improve tracking accuracy and robustness in different scenarios while ensuring online tracking.

[0125] Table 1 Performance Comparison of Methods on the MOT17 Validation Set

[0126]

[0127]

[0128] The visualization results of the multi-target tracking effect of this invention in a pedestrian monitoring scenario are as follows: Figure 2 (a)- Figure 2 As shown in (d).

[0129] This invention provides an electronic device, specifically, the electronic device includes a memory and a processor, the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the embodiments.

[0130] The memory may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk drive. Communication between this system network element and at least one other network element is achieved through at least one communication interface (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0131] The bus can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0132] The memory is used to store programs. After receiving an execution instruction, the processor executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor or implemented by the processor.

[0133] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0134] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A detection-adaptive visual multi-object tracking method, characterized in that... The method includes the following steps: Step S1, Target Detection Perform object detection on each frame of the video sequence, search for targets of interest, and construct a set of target detection boxes; Step S2, Data Association S2-1 iterates through the set of target interest detection boxes and performs hierarchical processing on the target interest detection boxes one by one according to the detection confidence to obtain high-scoring detection boxes and low-scoring detection boxes; S2-2 initializes the first frame of the video image, which is recorded as the first frame of the survival trajectory; S2-3 Determine whether the current frame image is the first frame image of the current trajectory. If not, proceed to step S2-4. If yes, create a corresponding trajectory set for the high-resolution detection box, initialize the state variables of the current trajectory using Kalman filtering, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory in the first frame. Repeat the current step S2-3. S2-4 is a one-stage association to achieve matching between high-resolution detection frames and survival trajectories; S2-5 two-stage association, which performs two-stage association matching between low-scoring detection boxes and mismatched trajectories; Step S3: Perform lifecycle management on the trajectory after data association; Step S4: Obtain all trajectories that are alive in the trajectory tracking status of Step S3, and complete the multi-target tracking of the entire video sequence.

2. The method according to claim 1, characterized in that, Step S2-1 is as follows: Detection boxes with a confidence level greater than the threshold a are defined as high-scoring detection boxes, 0 < a < 1; detection boxes with a confidence level greater than b and less than a are defined as low-scoring detection boxes, 0 < b < a < 1.

3. The method according to claim 1, characterized in that, Step S2-4 specifically refers to: S2-4-1 will use the posterior state of the current trajectory in the previous frame. Predicting the prior state estimate in the current frame using Kalman filtering. ; S2-4-2 uses the prior state estimate of the current frame and the position similarity of the high-resolution detection box as the first-stage cost matrix, and solves it using Hungarian matching to obtain the matching result; If the matching result is that the trajectory and the detection box are successfully matched, the posterior state of the corresponding trajectory is updated by Kalman filtering of the observation noise. If the matching result is a trajectory mismatch, that is, a mismatch in the prior state estimation of the current frame, then step S2-3 is executed to perform a two-stage association between the mismatched trajectory and the low-scoring detection box. If the matching result is a detection mismatch, i.e. a mismatch of high-scoring detection boxes, then step S3 is executed to perform trajectory management on the mismatched high-scoring detection boxes.

4. The method according to claim 3, characterized in that, Step S2-4-2 specifically is: The prior state estimate of the current frame And in the high-resolution detection box Calculate the coordinates of the top-left and bottom-right vertices of the bounding box. The bounding box represents the two-dimensional coordinates and size of the target center in the image. , Their minimum bounding rectangle is ,in( , )express The coordinates of the top left vertex, ( , )express The coordinates of the lower right vertex; Calculate the IoU distance between the two bounding boxes, and then calculate the relative distance between the corner points. : Based on the relative distance of the corner points IoU distance calculation distance metric PMIoU: in Indicates the weighting coefficient; Using 1-PMIOU as the first-stage cost matrix, the matching results are obtained through the Hungarian algorithm.

5. The method according to claim 3, characterized in that, The specific steps in S2-4-2, which involve updating the posterior state of the corresponding trajectory using adaptive observation noise Kalman filtering, are as follows: Update observation noise: in, This represents the confidence level of the k-th high-resolution detection box. This represents the i-th component of the unupdated observation noise, where i = u, v, w, h. This represents the i-th component of the observation noise after updating the confidence level. , Hyperparameters for controlling noise scale. > >1; Next, determine whether the current trajectory tracking status is a newly generated trajectory or a surviving trajectory. If so, use the updated observation noise. Update the posterior state of the current frame trajectory, as shown in Equation (5); otherwise, consider the current trajectory tracking state as a lost trajectory and do not perform an update operation. in, For the test results, To observe the noise, For prior covariance, For posterior covariance, This is the Kalman gain.

6. The method according to claim 1, characterized in that, Step S2-5 specifically refers to: The prior state estimate of the current frame and the position similarity of the high-resolution detection box are used as the cost matrix, and Hungarian matching is used to solve it to obtain the matching result. If the matching result is a successful match between the trajectory and the detection box, the posterior state of the corresponding trajectory is updated by using Kalman filtering of the adaptive observation noise; the updated observation noise is then used. Update the posterior state of the current frame trajectory; If the matching result is a trajectory mismatch, then proceed to step S3 to manage the mismatched trajectory boxes. If the matching result is a detection mismatch, then step S3 is executed to manage the trajectory of the mismatched detection boxes.

7. The method according to claim 6, characterized in that, The adaptive observation noise Kalman filtering described in steps S2-5 to update the posterior state of the corresponding trajectory specifically involves: Update observation noise: in, This is the detection threshold used in step 1 to distinguish between high-scoring and low-scoring detection boxes.

8. The method according to claim 1, characterized in that, Step S3 is as follows: The trajectory tracking status that is successfully matched in steps S2-4 and S2-5 is defined as a live trajectory. The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the trajectory to be newly generated; if the mismatch is detected in the current frame and a match can be successfully generated in c consecutive frames, the trajectory tracking state of the current frame is considered to be updated to a new trajectory, and the state variable of Kalman filtering is initialized; otherwise, the mismatch detection in the current frame is considered to be an error detection, and the corresponding trajectory is directly deleted; c>1; In steps S2-5, the trajectory tracking status corresponding to the mismatched trajectory is defined as the lost trajectory; If at least one frame in the subsequent d frames is successfully associated with a low-resolution detection box or a high-resolution detection box, the posterior state of the Kalman filter of the mismatched trajectory is updated using the detection result, and the trajectory tracking state corresponding to the mismatched trajectory is updated to a live trajectory. If a match fails for d consecutive frames, the tracking status of the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set; d > 1; In steps S2-5, if a mismatch is detected, the detection of mismatch in the current frame is determined to be an error, and the corresponding trajectory is directly deleted.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method according to any one of claims 1-8.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images