Visual multi-target tracking method based on detection self-adaption

Abstract

The invention provides a visual multi-target tracking method based on detection self-adaption, which comprises the following steps of: constructing a similarity matrix with a target track according to different quality detection results to realize cascade matching, and updating a posterior state of the track by adopting a Kalman filtering method of self-adaptive observation noise; and robust multi-target tracking in different complex environments is realized. According to the method, the penalty term of the angular point distance between the target frames is calculated, the central point distance and the shape similarity between the target frames are implicitly considered, and the observation noise is adaptively adjusted according to the detection results of different qualities, so that the robustness and the accuracy of the multi-target tracking system in a complex environment are greatly improved, and the identity switching is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and relates to a visual multi-target tracking method based on detection adaptation, which can adapt to the detection quality in different complex environments and realize robust tracking of multiple targets. Background Art

[0002] Multi-target tracking aims to detect and estimate the spatiotemporal trajectories of multiple targets in a dynamic scene in each frame of a given video. In order to complete the multi-target tracking task, it is first necessary to locate the target in a frame, and then assign a unique ID to each object. Each target with the same ID in consecutive frames will form a trajectory. Multi-target tracking is one of the most important tasks in computer vision and has a wide range of applications in autonomous driving, video surveillance, intelligent transportation and other fields.

[0003] The solution framework for multi-target tracking tasks is mainly divided into Tracking By Detection (TBD) and Joint Detection and Tracking (JDT). The former paradigm divides the problem into two consecutive subtasks. The first task is to detect the target appearing in each frame through the front detector. The second task is to perform data association between the detection results and the existing trajectory to complete the tracking of the same target. The latter paradigm aims to solve the detection and tracking tasks at the same time. Through an end-to-end deep neural network, it not only provides the detection results of the current frame, but also replaces the data association with regression between consecutive frames to transfer the identity. Although the shared network method in the JDT paradigm can effectively improve the inference speed, there is a problem of difficulty in collaborative training of various modules. The multi-stage structure in TBD is clear and easier to optimize. Its excellent performance makes it popular in the solution of multi-target tracking tasks.

[0004] The baseline of the TBD paradigm mainly consists of three parts: target detector, motion model, and data association. Benefiting from the rapid development of deep learning, powerful target detectors can more effectively acquire targets of interest in the scene, thereby reducing the need for complex components in the tracker. In general, the TBD-based tracking framework focuses more on how to mine instance-level identification features (including spatial and appearance information) of each potential surviving target from the detection results, and use these explicit or implicit clues to achieve better data association.

[0005] Motion models are often used to predict the state of a target. Typically, these motion models use Bayesian estimation to estimate the state of the target at the next moment by maximizing the posterior probability. Kalman filter is one of the most classic Bayesian filters and is a recursive Bayesian filter that follows a typical prediction-update cycle. It is assumed that the true state is an unobserved Markov process and the measurement is the observation from the hidden Markov model. Considering that the linear motion assumption limits the application of Kalman filter, algorithms such as extended Kalman filter and unscented Kalman filter were subsequently proposed, using first-order and third-order Taylor approximations to handle nonlinear motion. However, they still rely on Gaussian priors that approximate the Kalman filter assumption and require motion pattern assumptions. On the other hand, particle filtering solves nonlinear motion through sampling-based posterior estimation, but requires exponential computational costs. In addition to some classic filters, some algorithms choose to learn to predict motion, such as CenterTrack and TraDes. These methods often choose to design a neural network to learn the spatiotemporal displacement of the same target between adjacent frames. Considering the high dependence of online tracking on real-time calculations, Kalman filter is usually selected as the motion model in visual multi-target tracking. Although the target motion between consecutive frames can be approximately linear, the constant value setting for the observation noise further limits the Kalman filter. Therefore, in the present invention, the Kalman filter is further improved to map the observation noise to a more reasonable space.

[0006] Spatial similarity is an important part of calculating data association. The SORT algorithm proposes a distance calculation method based on position information. In this method, a Kalman filter is first used to predict the possible positions of the trajectories in the current frame, and then the IoU distances between these potential trajectory positions and the detection frames are calculated to construct a cost matrix. However, the IoU distance ignores more information between the candidate frames, resulting in confusion in associations in low frame rates or highly dynamic scenes. In the present invention, the IoU distance is further improved by taking the distance of the corresponding corner points as a penalty term, implicitly considering the center point distance and shape similarity between the target frames. Summary of the invention

[0007] The first purpose of the present invention is to propose a visual multi-target tracking method based on adaptive observation noise in response to the deficiencies of the prior art, which at least partially solves the problem of poor robustness in scenes of different complexity in the prior art. The present invention constructs a similarity matrix with the target trajectory based on detection results of different qualities to achieve cascade matching, and uses a Kalman filter method with adaptive observation noise to update the posterior state of the trajectory to achieve robust multi-target tracking in different complex environments. This method implicitly considers the center point distance and shape similarity between the target boxes by calculating the penalty term of the corner point distance between the target boxes, and adaptively adjusts the observation noise according to the detection results of different qualities, which greatly improves the robustness and accuracy of the multi-target tracking system in complex environments and reduces identity switching.

[0008] The present invention provides a method for visual multi-target tracking based on detection adaptation, comprising the following steps: Step S1: Target detection Perform target detection on each frame of the video sequence, search for the target of interest, and construct a set of detection frames of the target of interest; Each observation component in the set of object detection boxes of interest is defined as a five-tuple, expressed as ;in, represents the two-dimensional coordinates of the target center in the image, ( Indicates the scale of the target, Indicates the detection confidence of the target; Step S2: Data association S2-1 builds high-score detection frames and low-score detection frames: Traverse the set of detection frames of the target of interest, and grade the detection frames of the target of interest one by one according to the detection confidence to obtain high-score detection frames and low-score detection frames; S2-2 initializes the first frame image of the video, which is recorded as the first frame image of the survival trajectory; S2-3 determines whether the current frame image is the first frame image of the current trajectory. If not, execute step S2-4. If yes, create a corresponding trajectory set for the high-score detection frame, initialize the state variables of the Kalman filter for the current trajectory, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory of the first frame, and repeat the current step S2-3; S2-4 one-stage association to achieve matching between high-score detection boxes and survival tracks; S2-5 two-stage association, the low-score detection frame and the mismatched trajectory are matched in two stages; Step S3, performing lifecycle management on the trajectory after data association, for processing the mismatched trajectory and detection in step S2-4 and step S2-5; Step S4, obtaining all trajectories whose trajectory tracking status is survival trajectory in step S3, and completing multi-target tracking of the entire video sequence; Preferably, the target detection in step S1 adopts the anchor-free frame detector YOLOX.

[0009] Preferably, step S2-1 is specifically as follows: a detection frame whose detection confidence is greater than a threshold a is defined as a high-score detection frame, 0<a<1; a detection frame whose detection confidence is greater than b and less than a is defined as a low-score detection frame, 0<b<a<1; a detection frame whose detection confidence is less than b is defined as a false positive and does not participate in matching; Preferably, the state variable initialization in step S2-3 is defined as: in, It is expressed as the two-dimensional coordinates and size of the target center in the image. Then it is expressed as the velocity of the corresponding component; Preferably, step S2-4 specifically includes: S2-4-1 takes the posterior state of the current trajectory of the previous frame The prior state estimate in the current frame is predicted by Kalman filtering ; S2-4-2 uses the prior state estimation of the current frame and the position similarity of the high-score detection frame as the cost matrix, and uses Hungarian matching to solve it to obtain the matching result; If the matching result is that the trajectory and the detection box are successfully matched, the posterior state of the corresponding trajectory is updated through the Kalman filter of the adaptive observation noise; If the matching result is a trajectory mismatch, that is, the prior state estimation of the current frame is mismatched, then step S2-3 is executed to perform a two-stage association between the mismatched trajectory and the low-score detection frame; If the matching result is a detection mismatch, that is, the high-score detection frame mismatches, then step S3 is executed to send the mismatched high-score detection frame to the trajectory management module; Preferably, step S2-4-2 specifically includes: The prior state estimate of the current frame and high score detection frame According to formula (2), the coordinates of the upper left corner and the lower right corner of the bounding box are calculated respectively, and the result is defined as the bounding box and , their minimum enclosing rectangle is ,in( , )express The coordinates of the upper left vertex, ( , )express The coordinates of the lower right vertex; The IoU distance of the two bounding boxes is calculated according to formula (3), and then the relative distance of the corner points can be calculated according to formula (4): : The relative distance of the corner points By weight coefficient To reconcile the standard IoU distance, a new distance metric PMIoU is introduced, as shown in formula (5); weight ratio Is a hyperparameter that measures the IoU and degree of importance; Use 1-PMIOU as the first-stage cost matrix and obtain the matching results through the Hungarian algorithm; Preferably, the updating of the posterior state of the corresponding trajectory by the Kalman filter of the adaptive observation noise in step S2-4-2 is specifically as follows: Update the observation noise: in, represents the confidence of the kth high-score detection box, represents the i-th component of the unupdated observation noise, i=u,v,w,h, Represents the i-th component of the observation noise after confidence update; , A hyperparameter that controls the noise scale, > >1; Then continue to determine whether the current trajectory tracking state is a pending new trajectory or a surviving trajectory. If so, use the updated observation noise Update the posterior state of the current frame trajectory. The specific operation is shown in formula (7); otherwise, the current trajectory tracking state is considered to be a lost trajectory, and no update operation is performed; in, For the test results, is the observation noise, is the prior covariance, is the posterior covariance, is the Kalman gain; Preferably, step S2-5 specifically includes: The prior state estimation of the current frame and the position similarity of the high-score detection frame are used as the cost matrix, and Hungarian matching is used to solve it to obtain the matching result; specifically: First, the PMIOU between the low-score detection box and the mismatched trajectory prior state estimate is calculated as the position similarity, and a two-stage cost matrix is ​​constructed; and Redefined as the mismatch trajectory prior state estimation and low-score detection boxes, then calculate PMIOU according to formula (5), and finally obtain the matching result through the Hungarian algorithm; If the matching result is that the trajectory and the detection frame are successfully matched, it means that the tracking is successful. The corresponding detection is updated through the Kalman filter of the adaptive observation noise to update the posterior state of the corresponding trajectory. In the second stage, since the low-score detection frame represents a low-quality detection result, the detection result may be blurred, positionally offset, or have false detections. Consider using confidence to amplify the observation noise. The specific operation is as shown in formula (8); in, is the detection threshold for distinguishing high-score detection frames from low-score detection frames in step 1; Using the updated observation noise Update the posterior state of the current frame trajectory. The specific operation is as shown in formula (7); If the matching result is a trajectory mismatch, then execute step 3 to send the mismatched trajectory frame to the trajectory management module; If the matching result is a detection mismatch, execute step 3 to send the mismatched detection frame to the trajectory management module; Preferably, step 3 is as follows: The trajectory tracking state successfully matched in step S2-4 and step S2-5 is defined as a survival trajectory; The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the new trajectory to be generated; if the current frame detects a mismatch and the consecutive c frames can be successfully matched, the current frame trajectory tracking state is considered to be updated to the new trajectory, and the state variables of the Kalman filter are initialized; otherwise, the current frame mismatch detection is considered to be an error detection, and the corresponding trajectory is directly deleted; c>1; The trajectory tracking state corresponding to the mismatched trajectory in step S2-5 is defined as a lost trajectory; if there is at least one frame in the subsequent d frames that successfully associates with the low-score detection frame or the high-score detection frame, the detection result is used to update the posterior state of the Kalman filter of the mismatched trajectory, and the trajectory tracking state corresponding to the mismatched trajectory is updated to a surviving trajectory; if the matching fails for consecutive d frames, the trajectory tracking state corresponding to the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set; d>1; In step S2-5, the mismatch detection in the current frame is considered as an erroneous detection, and the corresponding track is directly deleted without participating in any operation of subsequent frames.

[0010] A second object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0011] A third object of the present invention is to provide a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0012] The beneficial effects of the present invention are: The present invention proposes a complete multi-target tracking solution. In order to solve the problems of identity exchange and trajectory fragmentation that are prone to occur in complex scenes, the detection set is decomposed, and a multi-stage association strategy is adopted based on the cascade idea. Taking into account the limitations of IoU as location information, a more robust PMIoU is proposed by reconciling the relative corner point distance as the penalty term of IoU. At the same time, considering the inconsistency of observation noise, a Kalman filtering method based on adaptive observation noise is proposed, which has stronger robustness to detection results of different qualities. Experiments are carried out on public data sets. The present invention can maintain the robustness of the multi-target tracking algorithm under detection results of different qualities, while effectively improving the tracking accuracy IDF1 and HOTA, and reducing ID switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the present invention.

[0014] Figure 2 Visualization results of multi-target tracking in pedestrian monitoring scenarios, including (a) the 28th frame, (b) the 30th frame, (c) the 45th frame, and (d) the 50th frame. DETAILED DESCRIPTION

[0015] The present invention is further analyzed below in conjunction with specific implementation methods; This embodiment provides a detection-adaptive based visual multi-target tracking method, which can be applied to multi-target tracking processes in scenarios such as military fields, security monitoring, traffic supervision, biological sciences, and sports video analysis.

[0016] See also Figure 1 The visual multi-target tracking method of this embodiment mainly includes the following steps: Step S1: Target detection Perform target detection on each frame of the video sequence, search for the target of interest, and construct a set of detection frames of the target of interest; Specifically, the target detection adopts the anchor-free frame detector YOLOX; the YOLOX target detector performs target detection on the input image sequence, completes the recognition of the target of interest in the scene, and obtains the position information and confidence score of the target.

[0017] The YOLOX algorithm consists of a backbone network (Backbone), a neck network (Neck) and a detection head network (Head). The original image size of the YOLOX network is 640×640×3. After processing with the Focus network structure and a CBS module, the output scale is 320×320×64 (the CBS module includes one convolution, batch normalization and SiLu activation function processing). After a 3×3 convolution downsampling, the output scale is 160×160×128. The step size of the first CBS component in each CSPLayer component is 2, so three times of 2-fold downsampling are performed. The dimensions of the three feature layers input to the neck network are 80×80×256, 40×40×512, and 20×20×1024, respectively. The extracted feature map is sent to the neck network, and the feature extraction is enhanced through the path aggregation network and the feature pyramid network. The detail information of the shallow feature map and the semantic information of the deep feature map are combined to realize multi-scale feature fusion. Finally, the detection results are output by three decoupling heads. P3 is responsible for detecting small targets, P4 is responsible for detecting medium targets, and P5 is responsible for detecting large targets.

[0018] At the same time, in data processing, the first half of the MOT Challenge video sequences are selected as the training set, and CrowdHuman, CityScapes, and VisDrone2019 are additionally mixed as supplementary training data, and the second half of the MOT Challenge video sequences are used as the test set.

[0019] Input the current frame image into the front YOLOX detector to obtain a set of interesting object detection frames. Each observation component in the interesting object detection frame set is defined as a five-tuple, expressed as ;in, represents the two-dimensional coordinates of the target center in the image, ( Indicates the scale of the target, Indicates the detection confidence of the target.

[0020] Step S2: Data association S2-1 builds high-score detection frames and low-score detection frames: Traverse the set of detection frames of the target of interest, and perform hierarchical processing on the detection frames of the target of interest one by one according to the detection confidence, to obtain high-score detection frames and low-score detection frames; wherein, the detection frame whose detection confidence is greater than the threshold a (0.6 in this embodiment) is defined as a high-score detection frame; the detection frame whose detection confidence is greater than b (0.1 in this embodiment) and less than a (0.6 in this embodiment) is defined as a low-score detection frame; the detection frame whose detection confidence is less than b (0.1 in this embodiment) is defined as a false positive and does not participate in matching; S2-2 initializes the first frame image of the video, which is recorded as the first frame image of the survival trajectory; S2-3 determines whether the current frame image is the first frame image of the current trajectory. If not, execute step S2-4. If yes, create a corresponding trajectory set for the high-score detection frame, initialize the state variables of the Kalman filter for the current trajectory, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory of the first frame, and repeat the current step S2-3; The state variable initialization is defined as: in, It is expressed as the two-dimensional coordinates and size of the target center in the image. Then it is expressed as the velocity of the corresponding component; S2-4 is a one-stage association to achieve the matching of high-score detection boxes and survival tracks; specifically: S2-4-1 takes the posterior state of the current trajectory of the previous frame The prior state estimate in the current frame is predicted by Kalman filtering : Given a state transition model (motion model) , observation model , process noise and observation noise , the following formula describes the process: in represents the prior error covariance of the current frame, represents the posterior error covariance of the previous frame, represents the state transition model of the current frame t, represents the process noise of the current frame t; S2-4-2 uses the prior state estimation of the current frame and the position similarity of the high-score detection frame as the cost matrix, uses Hungarian matching to solve, and obtains the matching result; specifically: In order to better complete the difference evaluation between the detection box and the trajectory, a PMIoU distance based on corner distance reconciliation is proposed. Specifically, the prior state estimate of the current frame and high score detection frame According to formula (2), the coordinates of the upper left corner and the lower right corner of the bounding box are calculated respectively, and the result is defined as the bounding box and , their minimum enclosing rectangle is ,in( , )express The coordinates of the upper left vertex, ( , )express The coordinates of the lower right vertex;

[0021] The IoU distance of the two bounding boxes is calculated according to formula (3), and then the relative distance of the corner points can be calculated according to formula (4): : The relative distance of the corner points By weight coefficient To reconcile the standard IoU distance, a new distance metric PMIoU is introduced, as shown in formula (5); weight ratio Is a hyperparameter that measures the IoU and degree of importance; Using 1-PMIOU as the first-stage cost matrix, the matching results are obtained through the Hungarian algorithm. The idea of ​​the Hungarian algorithm is to find as many matches as possible, that is, to make each detection target match a certain historical trajectory as much as possible. Its core approach is: starting from an initial match, expanding the match by finding an augmenting path until it can no longer be expanded.

[0022] There are three types of matching results for one-stage association: (1) If the matching result is that the trajectory and the detection frame are successfully matched, that is, the prior state estimation of the current frame trajectory and the high-score detection frame are successfully matched, it means that the previous frame and the current frame are tracked successfully. The corresponding detection is updated through the Kalman filter of adaptive observation noise to update the posterior state of the corresponding trajectory. In the first stage, since the high-score detection frame represents a high-quality detection result, the confidence is considered to reduce the observation noise. The specific operation is as shown in formula (6); in, represents the confidence of the kth high-score detection box, represents the i-th component of the unupdated observation noise, i=u,v,w,h, Represents the i-th component of the observation noise after confidence update; , represents the hyperparameter that controls the noise scale, =10, =50; Then continue to determine whether the current trajectory tracking state is a pending new trajectory or a surviving trajectory. If so, use the updated observation noise Update the posterior state of the current frame trajectory. The specific operation is shown in formula (7); otherwise, the current trajectory tracking state is considered to be a lost trajectory, and no update operation is performed; in, For the test results, is the observation noise, is the prior covariance, is the posterior covariance, is the Kalman gain; (2) If the matching result is a trajectory mismatch, that is, the prior state estimation of the current frame is mismatched, then step S2-4 is executed to perform a two-stage association between the mismatched trajectory and the low-score detection frame; (3) If the matching result is a detection mismatch, that is, the high-score detection frame mismatches, then execute step 3 and send the mismatched high-score detection frame to the trajectory management module; S2-4 Two-stage association, the low-score detection frame and the mismatched trajectory are matched in two stages; specifically: First, the PMIOU between the low-score detection box and the mismatched trajectory prior state estimate is calculated as the position similarity, and a two-stage cost matrix is ​​constructed; Bounding box and Redefined as the mismatch trajectory prior state estimation and low-score detection boxes, then calculate PMIOU according to formula (5), and finally obtain the matching result through the Hungarian algorithm; There are three types of matching results for the second-stage association: (1) If the matching result is that the trajectory and the detection frame are successfully matched, it means that the tracking is successful. The corresponding detection is updated through the Kalman filter of the adaptive observation noise to update the posterior state of the corresponding trajectory. In the second stage, since the low-score detection frame represents a low-quality detection result, the detection result may be blurred, positionally offset, or have false positives. Consider using confidence to amplify the observation noise. The specific operation is as shown in formula (8); in, is the detection threshold for distinguishing high-score detection frames from low-score detection frames in step 1, which is set to 0.6 in this embodiment; Using the updated observation noise Update the posterior state of the current frame trajectory. The specific operation is as shown in formula (7); (2) If the matching result is a trajectory mismatch, execute step 3 and send the mismatched trajectory frame to the trajectory management module; (3) If the matching result is a detection mismatch, execute step 3 and send the mismatched detection frame to the trajectory management module; Step 3: Track lifecycle management, which is used to handle mismatched tracks and detections in steps S2-4 and S2-5; the details are as follows: The trajectory tracking state successfully matched in step S2-4 and step S2-5 is defined as a survival trajectory; The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the new trajectory to be generated; if the current frame detects a mismatch and the consecutive c frames (the value c=3 in this embodiment) can be successfully matched, the current frame trajectory tracking state is determined to be updated to the new trajectory, and the state variables of the Kalman filter are initialized; otherwise, the current frame mismatch detection is determined to be an error detection, and the corresponding trajectory is directly deleted without initializing the trajectory tracking state; The trajectory tracking state corresponding to the mismatched trajectory in step S2-5 is defined as a lost trajectory; if there is at least one frame in the subsequent d frames (the value d=30 in this embodiment) that successfully associates with the low-score detection frame or the high-score detection frame, the detection result is used to update the posterior state of the Kalman filter of the mismatched trajectory, and the trajectory tracking state corresponding to the mismatched trajectory is updated to a surviving trajectory; if the matching fails for consecutive d frames, the trajectory tracking state corresponding to the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set; In step S2-5, the detection of mismatch in the current frame is considered as an error detection, and the corresponding track is directly deleted without participating in any operation of subsequent frames; Step 4: According to step 3, all trajectories whose tracking status is survival trajectory are obtained to complete the multi-target tracking of the entire video sequence.

[0023] Table 1 compares the performance indicators of the MOT17 validation set with the classic multi-target tracking algorithms DeepSORT, JDE, and FairMOT. HOTA, MOTA, and IDF1 represent tracking performance, IDs represents the number of ID switching, and FPS represents the inference speed. Through comparative experiments, it is found that the method of the present invention can effectively improve the tracking accuracy and robustness in different scenarios while ensuring online tracking.

[0024] Table 1 Performance comparison of various methods on the MOT17 validation set The visualization results of the multi-target tracking effect of the present invention in the pedestrian monitoring scene are as follows: Figure 2 Middle (a)- Figure 2 As shown in (d).

[0025] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a memory and a processor. The memory stores executable code. When the processor executes the executable code, the method described in any one of the implementation modes is implemented.

[0026] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (Non-volatile Memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0027] The bus may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.

[0028] Among them, the memory is used to store the program, and the processor executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor or implemented by the processor.

[0029] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0030] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0031] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0032] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for visual multi-target tracking based on detection adaptation, characterized in that The method comprises the following steps: Step S1: Target detection Perform target detection on each frame of the video sequence, search for the target of interest, and construct a set of detection frames of the target of interest; Step S2: Data association S2-1 traverses the set of detection frames of the target of interest, and classifies the detection frames of the target of interest one by one according to the detection confidence, to obtain high-score detection frames and low-score detection frames; S2-2 initializes the first frame image of the video, which is recorded as the first frame image of the survival trajectory; S2-3 determines whether the current frame image is the first frame image of the current trajectory. If not, execute step S2-4. If yes, create a corresponding trajectory set for the high-score detection frame, initialize the state variables of the Kalman filter for the current trajectory, then obtain the second frame image, and use the state variables of the first frame image as the posterior state of the current trajectory of the first frame, and repeat the current step S2-3; S2-4 one-stage association to achieve matching between high-score detection boxes and survival tracks; S2-5 two-stage association, the low-score detection frame and the mismatched trajectory are matched in two stages; Step S3, performing lifecycle management on the trajectory after data association; Step S4: Obtain all trajectories whose trajectory tracking status is a survival trajectory in step S3, and complete multi-target tracking of the entire video sequence.

2. The method according to claim 1, characterized in that: Step S2-1 is specifically: a detection frame whose detection confidence is greater than a threshold a is defined as a high-score detection frame, 0<a<1; a detection frame whose detection confidence is greater than b and less than a is defined as a low-score detection frame, 0<b<a<1.

3. The method according to claim 1, characterized in that: Step S2-4 is specifically: S2-4-1 takes the posterior state of the current trajectory of the previous frame The prior state estimate in the current frame is predicted by Kalman filtering ; S2-4-2 uses the prior state estimation of the current frame and the position similarity of the high-score detection frame as the first-stage cost matrix, and uses Hungarian matching to solve it to obtain the matching result; If the matching result is that the trajectory and the detection box are successfully matched, the posterior state of the corresponding trajectory is updated through the Kalman filter of the adaptive observation noise; If the matching result is a trajectory mismatch, that is, the prior state estimation of the current frame is mismatched, then step S2-3 is executed to perform a two-stage association between the mismatched trajectory and the low-score detection frame; If the matching result is a detection mismatch, that is, the high-score detection frame is mismatched, step S3 is executed to perform trajectory management on the mismatched high-score detection frame.

4. The method according to claim 3, characterized in that: Step S2-4-2 is specifically: The prior state estimate of the current frame and high score detection frame Calculate the coordinates of the upper left corner and lower right corner of the bounding box respectively. Represented as the two-dimensional coordinates and size of the target center in the image, recorded as the bounding box , , their minimum enclosing rectangle is ,in( , )express The coordinates of the upper left vertex, ( , )express The coordinates of the lower right vertex; Calculate the IoU distance of the two bounding boxes, and then calculate the relative distance of the corner points : According to the relative distance of the corner points And IoU distance calculation distance metric PMIoU: in represents the weight coefficient; Use 1-PMIOU as the first-stage cost matrix and obtain the matching results through the Hungarian algorithm.

5. The method according to claim 3, characterized in that: The specific method of updating the posterior state of the corresponding trajectory by adaptively observing the noise Kalman filter in step S2-4-2 is as follows: Update the observation noise: in, represents the confidence of the kth high-score detection box, represents the i-th component of the unupdated observation noise, i=u,v,w,h, Represents the i-th component of the observation noise after confidence update; , A hyperparameter that controls the noise scale, > >1; Then continue to determine whether the current trajectory tracking state is a pending new trajectory or a surviving trajectory. If so, use the updated observation noise Update the posterior state of the current frame trajectory. The specific operation is shown in formula (5). Otherwise, the current trajectory tracking state is considered to be a lost trajectory, and no update operation is performed. in, For the test results, is the observation noise, is the prior covariance, is the posterior covariance, is the Kalman gain.

6. The method according to claim 1, characterized in that: Step S2-5 is specifically: The prior state estimation of the current frame and the position similarity of the high-score detection frame are used as the cost matrix, and Hungarian matching is used to solve it to obtain the matching result; If the matching result is that the trajectory and the detection box are successfully matched, the posterior state of the corresponding trajectory is updated through the Kalman filter of the adaptive observation noise; the updated observation noise is used Update the posterior state of the current frame trajectory; If the matching result is a track mismatch, step S3 is executed to perform track management on the mismatched track frame; If the matching result is a detection mismatch, step S3 is executed to perform trajectory management on the mismatched detection frame.

7. The method according to claim 6, characterized in that: The Kalman filter of the adaptive observation noise in step S2-5 is used to update the posterior state of the corresponding trajectory: Update the observation noise: in, It is the detection threshold for distinguishing high-score detection boxes from low-score detection boxes in step 1.

8. The method according to claim 1, characterized in that: Step S3 is as follows: The trajectory tracking state successfully matched in step S2-4 and step S2-5 is defined as a survival trajectory; The trajectory tracking state corresponding to the mismatch detection in step S2-4 is defined as the new trajectory to be generated; if the current frame detects a mismatch and the consecutive c frames can be successfully matched, the current frame trajectory tracking state is considered to be updated to the new trajectory, and the state variables of the Kalman filter are initialized; otherwise, the current frame mismatch detection is considered to be an error detection, and the corresponding trajectory is directly deleted; c>1; The trajectory tracking state corresponding to the mismatched trajectory in step S2-5 is defined as a lost trajectory; If there is at least one frame in the subsequent d frames that is successfully associated with a low-score detection frame or a high-score detection frame, the detection result is used to update the posterior state of the Kalman filter of the mismatched trajectory, and the trajectory tracking state corresponding to the mismatched trajectory is updated to the survival trajectory; If the matching fails for d consecutive frames, the tracking state of the mismatched trajectory is updated to a dead trajectory, and the trajectory is permanently deleted from the trajectory set; d>1; The mismatch detection in step S2-5 determines that the mismatch detection in the current frame is an erroneous detection, and directly deletes the corresponding track.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

Images