Online multi-target tracking method, device and storage medium based on GIoU threshold mechanism

Abstract

This paper presents an online multi-target tracking method, device, and storage medium based on a GIoU threshold mechanism. It proposes a GIoU threshold mechanism (GIGM) incorporated into the cascade matching process to improve association performance and address the problem of long-distance identifier switching. When the target and detection distance are far apart, GIGM effectively prevents matching and long-distance identifier switching. GIoU matching is used as a secondary association strategy to achieve better and more reasonable tracking results. Even if the detected targets are highly similar in appearance, GIGM can prevent such mismatches when their locations are far apart.

Description

Technical Field

[0001] The present invention belongs to the technical field of video signal processing, and in particular relates to an online multi-target tracking method, device and storage medium based on a GIoU threshold mechanism (GIGM). Background Art

[0002] Mainstream real-time tracking methods usually use a threshold mechanism to filter out incorrect candidates and improve association accuracy. However, in some cases, the traditional distance threshold mechanism may be affected by long-distance identity switching. For example, in the DeepSORT algorithm, data association uses Mahalanobis threshold to generate candidate regions ( Figure 1 Mahalanobis threshold region), where only detections in this region can be compared with the current target ( Figure 1 However, when the target is about to leave the frame and is large in the frame, the Mahalanobis threshold, which combines object size and motion, becomes too large. Tracking may result in long-distance identity switching errors, such as when target #4 is mistakenly associated with detection #4 in the next frame.

[0003] Data association has always been a major challenge in multi-target tracking. To achieve better tracking results, researchers have developed various methods to facilitate data association. Popular online tracking algorithms such as DeepSORT use re-identification (RID) and Kalman filters to obtain the target's appearance features (cosine distance) and motion features (Mahalanobis threshold). However, the Mahalanobis threshold cannot address the long-distance identity switching problem in certain scenarios, resulting in mismatches. Summary of the Invention

[0004] The purpose of the present invention is to propose an online multi-target tracking method, device and storage medium based on the GIoU threshold mechanism, and propose a GIoU threshold mechanism (GIGM) added to the cascade matching process to improve association performance and solve the problem of long-distance identifier switching. When the target and detection distance are far, GIGM can effectively prevent matching and long-distance identifier switching. CIoU matching is used as a second association strategy to obtain better and more reasonable tracking results.

[0005] The online multi-target tracking method based on the GIoU threshold mechanism of the present invention comprises the following steps:

[0006] Step 1: Read image or video information through open-cv;

[0007] Step 2: Set the confidence threshold and use YOLOv4 to detect the target in each frame to obtain the target bounding box information;

[0008] Step 3: Pass the target bounding box information to the re-identification model to obtain the target appearance feature information;

[0009] Step 4: The target bounding box information and target appearance feature information are sent to the tracking system together;

[0010] Step 5. Start the GIoU cascade matching process. The tracking system obtains the target bounding box information and target appearance feature information of the existing track T[m] in the t-th frame image and the newly obtained detection target D[n] in the t+1-th frame image through step 4, where m is the number of tracks and n is the number of detection targets. The GIoU score G[n*m] between the track T[m] and the detection target D[n] is calculated based on the target bounding box information, and the appearance similarity cosine distance C[n*m] between the track T[m] and the detection target D[n] is calculated based on the target appearance feature, and cascade matching based on the GIoU threshold mechanism GIGM is implemented.

[0011] Step 6: After the GIoU cascade matching process is completed, the detection targets and trajectories that failed to match will be matched again with CIoU. The trajectories and detection targets that successfully matched with CIoU will enter step 7 together with the candidate pairs that successfully matched in the GIoU cascade matching in step 5 for Kalman filter update. The trajectories and detection targets that failed CIoU matching will enter step 8.

[0012] Step 7: The tracking system performs a Kalman filter update on the trajectory of the previous frame. Trajectories that match the detection target for three consecutive frames are considered confirmed trajectories, and other trajectories that do not match the detection target are considered failed trajectories.

[0013] Step 8: Perform Kalman filter prediction on all trajectories, update and correct the position information through the Kalman filter algorithm to obtain a new trajectory set, and send the confirmed trajectories to step 5 for the next GIoU cascade matching, and send the unconfirmed trajectories to step 6 for the next CIoU matching. After the matching is completed, proceed to step 10;

[0014] Step 9: For the tracks and detection targets that failed to match in step 7, different processing is performed according to different types:

[0015] Send the detected target as an unconfirmed trajectory into the trajectory set and proceed to step 8;

[0016] Determine whether the trajectory is in the confirmed state. If it is, determine whether its trajectory survival time a exceeds the threshold. Delete unconfirmed trajectories and trajectories with a trajectory survival time a greater than the threshold. Send the trajectories that have not been deleted to the trajectory collection and proceed to step 8.

[0017] Step 10: Visualize the trajectory of the new confirmation state.

[0018] The cascade matching based on the GIoU threshold mechanism GIGM in step 5 is specifically as follows:

[0019] Preset input parameters: maximum cosine distance of appearance similarity H1, GIoU threshold H2, maximum trajectory survival time A max , trajectory survival time a;

[0020] Initialize the matching set M1 and the unmatched detection set M2;

[0021] All trajectory survival time a is less than the maximum trajectory survival time A max The trajectories are selected in order of trajectory survival time a from small to large and put into the set T x ={j∈T|a j =x}, x=(0,…,A max -1), the set T x The trajectories in the set M2 are paired with the detection targets in the unmatched detection set M2 as candidate pairs. Let the appearance similarity cosine distance matrix between the candidate pairs be C and the GIoU score matrix be G;

[0022] Calculate the appearance similarity cosine distance C[i] between candidate pairs, i = (0, ..., n*m-1), and mark the candidate pairs whose appearance similarity cosine distance C[i] is greater than the threshold H1, so as to filter out the marked candidate pairs during Hungarian algorithm matching;

[0023] Calculate the GIoU score G[i] between candidate pairs, i = (0, ..., n * m-1). If the GIoU score G[i] is less than the threshold H2, set it to a maximum value and set the rest of G[i] to 0;

[0024] Add the processed GIoU score matrix G and the appearance similarity cosine distance matrix C to obtain the final cost matrix N;

[0025] By A max -1 Hungarian algorithm linear allocation to obtain the new matching set M1 and unmatched detection set M2 association results: M1, M2 = Hungarian algorithm (N, T i ,M1,M2), i=(0,…,A max -1), which means that the trajectory with the shortest survival time will be matched first. If the match is successful, the trajectory survival time a is set to zero, and the trajectory survival time a of the failed match is increased by 1.

[0026] The present invention provides an online multi-target tracker based on the GIoU threshold mechanism, comprising:

[0027] Acquisition device, reads image or video information through open-cv;

[0028] A processing device is connected to the acquisition device for receiving each frame of image output by the acquisition device, and the processing device includes a memory and a processor.

[0029] The memory is configured to store instructions to be executed by one or more processors of the system;

[0030] The processor executes the instructions to implement any of the above-mentioned online multi-target tracking methods based on the GIoU threshold mechanism.

[0031] The online multi-target tracking device of the IoU threshold mechanism of the present invention includes:

[0032] memory for storing instructions to be executed by one or more processors of the system, and

[0033] The processor is one of the processors of the system, and is used to execute the instructions to implement any of the above-mentioned online multi-target tracking methods based on the GIoU threshold mechanism.

[0034] The present invention provides a computer-readable storage medium, on which an online multi-target tracking program is stored. When the online multi-target tracking program is executed on a computer, the computer executes the steps of any one of the above-mentioned online multi-target tracking methods based on the GIoU threshold mechanism.

[0035] This paper proposes a GIoU threshold mechanism (GIGM), which provides a more accurate association threshold algorithm for the MOT data association process. It also proposes a two-stage data association algorithm, using CIoU matching as the second association strategy. This algorithm combines temporal, spatial, and appearance features to achieve better and more reasonable tracking results, solving the long-distance tag switching problem.

[0036] Unlike other methods that usually use complex strategies, the present invention uses GIoU as the threshold because the target (person) is unlikely to move too far within a few frames, which means that the GIoU between the target and the detection is unlikely to be too low. Compared to the IoU method, when the two bounding boxes do not overlap, GIoU will not be directly equal to 0, but will be expressed as a negative number, which can better reflect the position difference between the two bounding boxes. When the two bounding boxes intersect in different ways, the GIoU score will also be different. Therefore, even if the targets are highly similar in appearance, GIGM can prevent false matches when their positions are far apart. The results after adding the GIGM algorithm are as follows: Figure 2 As shown, the long-distance logo switching has been corrected, and when target 4 leaves the screen, detection 65 no longer mismatches him. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Illustration of region proposals generated using Mahalanobis thresholding for data association.

[0038] Figure 2 This is an illustration of the candidate regions generated by the present invention using GIoU as a threshold;

[0039] Figure 3 Schematic diagram of the system framework of the present invention;

[0040] Figure 4 This is a system flow chart of the GIGM algorithm of the present invention;

[0041] Figure 5 This is a cascade matching flow chart of the present invention;

[0042] Figure 6a Schematic diagram of IoU being calculated as the ratio of the intersection and connection between two bounding boxes;

[0043] Figure 6b The minimum outer rectangle of the target box and the detection box based on IoU is introduced for GIoU.

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0045] like Figures 3 to 5 FIG. 1 shows a first embodiment of the present invention, which relates to an online multi-target tracking method based on a GIoU threshold mechanism, specifically comprising the following steps:

[0046] Step 1: Read image or video information through open-cv;

[0047] Step 2: Set the confidence threshold and use YOLOv4 to detect the target in each frame to obtain the target bounding box information;

[0048] Step 3: Pass the target bounding box information to the ResNet18 re-identification model to obtain the target appearance features for subsequent appearance similarity cosine distance calculation;

[0049] Step 4: The target bounding box information and target appearance feature information are sent to the tracking system together;

[0050] Step 5: Start Figure 4In the GIoU cascade matching process shown in the figure, the tracking system obtains the target bounding box information and target appearance feature information of the existing track T[m] in the t-th frame image and the newly obtained detection target D[n] in the t+1-th frame image through step 4, where m is the number of tracks and n is the number of detection targets. The GIoU score G[n*m] between the track T[m] and the detection target D[n] is calculated by the target bounding box information, and the appearance similarity cosine distance C[n*m] between the track T[m] and the detection target D[n] is calculated by the target appearance feature, and the following is implemented: Figure 5 The cascade matching based on the GIoU threshold mechanism GIGM is shown:

[0051] Preset input parameters: maximum cosine distance of appearance similarity H1, GIoU threshold H2, maximum trajectory survival time A max , trajectory survival time a;

[0052] Initialize the matching set M1 and the unmatched detection set M2;

[0053] All trajectory survival time a is less than the maximum trajectory survival time A max The trajectories are selected in order of trajectory survival time a from small to large and put into the set T x ={j∈T|a j =x}, x=(0,…,A max -1), the set T x The trajectories in the set M2 are paired with the detection targets in the unmatched detection set M2 as candidate pairs. Let the appearance similarity cosine distance matrix between the candidate pairs be C and the GIoU score matrix be G;

[0054] Calculate the appearance similarity cosine distance C[i] between candidate pairs, i = (0, ..., n*m-1), and mark the candidate pairs whose appearance similarity cosine distance C[i] is greater than the threshold H1 (the marking is to add 1e-5 to the C[i] value) so that the marked candidate pairs can be filtered out during Hungarian algorithm matching;

[0055] Calculate the GIoU score G[i] between candidate pairs, i = (0, ..., n * m-1). If the GIoU score G[i] is less than the threshold H2, set it to a maximum value and set the rest of G[i] to 0;

[0056] Add the processed GIoU score matrix G and the cosine distance matrix C to get the final cost matrix N:

[0057] N=C+G

[0058] By A max-1 Hungarian algorithm linear allocation to obtain the new matching set M1 and unmatched detection set M2 association results: M1, M2 = Hungarian algorithm (N, T i ,M1,M2), i=(0,…,A max -1), which means that the trajectory with the shortest survival time will be matched first. If the match is successful, the trajectory survival time a is set to zero, and the trajectory survival time a of the failed match is increased by 1;

[0059] Step 6: After the GIoU cascade matching process is completed, the detection targets and trajectories that failed to match will be re-matched with CIoU to improve the matching results. The trajectories and detection targets that successfully matched with CIoU will enter step 7 together with the candidate pairs that successfully matched in the GIoU cascade matching in step 5 for Kalman filter update; the trajectories and detection targets that failed CIoU matching will enter step 8;

[0060] Step 7: The tracking system performs a Kalman filter update on the trajectory of the previous frame. Trajectories that match the detection target for three consecutive frames are considered confirmed trajectories, and other trajectories that do not match the detection target are considered failed trajectories.

[0061] Step 8: Perform Kalman filter prediction on all trajectories, update and correct the position information through the Kalman filter algorithm to obtain a new trajectory set, and send the confirmed trajectories to step 5 for the next GIoU cascade matching, and send the unconfirmed trajectories to step 6 for the next CIoU matching. After the matching is completed, proceed to step 10;

[0062] Step 9: For the trajectories and detection targets that failed to match in step 7, different processing is performed according to their types: the detection targets are sent to the trajectory set as unconfirmed trajectories and go to step 8;

[0063] Determine whether the trajectory is in the confirmed state. If it is, determine whether its trajectory survival time a exceeds the threshold. Delete unconfirmed trajectories and trajectories with a trajectory survival time a greater than the threshold. Send the trajectories that have not been deleted to the trajectory collection and proceed to step 8.

[0064] Step 10: Visualize the newly confirmed trajectory, such as drawing a bounding box, trajectory route, etc., and save the processed video file. Finally, save the data required for the benchmark test for future use.

[0065] Popular online tracking algorithms like DeepSORT use re-identification (RID) and a Kalman filter to obtain the target's appearance features (cosine distance) and motion features (Mahalanobis threshold). Because Mahalanobis thresholds cannot address the long-distance identity switching problem in certain scenarios, this paper proposes using GIGM in a cascaded matching process.

[0066] like Figure 6a As shown, IoU is calculated as the ratio of the intersection and connection between two bounding boxes, as Figure 6b As shown in Figure 1, GIoU introduces the minimum outer rectangle of the target box and the detection box based on IoU, which is calculated as follows:

[0067]

[0068]

[0069] The present invention uses GIoU as the threshold because the target (person) cannot move too far within a few frames, and there will be a certain degree of overlap, which means that the GIoU score between the trajectory and the detected target should not be too low under normal circumstances. Compared with the IoU method, which can only be a minimum of 0, when the two bounding boxes are separated, the GIoU score will get a negative number, which can reflect a more accurate difference between the positions of the two bounding boxes. When the two bounding boxes intersect in different ways, the GIoU scores will also be different. Therefore, even if the detected targets are highly similar in appearance, GIGM can prevent this false match when their positions are far apart. The results after the GIGM algorithm are as follows Figure 2 As shown, the long-distance logo switching has been corrected. After target 4 leaves the frame, its trajectory no longer successfully matches target 65.

[0070] After GIoU cascade matching, some tracks and detection targets may not be successfully matched due to occlusion and other reasons. Therefore, the present invention uses CIoU matching after GIoU cascade matching to improve data association. If the CIoU score of two bounding boxes is greater than a threshold, the track and detection target are successfully matched.

[0071] Although this algorithm only matches bounding boxes, most detections are successfully matched in the GIoU cascade matching, and only a few frames require CIoU matching, which can reduce missed detections to a certain extent. CIoU also considers the position, area, and aspect ratio of the bounding box, making the association more comprehensive.

[0072] A second embodiment of the present invention relates to an online multi-target tracking device based on a GIoU threshold mechanism, comprising:

[0073] memory for storing instructions to be executed by one or more processors of the system, and

[0074] The processor is one of the processors of the system, and is configured to execute the instructions to implement any possible method of the first embodiment described above.

[0075] The first embodiment is a method implementation corresponding to this embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0076] A third embodiment of the present invention relates to a computer-readable storage medium encoded with a computer program, on which an online multi-target tracking program is stored. When the online multi-target tracking program is executed on a computer, the computer executes the steps of any possible method of the above-mentioned first embodiment.

[0077] The first embodiment is a method implementation corresponding to this embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0078] A fourth embodiment of the present invention relates to an online multi-target tracker based on a GIoU threshold mechanism, the online multi-target tracker comprising:

[0079] Acquisition device, reads image or video information through open-cv;

[0080] A processing device is connected to the acquisition device for receiving each frame of image output by the acquisition device, and the processing device includes a memory and a processor.

[0081] The memory is configured to store instructions to be executed by one or more processors of the system;

[0082] The processor executes the instructions to implement any possible method of the first embodiment.

[0083] The first embodiment is a method implementation corresponding to this embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0084] It should be noted that each embodiment of the present invention can be implemented in software, hardware, firmware, etc. Regardless of whether the present invention is implemented in software, hardware, or firmware, the instruction code can be stored in any type of computer-accessible memory (e.g., permanent or modifiable, volatile or non-volatile, solid or non-solid, fixed or removable media, etc.). Similarly, the memory can be programmable array logic (PAL), random access memory (RAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EPROM), magnetic disk, optical disk, digital versatile disk (DVD), etc.

[0085] It should be noted that the various units / modules mentioned in the various embodiments of the present invention are logical units / modules. Physically, a logical unit can be a physical unit, a portion of a physical unit, or a combination of multiple physical units. The physical implementation of these logical units themselves is not the most important. The combination of functions implemented by these logical units is the key to solving the technical problems proposed by the present invention. In addition, in order to highlight the innovative aspects of the present invention, the above-mentioned embodiments of the present invention do not introduce units that are not closely related to solving the technical problems proposed by the present invention. This does not mean that other units do not exist in the above-mentioned embodiments.

[0086] It should be noted that, in the claims and description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a" does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0087] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

1. Online multi-target tracking method based on GIoU threshold mechanism, characterized by The steps include: Step 1: Read image or video information through open-cv; Step 2: Set the confidence threshold and use YOLOv4 to detect the target in each frame to obtain the target bounding box information; Step 3: Pass the target bounding box information to the re-identification model to obtain the target appearance feature information; Step 4: The target bounding box information and target appearance feature information are sent to the tracking system together; Step 5. Start the GIoU cascade matching process. The tracking system obtains the target bounding box information and target appearance feature information of the existing track T[m] in the t-th frame image and the newly obtained detection target D[n] in the t+1-th frame image through step 4, where m is the number of tracks and n is the number of detection targets. The GIoU score G[n*m] between the track T[m] and the detection target D[n] is calculated based on the target bounding box information, and the appearance similarity cosine distance C[n*m] between the track T[m] and the detection target D[n] is calculated based on the target appearance feature, and cascade matching based on the GIoU threshold mechanism GIGM is implemented. Step 6: After the GIoU cascade matching process is completed, the detection targets and trajectories that failed to match will be matched again with CIoU. The trajectories and detection targets that successfully matched with CIoU will enter step 7 together with the candidate pairs that successfully matched in the GIoU cascade matching in step 5 for Kalman filter update. The trajectories and detection targets that failed CIoU matching will enter step 8. Step 7: The tracking system performs a Kalman filter update on the trajectory of the previous frame. Trajectories that match the detection target for three consecutive frames are considered confirmed trajectories, and other trajectories that do not match the detection target are considered failed trajectories. Step 8: Perform Kalman filter prediction on all trajectories, update and correct the position information through the Kalman filter algorithm to obtain a new trajectory set, and send the confirmed trajectories to step 5 for the next GIoU cascade matching, and send the unconfirmed trajectories to step 6 for the next CIoU matching. After the matching is completed, proceed to step 10; Step 9: For the tracks and detection targets that failed to match in step 7, different processing is performed according to different types: Send the detected target as an unconfirmed trajectory into the trajectory set and proceed to step 8; Determine whether the trajectory is in the confirmed state. If it is, determine whether the trajectory survival time a exceeds the threshold. Delete unconfirmed trajectories and trajectories with a trajectory survival time a greater than the threshold. Add the trajectories that have not been deleted to the trajectory collection and proceed to step 8. Step 10: Visualize the trajectory of the new confirmation state.

2. The online multi-target tracking method based on the GIoU threshold mechanism according to claim 1 is characterized in that The cascade matching based on the GIoU threshold mechanism GIGM in step 5 is specifically as follows: Preset input parameters: maximum cosine distance of appearance similarity H1, GIoU threshold H2, maximum trajectory survival time A max , trajectory survival time a; Initialize the matching set M1 and the unmatched detection set M2; All trajectory survival time a is less than the maximum trajectory survival time A max The trajectories are selected in order of trajectory survival time a from small to large and put into the set T x ={j∈T|a j =x}, x=(0,…,A max -1), the set T x The trajectories in the set M2 are paired with the detection targets in the unmatched detection set M2 as candidate pairs. Let the appearance similarity cosine distance matrix between the candidate pairs be C and the GIoU score matrix be G; Calculate the appearance similarity cosine distance C[i] between candidate pairs, i = (0, ..., n*m-1), and mark the candidate pairs whose appearance similarity cosine distance C[i] is greater than the threshold H1, so as to filter out the marked candidate pairs during Hungarian algorithm matching; Calculate the GIoU score G[i] between candidate pairs, i = (0, ..., n * m-1). If the GIoU score G[i] is less than the threshold H2, set it to a maximum value and set the rest of G[i] to 0; Add the processed GIoU score matrix G and the appearance similarity cosine distance matrix C to obtain the final cost matrix N; By A max -1 Hungarian algorithm linear allocation to obtain the new matching set M1 and unmatched detection set M2 association results: M1, M2 = Hungarian algorithm (N, T i ,M1,M2), i=(0,…,A max -1), which means that the trajectory with the shortest survival time will be matched first. If the match is successful, the trajectory survival time a is set to zero, and the trajectory survival time a of the failed match is increased by 1.

3. Online multi-target tracker based on GIoU threshold mechanism, characterized by include: Acquisition device, reads image or video information through open-cv; A processing device is connected to the acquisition device for receiving each frame of image output by the acquisition device, and the processing device includes a memory and a processor. The memory is used to store instructions executed by one or more processors of the system; the processor executes the instructions to implement any one of the online multi-target tracking methods based on the GIoU threshold mechanism in claims 1-2.

4. Online multi-target tracking device based on GIoU threshold mechanism, characterized by include: memory for storing instructions to be executed by one or more processors of the system, and The processor is one of the processors of the system, and is used to execute the instructions to implement any one of the online multi-target tracking methods based on the GIoU threshold mechanism in claims 1-2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an online multi-target tracking program. When the online multi-target tracking program is executed on a computer, the computer executes the steps of any one of the online multi-target tracking methods based on the GIoU threshold mechanism in claims 1-2.

Images