Method and device for multi-target tracking and storage medium

Abstract

The invention relates to a multi-target tracking method and device and a storage medium. According to one embodiment, the method includes: determining a plurality of target whole body detection frames in a current input image by performing target whole body detection, and determining a plurality of target head detection frames in the current input image by performing target head detection; determining whole body identifiers of the plurality of target whole body detection frames by executing whole body trajectory association; determining head identifiers of the plurality of target head detection frames by executing head trajectory association; determining a plurality of target whole body prediction frames corresponding to the plurality of target head detection frames based on the positions and sizes of the plurality of target head detection frames; determining a target whole body association frame of the plurality of target head detection frames in the plurality of target whole body detection frames based on areas occupied by the plurality of target whole body prediction frames of the plurality of target head detection frames in the input image; and updating the target whole body trajectory set based on the whole body identifier of the target whole body association frame.

Description

Technical Field

[0001] This disclosure relates generally to image processing, and more specifically to methods for multi-object tracking, apparatus for multi-object tracking, and computer-readable non-transitory storage media storing programs. Background Technology

[0002] With the development of computer science and artificial intelligence, using computers to run neural network-based artificial intelligence models for information processing is becoming increasingly common and effective. Computer vision is an important application area of ​​artificial intelligence models.

[0003] A hot topic in computer vision technology is multi-target tracking. Multi-target tracking, often abbreviated as MTT (Multiple Target Tracking; sometimes also abbreviated as MOT: Multiple Object Tracking), is used to detect and assign IDs to targets of interest, such as pedestrians, cars, and / or animals, in videos. The desired tracking result is that, for a captured video segment, after multi-target tracking, the same target is assigned a unique ID in different frames, and different targets in the same frame are assigned different IDs. In a video segment with identified targets, each target in each frame has a position parameter P and a time parameter tm. Therefore, multi-target tracking can determine the sequence of position-time parameter pairs (P, tm) for target Tg[n]. This sequence can indicate the trajectory Tr[n] of target Tg[n]. That is, multi-target tracking can track individual targets across frames and determine the trajectory of each target.

[0004] Taking the tracked object as an example, multi-target tracking of various targets across frames usually includes three steps: (1) Detecting the target (e.g., a pedestrian) in frame t and determining the detection position of the target (e.g., determining the detection box Bx of the target in this frame); (2) Using the stored previous trajectory segments (from frame 1 to frame t-1) to predict the position of the target in frame t; (3) By comparing the predicted position with the detection position in frame t (i.e., the position of the detection box), associating the detection position of the target in frame t with the previously stored trajectory segments (e.g., if the position of Bx is close to the target prediction position of the previous trajectory segment Tr[n] of the target Tg[n] with IDn, then assigning the ID attribute of Bx to "IDn" to realize the association between Bx and the previous trajectory segment Tr[n]. Correspondingly, the trajectory segment Tr[n] adds a trajectory point corresponding to Bx, that is, the trajectory segment Tr[n] is updated), thereby completing the update of the tracking trajectory.

[0005] For example, Patent Document 1 (CN116958873A) discloses a pedestrian tracking method, wherein a target detection model outputs at least one head detection box and at least one human body detection box. Under a predetermined crowd density, for a head detection box that does not match a human body detection box, an estimated human body detection box is estimated based on the head detection box, and a human multi-target tracker outputs the target identifier of the estimated human body detection box.

[0006] In current multi-target tracking techniques, ID switching is very common, especially in crowded scenes. ID switching corresponds to the phenomenon where the trajectory segment Tr[n] of target Tg[n] actually contains the trajectory points of another target Tg[n']. That is, the tracking trajectory contains incorrect trajectory points. To improve the overall performance of the tracking method, it is desirable to correct ID switching. Summary of the Invention

[0007] A brief overview of this disclosure is provided below to offer a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0008] The inventors have researched and experimented with existing multi-target tracking methods and proposed the solution disclosed herein, with the aim of reducing ID switching and improving the accuracy of multi-target tracking.

[0009] According to one aspect of this disclosure, a method for multi-target tracking is provided. The method includes: determining multiple whole-body bounding boxes of targets in a current input image by performing whole-body target detection; and determining multiple head bounding boxes of targets in the current input image by performing head target detection; determining target identifiers of the multiple whole-body bounding boxes of targets by performing whole-body trajectory association to update a set of whole-body target trajectories; determining target identifiers of the multiple head bounding boxes of targets by performing head trajectory association to update a set of head bounding boxes of targets; determining multiple whole-body predicted bounding boxes of targets corresponding to the multiple head bounding boxes of targets based on the position and size of the multiple head bounding boxes of targets; determining target whole-body associated bounding boxes of the multiple head bounding boxes of targets within the multiple whole-body bounding boxes of targets based on the regions occupied by the multiple whole-body predicted bounding boxes of targets within the multiple head bounding boxes of targets in the input image; and updating the set of whole-body target trajectories based on the target identifiers of the target whole-body associated bounding boxes of targets.

[0010] According to one aspect of this disclosure, an apparatus for converting data is provided. The apparatus includes: a memory storing instructions thereon; and at least one processor configured to execute the instructions to implement the aforementioned method for multi-target tracking.

[0011] According to another aspect of this disclosure, a computer-readable non-transitory storage medium storing a program is provided. When the program is executed by a computer, the program causes the computer to implement the aforementioned method for multi-target tracking.

[0012] The beneficial effects of the methods, apparatus, and storage media disclosed herein include at least one of the following: reduced ID switching and improved accuracy of multi-target tracking. Attached Figure Description

[0013] The embodiments of this disclosure are described below with reference to the accompanying drawings, which will help to more easily understand the above and other objects, features, and advantages of this disclosure. The drawings are only for illustrating the principles of this disclosure. The dimensions and relative positions of the elements are not necessarily drawn to scale in the drawings. The same reference numerals may denote the same features. In the drawings:

[0014] Figure 1 An exemplary flowchart of a method for multi-target tracking according to an embodiment of the present disclosure is shown;

[0015] Figure 2a An exemplary schematic diagram of an input image of frame (t-1) according to an embodiment of the present disclosure is shown;

[0016] Figure 2b An exemplary schematic diagram of a t-frame input image according to an embodiment of the present disclosure is shown;

[0017] Figure 2c It shows the corresponding Figure 2a and Figure 2b The true trajectory of the target;

[0018] Figure 3a It shows Figure 2b The target full-body detection bounding box is shown in the input image.

[0019] Figure 3b It shows Figure 2b The target head detection bounding box is shown in the input image.

[0020] Figure 4a It shows Figure 3a The full-body identifier assigned to the target's full-body detection bounding box;

[0021] Figure 4b It shows the relationship with Figure 4aThe full-body marker shown corresponds to the target's full-body trajectory;

[0022] Figure 5a It shows Figure 3b The header identifier assigned to the target header detection bounding box shown;

[0023] Figure 5b It shows the relationship with Figure 5a The target head trajectory corresponding to the head icon shown;

[0024] Figure 6a It shows Figure 2b The target head detection box shown corresponds to the target whole-body prediction box;

[0025] Figure 6b It shows the relationship with Figure 2b The target head detection bounding box is shown as the target whole-body associated bounding box;

[0026] Figure 7a It shows the Figure 4a The full-body identifier of the detection box corresponding to the current trajectory point in the image is corrected;

[0027] Figure 7b It shows the basis Figure 7a The target's whole-body trajectory is obtained from the corrected whole-body identifier;

[0028] Figure 8 An exemplary block diagram of an apparatus for multiple targets according to one embodiment of the present disclosure is shown;

[0029] Figure 9 An exemplary block diagram of an apparatus for multiple targets according to an embodiment of the present disclosure is shown; and

[0030] Figure 10 This is an exemplary block diagram of an information processing device according to one embodiment of the present disclosure. Detailed Implementation

[0031] Exemplary embodiments of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0032] It should also be noted that, in order to avoid obscuring the contents of this disclosure with unnecessary details, only the device structure closely related to the solution according to this disclosure is shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0033] It should be understood that this disclosure is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0034] Computer program code used to perform the operations of various aspects of embodiments of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0035] The method disclosed herein can be implemented using circuitry with corresponding functional configurations. The circuitry includes circuitry for a processor.

[0036] One aspect of this disclosure relates to a method for multi-target tracking. This method can be implemented using a computer. The inventors studied multi-target tracking models and found that in the input frame, part or all of the target's torso is more likely to be occluded relative to the target's head, which may lead to ID switching and decreased tracking accuracy. Therefore, through experimentation, the inventors proposed a joint head-and-whole-body multi-target tracking method to mitigate the aforementioned problems.

[0037] The following is for reference. Figure 1 The method is described exemplarily.

[0038] Figure 1 An exemplary flowchart of a method 100 for multi-target tracking according to an embodiment of the present disclosure is shown. In one example, method 100 is implemented by a computer running a corresponding computer program.

[0039] In operation Op101, multiple target full-body detection boxes (hereinafter, B[I] represents a target full-body detection box, and {B[I]} represents a set of target full-body detection boxes) are determined in the current input image Im[t] by performing target full-body detection, and multiple target head detection boxes (hereinafter, b[i] represents a target head detection box, and {b[i]} represents a set of target head detection boxes) are determined in the current input image by performing target head detection. t is the index of the input image, for example, t is the frame number of the input image. The target is a human. The target head detection box is used to indicate the region where the head of the target is located in the image. The target full-body detection box is used to indicate the region where the whole body of the target (including the head and torso) is located in the image. Based on the output of operation Op101, the height, width, and position (e.g., the coordinates of the top left corner and / or the bottom right corner) of each detection box can be determined.

[0040] The input images can come from cameras monitoring locations of interest. The camera's installation height is preferably greater than or equal to 1.5 meters. Examples of installation heights include 1.5 meters, 1.6 meters, 1.7 meters, 1.8 meters, 1.9 meters, 2.0 meters, or 2.4 meters. When the camera is installed indoors, it can be mounted near the ceiling.

[0041] In one example, the same model dM is used to implement both whole-body target detection and head target detection. This can be represented as:

[0042] ({B[I]},{b[i]})=dM(Im[t]).

[0043] For example, inputting image Im[t] into model dM outputs multiple full-body detection boxes and multiple head detection boxes. Optionally, it can also output related parameters (such as width, height, and confidence). The number of detected full-body detection boxes and the number of detected head detection boxes may be equal or unequal (including cases where they are greater or less). Model dM can be a neural network-based model. The model obtained after training the model with samples can be used to implement operation Op101. For example, there are two targets Tg[1] and Tg[2] in the camera's field of view; Figure 2a As shown, targets Tg[1] and Tg[2] are located on the left side of the field of view in frame (t-1); Figure 2b As shown, targets Tg[1] and Tg[2] are located on the right side of the field of view at frame t (i.e., moving to the right side); Figure 2c The actual trajectories tTr[1] and tTr[2] formed by this movement are shown, where, for clarity, trajectory points at earlier moments are omitted; after the image Im[t] is input into the model dM, the model dM will output 4 detection boxes: Figure 3aThe target full-body detection boxes B[1] and B[2] are shown (this stage belongs to the detection stage, and the full-body identifier wID has not yet been assigned to each detection box). Figure 3b The target head detection boxes b[1] and b[2] are shown (this stage belongs to the detection stage, and the head identifier hID has not yet been assigned to each detection box).

[0044] In operation Op103, the whole-body identifiers of multiple target whole-body detection boxes are determined by performing whole-body trajectory association to update the target whole-body trajectory set {wTr[j]}. For example, a whole-body identifier (B[I]).wID = widX; that is, the target identifier that distinguishes the trajectories of different targets is assigned to each (B[I]) of multiple target whole-body detection boxes based on the previous whole-body trajectory set. The assigned whole-body identifier comes from the whole-body identifier set corresponding to the previous whole-body trajectory set or a new whole-body identifier. That is, if a new target appears, the whole-body target identifier set will also be updated. For example, if the position of B[I] is close to the target predicted position of the last trajectory point of the previous whole-body trajectory segment wTr[n] of the target Tg[n] with ID "widn", then the ID attribute of B[I] is assigned the value "widn" to achieve the association between B[I] and the previous whole-body trajectory segment wTr[n]. Figure 4a An example is shown for Figure 3a The full-body identifiers “wid01” and “wid02” associated with the target full-body detection boxes B[1] and B[2] are shown, and the assigned full-body identifier of the previous target full-body detection box Bp at frame t-1 is also shown. Figure 4b It shows the relationship with Figure 4a The full-body identifiers shown correspond to the target full-body trajectories wTr[1] and wTr[2]. (The rest of the text appears to be incomplete and requires further context.) Figure 2c By comparing the actual trajectories shown, we can know that Figure 4a The associated full-body identifiers for the two target full-body detection boxes in frame t shown are incorrect; an identifier switch has occurred, which leads to... Figure 4b The trajectory shown is also incorrect (where earlier trajectory points are omitted for clarity). See the following description, where method 100 is able to correct this error based on a joint head and whole-body approach.

[0045] In operation Op105, head identifiers for multiple target head detection boxes are determined by performing head trajectory association to update the target head trajectory set {hTw[k]}. For example, a head identifier (b[i].hID = hidX) is assigned to each (b[i]) in the multiple target head detection boxes based on the previous head trajectory set. The assigned head identifier comes from the head identifier set corresponding to the previous head trajectory set or a new head identifier. That is, if a new target appears, the head target identifier set will also be updated. The intersection of the head target identifier set and the whole-body target identifier set is an empty set. Figure 5a An example is shown for Figure 3b The target head detection boxes b[1] and b[2] are associated with the head identifiers “hid01” and “hid02”. The assigned head identifiers of the previous target head detection box bp at frame t-1 are also shown. It can be seen that the same head identifier corresponds to the same target, that is, the trajectory generated based on the target head detection box is correct. Figure 5b It shows the relationship with Figure 5a The head identifiers shown correspond to the target head trajectories hTr[1] and hTr[2], where Pc represents the current trajectory point corresponding to frame t, and Pp represents the previous trajectory point corresponding to frame (t-1). Figure 2c By comparing the actual trajectories shown, we can know that Figure 5b The trajectory shown is correct. The inventors analyzed the captured images used for multi-target tracking and concluded that: occlusion easily leads to label switching; in the captured images, the probability of the head being severely occluded is lower than that of the torso. Therefore, the correct trajectory generated by target tracking based on the head detection box is more likely, and the tracking results based on the whole-body detection box can be corrected based on the tracking results of the head detection box.

[0046] In operation Op107, multiple target whole-body prediction boxes are determined based on the position and size of multiple target head detection boxes. The target whole-body prediction box corresponding to the target head detection box b[i] can be represented by b[i].B' or B'[i]. Figure 6a It shows Figure 2b The target head detection boxes b[1] and b[2] shown are the corresponding target whole body prediction boxes B'[1] and B'[2].

[0047] In operation Op109, the target-whole-body association box of the multiple target head detection boxes is determined based on the region occupied by the multiple target whole-body prediction boxes in the input image Im[t]. The target-whole-body association box of the target head detection box b[i] can be represented by b[i].B” or B”[i]. The target-whole-body association box B”[i] of the target head detection box b[i] is determined based on the region occupied by the target-whole-body prediction box B'[i] of the target head detection box b[i] in the input image Im[t]. That is, a target whole-body detection box is selected from the target whole-body detection box set {B[i]} as the target-whole-body association box B”[i] of the target head detection box b[i]. Figure 6b It shows Figure 2bThe target head detection boxes b[1] and b[2] shown are target whole-body association boxes B”[1] and B”[2], where B”[1] = B[1]; B”[2] = B[2], that is, the whole-body identifier of detection box b[1] determined by its target whole-body association box is “wid01”, and the whole-body identifier of detection box b[2] determined by its target whole-body association box is “wid02”. Figure 6b The target's full-body bounding box "Bp" and its full-body identifier are also shown in frame t-1. (Comparison) Figure 5a and Figure 6b It can be seen that the whole-body identifier ("wid01") of the associated box (B"[1] = B[1]) of the current trajectory point Pc (corresponding to detection box b[1]) with the head identifier "hid01" is different from the whole-body identifier ("wid02") of the associated box (Bp[2]) of the previous trajectory point Pp (corresponding to detection box bp[2]). That is, the whole-body identifiers of the two corresponding whole-body associated boxes (e.g., B[1] and Bp[2]) of the two recent trajectory points of the target trajectory of the same target determined by the head detection box are inconsistent. This indicates that the whole-body identifier ("wid01") of the recent corresponding whole-body associated box (e.g., B[1]) in the two corresponding whole-body associated boxes is likely incorrect and needs to be corrected.

[0048] In operation Op111, the target whole-body trajectory set {wTr[j]} is updated based on the whole-body identifiers of the target whole-body association boxes of multiple target head detection boxes. That is, the previously obtained target whole-body trajectory set is corrected based on the whole-body identifiers of the target whole-body association boxes.

[0049] In one embodiment, updating the target full-body trajectory set based on the full-body identifiers of the target full-body associated bounding boxes of multiple target head detection boxes includes: for each trajectory in the target head trajectory set {hTr[k]}, if the full-body identifier Pc.B”.wID of the target full-body associated bounding box of the current trajectory point Pc of the trajectory is different from the full-body identifier Pp.B”.wID of the target full-body associated bounding box of the previous trajectory point Pp of the trajectory, then the full-body identifier of the target full-body associated bounding box of the current trajectory point is replaced with the full-body identifier of the target full-body associated bounding box of the previous trajectory point. If the two full-body identifiers are the same, no replacement is performed. It can be understood that if the current trajectory point is the first trajectory point of the target head trajectory, then the aforementioned judgment operation is not performed on the target head trajectory, and correspondingly, the aforementioned replacement operation is not performed. Figure 7a It shows the Figure 4a The full-body identifiers of the detection boxes B[1] and B[2] corresponding to the current trajectory point in the current trajectory point are corrected and then used to correct the full-body identifiers. Figure 7b It shows the basis Figure 7a The corrected whole-body identifiers in the model yielded the target's whole-body trajectory wTr[1] and wTr[2]. Figure 5bThe relevant information of the current trajectory point Pc and the previous trajectory point Pp of the target head trajectory hTr[1] is shown in Table 1.

[0050] Table 1. Relevant information on the target head trajectory hTr[1].

[0051]

[0052] right Figure 5b The relevant information of the current trajectory point Pc and the previous trajectory point Pp of the target head trajectory hTr[2] is shown in Table 2.

[0053] Table 2. Relevant information on the target head trajectory hTr[2].

[0054]

[0055] Figure 7b The trajectories wTr[1] and wTr[2] shown are... Figure 2c The actual trajectories tTr[1] and tTr[2] shown are consistent. Therefore, it is beneficial to use the joint head and body to correct the whole body trajectory of the target based on the association box. This is beneficial to reduce the switching of labels and improve the accuracy of multi-target tracking.

[0056] In one embodiment, a single object detection model (also known as a "head-body joint detection model") is used to determine multiple whole-body bounding boxes and multiple head bounding boxes in the current input image. The head-body joint detection model can be a conventional YOLOX-based detection model.

[0057] In one embodiment, method 100 can be configured to perform offline multi-object tracking or online multi-object tracking.

[0058] In one embodiment, determining the full-body identifiers of multiple target full-body detection boxes {B[i]} (i.e., associating each target full-body detection box with the corresponding target full-body trajectory as the latest trajectory point of the corresponding target full-body trajectory) and determining the head identifiers of multiple target head detection boxes {b[i]} (i.e., associating each target head detection box with the corresponding target head trajectory as the latest trajectory point of the corresponding target head trajectory) is based on a target tracking algorithm such as Bytetrack. Further, the full-body identifiers of multiple target full-body detection boxes in the current input image are determined using a first Kalman filter; and the head identifiers of multiple target head detection boxes in the current input image are determined using a second Kalman filter, which is different from the first Kalman filter.

[0059] In one embodiment, determining the target whole-body prediction box corresponding to one of the target head detection boxes among multiple target head detection boxes includes: determining the abscissa component of the target whole-body prediction box by linearly combining the abscissa component of the target head detection box's position and its width; determining the ordinate component of the target whole-body prediction box's position by linearly combining the ordinate component of the target head detection box's position and its height; determining the width of the target whole-body prediction box by increasing the width of the target head detection box; and determining the height of the target whole-body prediction box by increasing the height of the detection box. Exemplary calculation formulas for the position, height, and width of the target whole-body prediction box are shown in equations (1), (2), (3), and (4).

[0060] x body =xh ead +C1*w head (1)

[0061] y body =yh ead +C2*h head (2)

[0062] w body =C3*w head (3)

[0063] h body =C4*h head (4)

[0064] Among them, (x head y head The coordinates of the top-left corner of the target head detection bounding box b[i]; w head h is the width of the target head detection bounding box b[i]; head The height of the target head detection bounding box b[i]; (x body y body The coordinates of the top-left corner of the target head detection bounding box b[i] and the target body prediction bounding box B'[i]; w body h is the width of the target head detection bounding box b[i] and the target whole-body prediction bounding box B'[i]; body Let B'[i] be the height of the whole-body prediction box B[i] of the target head detection box b[i]; C1, C2, C3 and C4 are constants whose values ​​can be determined empirically.

[0065] In one embodiment, determining the target whole-body association box in the multiple target head detection boxes includes: determining the intersection over union (IoU) matrix based on the multiple target whole-body prediction boxes and the multiple target whole-body detection boxes; and applying the Hungarian algorithm to the IoU matrix to determine the corresponding target whole-body association box for each target head detection box; wherein each element in the IoU matrix is ​​the intersection over union ratio between the corresponding target whole-body detection box in the multiple target whole-body detection boxes and the corresponding target whole-body prediction box in the multiple target head detection boxes.

[0066] In one embodiment, performing whole-body trajectory association includes associating the current target whole-body detection box with a target whole-body trajectory in the generated target whole-body trajectory set based on the Hungarian algorithm; and performing head trajectory association includes associating the current target head detection box with a target head trajectory in the generated target head trajectory set based on the Hungarian algorithm.

[0067] Generally, the training samples for a detection model used to determine the full-body bounding box of a target are sufficient, but the number of samples that simultaneously annotate the head and body bounding boxes may be insufficient. For example, the known dataset MOT20 in the MOT field only has body annotations and no head annotations. This results in insufficient samples for training the target detection model of the present invention. Therefore, the inventors conceived the following solution. In one embodiment, a single target detection model is a model obtained by performing the following operations: training a neural network-based first target detection model using a first dataset including target head bounding box annotations and target full-body bounding box annotations (e.g., a known dataset in the MOT field: CrowdHuman) so that the first target detection model can output the target head bounding box and target full-body bounding box of the test image; adding the missing annotations to a second dataset lacking target head bounding box annotations or target full-body bounding box annotations using the trained first target detection model; and training a second target detection model as a single target detection model using the first dataset and the second dataset with the missing annotations added; wherein the second dataset has more training samples than the first dataset.

[0068] The corrected target trajectory generated by Method 100 can be used for subsequent evaluation or extraction of target appearance features.

[0069] According to one aspect of this disclosure, an apparatus for multiple targets is provided. Figure 8 An exemplary block diagram of a multi-target apparatus 800 according to one embodiment of the present disclosure is shown.

[0070] The apparatus 800 includes: a memory 801 storing instructions Inst; and at least one processor 808 configured to execute instructions Inst to implement method 100.

[0071] According to one aspect of this disclosure, a computer-readable non-transitory storage medium storing a program is provided. When the program is executed by a computer, the program causes the computer to perform the following operations: determining a plurality of target full-body detection boxes in a current input image by performing target full-body detection, and determining a plurality of target head detection boxes in the current input image by performing target head detection; determining full-body identifiers of the plurality of target full-body detection boxes by performing full-body trajectory association to update the target full-body trajectory set; determining head identifiers of the plurality of target head detection boxes by performing head trajectory association to update the target head trajectory set; determining a plurality of target full-body prediction boxes corresponding to the plurality of target head detection boxes based on the position and size of the plurality of target head detection boxes; determining target full-body association boxes of the plurality of target head detection boxes within the plurality of target full-body detection boxes based on the regions occupied by the plurality of target full-body prediction boxes in the input image; and updating the target full-body trajectory set based on the full-body identifiers of the target full-body association boxes of the plurality of target head detection boxes. Further details of the program can be found in the description of method 100.

[0072] According to one aspect of this disclosure, another apparatus for multiple targets is also provided. Figure 9 An exemplary block diagram of a multi-target apparatus 900 according to one embodiment of the present disclosure is shown.

[0073] The apparatus 900 includes a detection unit 901, a full-body tracking unit 903, a head tracking unit 905, and a correction unit 907. The detection unit 901 is configured to: determine multiple target full-body detection boxes in the current input image Im[t] by performing target full-body detection, and determine multiple target head detection boxes in the current input image by performing target head detection. The full-body tracking unit 903 is configured to update the target full-body trajectory set by determining the full-body identifiers of the multiple target full-body detection boxes through performing full-body trajectory association. Figure 9 The example illustrates the whole-body identifiers wid01, wid02, wid03, and wid04 determined by the whole-body detection box of the target in the image Im[t] for the whole-body tracking unit 903. The head tracking unit 905 is configured to determine the head identifiers of multiple target head detection boxes by performing head trajectory association to update the target head trajectory set. Figure 9The image exemplifies the head identifiers hid01, hid02, hid03, and hid04 determined by the head tracking unit 905 for target head detection boxes in image Im[t]. The correction unit 907 includes a prediction unit 971, an association unit 973, and an update unit 975. The prediction unit 971 is configured to determine multiple target whole-body prediction boxes corresponding to the multiple target head detection boxes based on the positions and sizes of the multiple target head detection boxes (see [link to documentation]). Figure 9 (The dashed box is shown in the image). The association unit 973 is configured to determine the target-whole association box within the multiple target-whole prediction boxes based on the region occupied by the multiple target-head detection boxes in the input image (see [reference]). Figure 9 Each head identifier has been associated with a full-body identifier. For example, it is assumed that the full-body identifiers of the target full-body detection boxes (i.e., the target full-body associated boxes) of two targets with head identifiers hid01 and hid02 have undergone identifier switching. The update unit 975 is configured to update the target full-body trajectory set based on the full-body identifiers of the target full-body associated boxes of multiple target head detection boxes (see [link to documentation]). Figure 9 (The labeling of the full-body detection bounding boxes for the two targets with head labels hid01 and hid02 has been corrected). Table 3 shows the correction status of the current trajectory points (corresponding to the input image Im[t]) for the trajectory with the full-body label "wid01". More details of the device 900 can be found in the description of method 100.

[0074] Table 3 Correction of the trajectory marked "wid01" for the whole body

[0075]

[0076]

[0077] According to one aspect of this disclosure, an information processing device is also provided.

[0078] Figure 10 This is an exemplary block diagram of an information processing device 1000 according to one embodiment of the present disclosure. Figure 10 In this system, the central processing unit (CPU) 1001 performs various processes based on programs stored in the read-only memory (ROM) 1002 or programs loaded from the storage device 1008 into the random access memory (RAM) 1003. The RAM 1003 also stores data required by the CPU 1001 for performing various processes, as needed.

[0079] CPU 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output interface 1005 is also connected to bus 1004.

[0080] The following components are connected to the input / output interface 1005: an input device 1006 including a soft keyboard, etc.; an output device 1007 including a display such as a liquid crystal display (LCD) and speakers, etc.; a storage device 1008 such as a hard disk; and a communication device 1009 including a network interface card such as a LAN card, a modem, etc. The communication device 1009 performs communication processing via a network such as the Internet, a local area network, a mobile network, or a combination thereof.

[0081] The driver 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a semiconductor memory, is installed on the driver 1010 as needed, so that programs read from it can be installed into the storage device 1008 as needed.

[0082] CPU 1001 can run programs corresponding to the methods for multi-target tracking disclosed herein.

[0083] To verify the effectiveness of Method 100, the inventors conducted tests on two different test datasets: FRDCCrowd (internal dataset) and MOT20 (public dataset), and the results are shown in Table 4. The results show that, due to the correction of ID switching in the whole-body tracking trajectory using the head-whole-body joint tracking method, Method 100 can improve the accuracy of multi-target tracking (see the value of the IDF1 evaluation metric for multi-target tracking MOT).

[0084] Table 4 Test Results of Method 100

[0085]

[0086]

[0087] In this disclosure: the target head trajectory is used to check whether an ID switch has occurred in the associated target full-body trajectory. If the tracking ID of the associated full-body detection box changes, it is considered an ID switch, and the target full-body trajectory is corrected by changing its tracking ID. The application areas of this disclosure include, but are not limited to: video surveillance in public places, intelligent monitoring, behavior recognition, and personnel tracking. The beneficial effects of the methods, apparatus, and storage media of this disclosure include at least one of the following: reducing ID switching and improving the accuracy of multi-target tracking.

[0088] As described above, the principles of data conversion have been disclosed according to this disclosure. It should be noted that the effects of the solutions disclosed herein are not necessarily limited to those described above, and any effect shown in this specification or other effects that can be understood from this specification can be achieved in addition to or in place of the effects described in the preceding paragraphs.

[0089] Although the invention has been disclosed above through a description of specific embodiments, it should be understood that those skilled in the art can devise various modifications (including, where applicable, combinations or substitutions of features between embodiments), improvements, or equivalents to the invention within the scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the scope of this disclosure.

[0090] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0091] Furthermore, the methods of the various embodiments of the present invention are not limited to being performed in the chronological order described in the specification or shown in the drawings, but may also be performed in other chronological orders, in parallel, or independently. Therefore, the execution order of the methods described in this specification does not constitute a limitation on the technical scope of the present invention.

[0092] Postscript

[0093] This disclosure includes, but is not limited to, the following schemes.

[0094] 1. A method for multi-target tracking, comprising:

[0095] Multiple target full-body detection boxes are determined in the current input image by performing target full-body detection, and multiple target head detection boxes are determined in the current input image by performing target head detection;

[0096] The whole-body identifiers of the multiple target whole-body detection boxes are determined by performing whole-body trajectory association to update the target whole-body trajectory set;

[0097] The head identifier of the multiple target head detection boxes is determined by performing head trajectory association to update the target head trajectory set;

[0098] Based on the position and size of the multiple target head detection boxes, determine multiple target whole-body prediction boxes corresponding to the multiple target head detection boxes;

[0099] Based on the region occupied by the multiple target whole-body prediction boxes of the multiple target head detection boxes in the input image, determine the target whole-body association box of the multiple target head detection boxes within the multiple target whole-body detection boxes; and

[0100] The target whole-body trajectory set is updated based on the whole-body association box of the target whole-body detection boxes of the multiple target head detection boxes.

[0101] 2. The method according to Appendix 1, wherein updating the target whole-body trajectory set based on the whole-body identifier of the target whole-body association box of the plurality of target head detection boxes includes:

[0102] For each trajectory in the target head trajectory set, if the full-body identifier of the target full-body bounding box of the current trajectory point is different from the full-body identifier of the target full-body bounding box of the previous trajectory point, then the full-body identifier of the target full-body bounding box of the current trajectory point is replaced with the full-body identifier of the target full-body bounding box of the previous trajectory point.

[0103] 3. The method according to Appendix 1, wherein a single target detection model is used to determine the plurality of whole-body target detection boxes and the plurality of head target detection boxes in the current input image.

[0104] 4. The method according to Appendix 1, wherein the method is configured to be suitable for online multi-target tracking.

[0105] 5. The method according to Appendix 1, wherein determining the full-body identifier of the plurality of target full-body detection boxes and determining the head identifier of the plurality of target head detection boxes are based on a target tracking algorithm.

[0106] 6. The method according to Appendix 5, wherein the whole-body identifiers of the plurality of target whole-body detection boxes in the current input image are determined by a first Kalman filter; and

[0107] The head identifiers of the plurality of target head detection boxes in the current input image are determined by a second Kalman filter, which is different from the first Kalman filter.

[0108] 7. According to the method described in Appendix 1, determining the target whole-body prediction box corresponding to one of the plurality of target head detection boxes includes:

[0109] The x-coordinate component of the target's whole-body prediction box is determined by linearly combining the x-coordinate component of the target's head detection box and its width.

[0110] The ordinate component of the target's whole-body prediction box is determined by linearly combining the ordinate component of the target's head detection box position and its height.

[0111] The width of the target's whole-body prediction box is determined by increasing the width of the target's head detection box; and

[0112] The height of the whole-body prediction box for the target is determined by magnifying the height of the detection box.

[0113] 8. The method according to Appendix 1, wherein determining the target whole-body association box in the plurality of target head detection boxes within the plurality of target whole-body detection boxes includes:

[0114] A cross-union matrix is ​​determined based on the multiple target whole-body prediction bounding boxes and the multiple target whole-body detection bounding boxes; and

[0115] The Hungarian algorithm is applied to the intersection-union matrix to determine the corresponding whole-body bounding box for each head detection box; and

[0116] Wherein, each element in the intersection-union ratio matrix is ​​the intersection-union ratio between the corresponding target whole-body detection box in the plurality of target whole-body detection boxes and the corresponding target whole-body prediction box in the plurality of target whole-body prediction boxes.

[0117] 9. The method according to Appendix 1, wherein performing the whole-body trajectory association includes associating the current target whole-body detection box with a target whole-body trajectory in the generated target whole-body trajectory set based on the Hungarian algorithm; and

[0118] Performing the head trajectory association includes associating the current target head detection box with a target head trajectory in the generated target head trajectory set based on the Hungarian algorithm.

[0119] 10. The method according to Appendix 3, wherein the single target detection model is a model obtained by performing the following operations:

[0120] The first target detection model based on a neural network is trained using a first dataset that includes target head bounding box annotations and target full-body bounding box annotations, so that the first target detection model can output target head bounding boxes and target full-body bounding boxes for the test image;

[0121] The trained first object detection model was used to add missing annotations to the second dataset, which lacked head or full-body bounding box annotations; and

[0122] The second object detection model is trained using the first dataset and the second dataset with the missing annotations added, and serves as the single object detection model.

[0123] The second dataset has more training samples than the first dataset.

[0124] 11. An apparatus for multi-target tracking, characterized in that it comprises:

[0125] A memory, on which instructions are stored; and

[0126] At least one processor is configured to execute the instructions to implement the method described in any one of Appendices 1 to 10.

[0127] 12. A computer-readable non-transitory storage medium storing a program, characterized in that, when the program is executed by a computer, the program causes the computer to:

[0128] Multiple target full-body detection boxes are determined in the current input image by performing target full-body detection, and multiple target head detection boxes are determined in the current input image by performing target head detection;

[0129] The whole-body identifiers of the multiple target whole-body detection boxes are determined by performing whole-body trajectory association to update the target whole-body trajectory set;

[0130] The head identifier of the multiple target head detection boxes is determined by performing head trajectory association to update the target head trajectory set;

[0131] Based on the position and size of the multiple target head detection boxes, determine multiple target whole-body prediction boxes corresponding to the multiple target head detection boxes;

[0132] Based on the region occupied by the multiple target whole-body prediction boxes of the multiple target head detection boxes in the input image, determine the target whole-body association box of the multiple target head detection boxes within the multiple target whole-body detection boxes; and

[0133] The target whole-body trajectory set is updated based on the whole-body association box of the target whole-body detection boxes of the multiple target head detection boxes.

[0134] 13. The computer-readable non-transitory storage medium according to Appendix 12, wherein updating the target whole-body trajectory set based on the whole-body identifier of the target whole-body association box of the plurality of target head detection boxes includes:

[0135] For each trajectory in the target head trajectory set, if the full-body identifier of the target full-body bounding box of the current trajectory point is different from the full-body identifier of the target full-body bounding box of the previous trajectory point, then the full-body identifier of the target full-body bounding box of the current trajectory point is replaced with the full-body identifier of the target full-body bounding box of the previous trajectory point.

[0136] 14. The computer-readable non-transitory storage medium according to Appendix 12, wherein the plurality of whole-body detection boxes and the plurality of head detection boxes in the current input image are determined using a single target detection model.

[0137] 15. The computer-readable non-transitory storage medium according to Appendix 12, wherein the method is configured to be suitable for online multi-target tracking.

[0138] 16. The computer-readable non-transitory storage medium according to Appendix 12, wherein determining the whole-body identifier of the plurality of target whole-body detection boxes and determining the head identifier of the plurality of target head detection boxes are based on a target tracking algorithm.

[0139] 17. The computer-readable non-transitory storage medium according to Appendix 16, wherein the whole-body identifiers of the plurality of target whole-body detection boxes in the current input image are determined by a first Kalman filter; and

[0140] The head identifiers of the plurality of target head detection boxes in the current input image are determined by a second Kalman filter, which is different from the first Kalman filter.

[0141] 18. The computer-readable non-transitory storage medium according to Appendix 13, wherein determining the target whole-body prediction box corresponding to one of the plurality of target head detection boxes includes:

[0142] The x-coordinate component of the target's whole-body prediction box is determined by linearly combining the x-coordinate component of the target's head detection box and its width.

[0143] The ordinate component of the target's whole-body prediction box is determined by linearly combining the ordinate component of the target's head detection box position and its height.

[0144] The width of the target's whole-body prediction box is determined by increasing the width of the target's head detection box; and

[0145] The height of the whole-body prediction box for the target is determined by magnifying the height of the detection box.

[0146] 19. The computer-readable non-transitory storage medium according to Appendix 12, wherein determining the target whole-body association box in the plurality of target head detection boxes includes:

[0147] A cross-union matrix is ​​determined based on the multiple target whole-body prediction bounding boxes and the multiple target whole-body detection bounding boxes; and

[0148] The Hungarian algorithm is applied to the intersection-union matrix to determine the corresponding whole-body bounding box for each head detection box; and

[0149] Wherein, each element in the intersection-union ratio matrix is ​​the intersection-union ratio between the corresponding target whole-body detection box in the plurality of target whole-body detection boxes and the corresponding target whole-body prediction box in the plurality of target whole-body prediction boxes.

[0150] 20. The computer-readable non-transitory storage medium according to Appendix 12, wherein performing the whole-body trajectory association includes associating the current target whole-body detection box with a target whole-body trajectory in the generated target whole-body trajectory set based on the Hungarian algorithm; and

[0151] Performing the head trajectory association includes associating the current target head detection box with a target head trajectory in the generated target head trajectory set based on the Hungarian algorithm.

Claims

1. A method for multi-target tracking, comprising: Multiple target full-body detection boxes are determined in the current input image by performing target full-body detection, and multiple target head detection boxes are determined in the current input image by performing target head detection; The whole-body identifiers of the multiple target whole-body detection boxes are determined by performing whole-body trajectory association to update the target whole-body trajectory set; The head identifier of the multiple target head detection boxes is determined by performing head trajectory association to update the target head trajectory set; Based on the position and size of the multiple target head detection boxes, determine multiple target whole-body prediction boxes corresponding to the multiple target head detection boxes; Based on the region occupied by the multiple target head detection boxes and the multiple target whole body prediction boxes in the input image, the target whole body association box of the multiple target head detection boxes in the multiple target whole body detection boxes is determined; as well as The target whole-body trajectory set is updated based on the whole-body association box of the target whole-body detection boxes of the multiple target head detection boxes.

2. The method according to claim 1, wherein, Updating the target whole-body trajectory set based on the whole-body association box of the multiple target head detection boxes includes: For each trajectory in the target head trajectory set, if the full-body identifier of the target full-body bounding box of the current trajectory point is different from the full-body identifier of the target full-body bounding box of the previous trajectory point, then the full-body identifier of the target full-body bounding box of the current trajectory point is replaced with the full-body identifier of the target full-body bounding box of the previous trajectory point.

3. The method according to claim 1, wherein, A single object detection model is used to determine the multiple full-body detection boxes and the multiple head detection boxes of the objects in the current input image.

4. The method according to claim 1, wherein, The determination of the full-body identifier of the multiple target full-body detection boxes and the determination of the head identifier of the multiple target head detection boxes are based on the target tracking algorithm.

5. The method according to claim 4, wherein, The full-body identifiers of the plurality of target full-body detection boxes in the current input image are determined by a first Kalman filter; and The head identifiers of the plurality of target head detection boxes in the current input image are determined by a second Kalman filter, which is different from the first Kalman filter.

6. The method according to claim 1, wherein, Determining the target whole-body prediction box corresponding to one of the multiple target head detection boxes includes: The x-coordinate component of the target's whole-body prediction box is determined by linearly combining the x-coordinate component of the target's head detection box and its width. The ordinate component of the target's whole-body prediction box is determined by linearly combining the ordinate component of the target's head detection box position and its height. The width of the target's whole-body prediction box is determined by increasing the width of the target's head detection box; and The height of the whole-body prediction box for the target is determined by magnifying the height of the detection box.

7. The method according to claim 1, wherein, Determining the target whole-body association box within the plurality of target head detection boxes in the plurality of target whole-body detection boxes includes: A cross-union matrix is ​​determined based on the multiple target whole-body prediction bounding boxes and the multiple target whole-body detection bounding boxes; and The Hungarian algorithm is applied to the intersection-union matrix to determine the corresponding whole-body bounding box for each head detection box; and Wherein, each element in the intersection-union ratio matrix is ​​the intersection-union ratio between the corresponding target whole-body detection box in the plurality of target whole-body detection boxes and the corresponding target whole-body prediction box in the plurality of target whole-body prediction boxes.

8. The method according to claim 1, wherein, Performing the whole-body trajectory association includes associating the current target whole-body detection box with a target whole-body trajectory from the already generated target whole-body trajectory set based on the Hungarian algorithm; and Performing the head trajectory association includes associating the current target head detection box with a target head trajectory in the generated target head trajectory set based on the Hungarian algorithm.

9. A device for multi-target tracking, characterized in that, include: A memory that stores instructions; as well as At least one processor is configured to execute the instructions to implement the method of any one of claims 1 to 8.

10. A computer-readable non-transitory storage medium storing a program, characterized in that, When the program is executed by a computer, the program causes the computer to: Multiple target full-body detection boxes are determined in the current input image by performing target full-body detection, and multiple target head detection boxes are determined in the current input image by performing target head detection; The whole-body identifiers of the multiple target whole-body detection boxes are determined by performing whole-body trajectory association to update the target whole-body trajectory set; The head identifier of the multiple target head detection boxes is determined by performing head trajectory association to update the target head trajectory set; Based on the position and size of the multiple target head detection boxes, determine multiple target whole-body prediction boxes corresponding to the multiple target head detection boxes; Based on the region occupied by the multiple target head detection boxes and the multiple target whole body prediction boxes in the input image, the target whole body association box of the multiple target head detection boxes in the multiple target whole body detection boxes is determined; as well as The target whole-body trajectory set is updated based on the whole-body association box of the target whole-body detection boxes of the multiple target head detection boxes.

Images