A multi-target tracking method, a terminal device, and a storage medium

Abstract

The application relates to a multi-target tracking method, a terminal device and a storage medium, and the method comprises the following steps: reading video information; performing target segmentation on frame images in the video information to obtain pixel-level information and apparent feature information of the target; predicting a track in each frame image based on a Kalman filtering algorithm; calculating appearance similarity between the track in a previous frame image and apparent feature information of the target in a current frame image, extracting a track and a target with appearance similarity greater than a similarity threshold as a pre-matching track and a pre-matching target, and storing the pre-matching target in a matching target set; calculating a cost matrix between the pre-matching track and the pre-matching target, calculating a Mask-IoU score between an un-matching track and an un-matching target, and fusing the two to obtain a final cost matrix; and obtaining a track matching result through a Hungarian algorithm. Compared with the prior art, the application can have both efficiency and performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video signal processing, in particular to a multi-target tracking method, a terminal device and a storage medium. BACKGROUND

[0002] The existing mainstream real-time tracking method usually uses a tracking method constructed based on deep learning to improve the association accuracy. However, in some cases, the tracking method constructed based on deep learning cannot achieve high tracking accuracy, fast operation and low cost. The high complexity of these methods often leads to a balance between performance and efficiency. SUMMARY

[0003] In order to solve the above problems, the present application provides a multi-target tracking method, a terminal device and a storage medium.

[0004] The specific scheme is as follows:

[0005] A multi-target tracking method, comprising the following steps:

[0006] S1: reading video information to be tracked;

[0007] S2: performing target segmentation on the frame image in the video information to obtain pixel-level information of the target; based on the pixel-level information of the target, obtaining appearance feature information of the target through a re-identification model;

[0008] S3: predicting the trajectory in each frame image based on a Kalman filter algorithm;

[0009] S4: based on the trajectory in the previous frame image and the appearance feature information of the target in the current frame image, calculating the appearance similarity between the two, extracting the trajectory and the target with the appearance similarity greater than the similarity threshold as the pre-matching trajectory and the pre-matching target, storing the pre-matching target in the matching target set, and taking other targets in the current frame image except the pre-matching target as the unmatched target, and taking other trajectories in the previous frame image except the pre-matching trajectory as the unmatched trajectory;

[0010] S5: calculating the Mask-IoU score between the pre-matching trajectory and the pre-matching target through the pixel-level information of the pre-matching trajectory and the pre-matching target; calculating the appearance similarity between the pre-matching trajectory and the pre-matching target through the appearance feature information of the pre-matching trajectory and the pre-matching target; taking the weighted sum result of the Mask-IoU score and the appearance similarity as the cost matrix of the pre-matching trajectory and the pre-matching target; calculating the Mask-IoU score between the unmatched trajectory and the unmatched target through the pixel-level information of the unmatched trajectory and the unmatched target; and fusing the cost matrix of the pre-matching trajectory and the pre-matching target and the Mask-IoU score between the unmatched trajectory and the unmatched target to obtain the final cost matrix;

[0011] S6: obtaining a matching result of each target and a track in the previous frame image based on the final cost matrix and the track set of the previous frame image by using a Hungarian algorithm.

[0012] Further, the step S4 includes the following specific processes:

[0013] extracting all tracks in the track set of the previous frame image whose track survival time is less than the maximum track survival time to form a survival track set;

[0014] adding the target in the current frame image to the unmatched target set;

[0015] pairing the tracks in the survival track set and the targets in the unmatched target set into candidate pairs;

[0016] calculating the appearance similarity between the tracks and the targets in each candidate pair, and extracting the track and the target in the candidate pair whose appearance similarity is greater than a similarity threshold as a pre-matching track and a pre-matching target.

[0017] Further, the track survival time is initialized to 0, and the track survival time of the track determined as the pre-matching track in each frame is set to 0, and the track survival time of other tracks is increased by 1.

[0018] Further, the calculation formula of the cost matrix N is as follows:

[0019] N = λC + (1-λ)I

[0020] wherein λ represents a weight, C represents an appearance similarity matrix, and I represents a Mask-IoU score matrix.

[0021] Further, the track matched to the target in the previous frame and the previous n consecutive frames is regarded as a confirmed track of the previous frame, and other tracks are regarded as unconfirmed tracks of the previous frame, the track in the previous frame image in the step S4 is the confirmed track of the previous frame, and the track set of the previous frame image in the step S6 is a set composed of all tracks in the previous frame image.

[0022] A multi-target tracking terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described above in the embodiments of the present application when executing the computer program.

[0023] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps of the method described above in the embodiments of the present application.

[0024] The present application adopts the above technical solution, which can have both efficiency and performance compared with the existing method, and can meet the actual application requirements of low cost and instantaneity at the present stage. Attached Figure Description

[0025] Figure 1 The diagram shown is a flowchart of Embodiment 1 of the present invention. Detailed Implementation

[0026] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0027] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0028] Example 1:

[0029] This invention provides a multi-target tracking method, such as... Figure 1 As shown, the method includes the following steps:

[0030] S1: Read the video information of the target to be tracked.

[0031] In this embodiment, video information is read through the OpenCV interface.

[0032] S2: Perform target segmentation on the frame images in the video information to obtain the pixel-level information of the target; based on the pixel-level information of the target, obtain the appearance feature information of the target through the re-identification model.

[0033] Both the target segmentation model and the re-identification model can use existing models, and no restrictions are imposed here.

[0034] S3: Predict the trajectory in each frame of the image based on the Kalman filter algorithm.

[0035] S4: Based on the appearance feature information of the trajectory in the previous frame image and the target in the current frame image, calculate the appearance similarity between the two, extract the trajectory and target with appearance similarity greater than the similarity threshold as pre-matching trajectory and pre-matching target, store the pre-matching target in the matching target set, take other targets in the current frame image other than the pre-matching target as unmatched targets, and take other trajectories in the previous frame image other than the pre-matching trajectory as unmatched trajectories.

[0036] Step S4 belongs to the pre-matching stage, which includes the following steps:

[0037] S401: Extract all trajectories from the previous frame image whose trajectory survival time is less than the maximum trajectory survival time, and sort them in ascending order of trajectory survival time to form a set of surviving trajectories.

[0038] S402: Add the target in the current frame image to the unmatched target set;

[0039] S403: Pair the trajectories in the surviving trajectory set with the targets in the unmatched target set into candidate pairs;

[0040] S404: Calculate the appearance similarity between the trajectories and the targets in each candidate pair, and extract the trajectories and the targets in the candidate pairs with the appearance similarity greater than the similarity threshold value as the pre-matching trajectories and the pre-matching targets.

[0041] In the appearance similarity calculation, the cosine distance algorithm is adopted, and the similarity threshold value and the maximum trajectory survival time are set by the person skilled in the art according to the requirements.

[0042] Each pre-matching target has only one pre-matching trajectory, if there are two or more candidate pairs with the appearance similarity greater than the similarity threshold value for the pre-matching target, the trajectory in the candidate pair with the maximum appearance similarity is selected as the matching trajectory of the pre-matching target.

[0043] In the setting of the trajectory survival time, the initial setting is 0, the trajectory survival time of the trajectory determined as the pre-matching trajectory in each frame is set to 0, and the trajectory survival time of other trajectories is increased by 1.

[0044] S5: Calculate the Mask-IoU score between the pre-matching trajectories and the pre-matching targets through the pixel-level information of the pre-matching trajectories and the pre-matching targets, calculate the appearance similarity between the pre-matching trajectories and the pre-matching targets through the appearance feature information of the pre-matching trajectories and the pre-matching targets, take the weighted sum result of the Mask-IoU score and the appearance similarity as the cost matrix of the pre-matching trajectories and the pre-matching targets, calculate the Mask-IoU score between the unmatched trajectories and the unmatched targets through the pixel-level information of the unmatched trajectories and the unmatched targets, and fuse the cost matrix of the pre-matching trajectories and the pre-matching targets and the Mask-IoU score between the unmatched trajectories and the unmatched targets to obtain the final cost matrix.

[0045] The calculation formula of the cost matrix N in this embodiment is:

[0046] N = λC + (1-λ)I

[0047] Wherein, λ represents the weight, C represents the appearance similarity matrix, and I represents the Mask-IoU score matrix.

[0048] The calculation formula of the final cost matrix F is:

[0049]

[0050] Wherein, H represents the similarity threshold value.

[0051] S6: Based on the final cost matrix and the track set of the previous frame image, the matching result of each target and the track in the previous frame image is obtained by using the Hungarian algorithm.

[0052] In order to further improve the matching efficiency, in this embodiment, the track of the previous frame and the track of the target matched in the n continuous frames before the previous frame are regarded as the confirmed state track of the previous frame, and the other tracks are regarded as the unconfirmed state track of the previous frame. The track in the previous frame image in step S4 is the confirmed state track of the previous frame, and the track set of the previous frame image in step S6 is the set composed of all the tracks in the previous frame image. In this way, the matching efficiency is ensured, and the omission of the track when the target corresponding to the unconfirmed state track reappears in the subsequent video is avoided.

[0053] n is an integer greater than 2, and in this embodiment, n is set to 3.

[0054] The embodiment of the present application proposes a multi-target tracking method capable of combining efficiency and performance. The method uses the strategy of first frame-by-frame detection and segmentation, and then data association, uses traditional appearance and motion information for matching, and integrates pre-matching information and a fast association algorithm with selection and weighting. The embodiment proposes a simple and flexible tracking method capable of achieving good tracking effect, and can meet the needs of low-cost and real-time practical applications.

[0055] Embodiment two:

[0056] The present application also provides a multi-target tracking terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above method embodiments of the embodiment one of the present application when executing the computer program.

[0057] Further, as an executable scheme, the multi-target tracking terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The multi-target tracking terminal device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above multi-target tracking terminal device is only an example of the multi-target tracking terminal device, and does not constitute a limitation on the multi-target tracking terminal device, and can include more or fewer components than the above, or combine certain components, or different components, for example, the multi-target tracking terminal device can also include an input / output device, a network access device, a bus, and the like, and the embodiments of the present application do not limit this.

[0058] Further, as an executable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, which is a control center of the multi-target tracking terminal device and connects various parts of the multi-target tracking terminal device through various interfaces and lines.

[0059] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the multi-target tracking terminal device by running or executing the computer program and / or modules stored in the memory and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0060] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the method provided in the embodiments of the application.

[0061] The modules / units integrated in the multi-target tracking terminal device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium, etc.

[0062] Although the present application is specifically shown and described in conjunction with the preferred embodiments, it should be understood that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A multi-target tracking method characterized by, The method comprises the following steps: S1: reading video information to be tracked; S2: performing target segmentation on the frame images in the video information to obtain pixel-level information of the target; and obtaining appearance feature information of the target based on the pixel-level information of the target through a re-identification model; S3: predicting the trajectories in each frame image based on a Kalman filtering algorithm; S4: calculating the appearance similarity between the trajectories in a previous frame image and the appearance feature information of the target in a current frame image, extracting the trajectories and the target with the appearance similarity greater than a similarity threshold as pre-matching trajectories and pre-matching targets, storing the pre-matching targets in a matching target set, and taking the other targets in the current frame image as un-matching targets and taking the other trajectories in the pre-matching trajectories in the previous frame image as un-matching trajectories; S5: calculating the Mask-IoU score between the pre-matching trajectories and the pre-matching targets through the pixel-level information of the pre-matching trajectories and the pre-matching targets; calculating the appearance similarity between the pre-matching trajectories and the pre-matching targets through the appearance feature information of the pre-matching trajectories and the pre-matching targets; taking the weighted sum of the Mask-IoU score and the appearance similarity as the cost matrix of the pre-matching trajectories and the pre-matching targets; and calculating the Mask-IoU score between the un-matching trajectories and the un-matching targets through the pixel-level information of the un-matching trajectories and the un-matching targets; and fusing the cost matrix of the pre-matching trajectories and the pre-matching targets and the Mask-IoU score between the un-matching trajectories and the un-matching targets to obtain a final cost matrix; S6: obtaining the matching results of each target and the trajectories in the previous frame image based on the final cost matrix and the trajectory set of the previous frame image through a Hungarian algorithm; The specific process of step S4 comprises: extracting all trajectories in the trajectory set of the previous frame image with a trajectory survival time less than a maximum trajectory survival time to form a survival trajectory set; adding the targets in the current frame image to the un-matching target set; pairing the trajectories in the survival trajectory set and the targets in the un-matching target set as candidate pairs; extracting the trajectories and the targets in the candidate pairs with an appearance similarity greater than a similarity threshold as pre-matching trajectories and pre-matching targets; the calculation formula of the cost matrix N is: N = C +（ ） I wherein, denotes a weight, C denotes an appearance similarity matrix, I denotes a Mask-IoU score matrix.

2. The multi-target tracking method of claim 1, wherein: the trajectory survival time is initialized as 0, the trajectory survival time of the trajectory determined as a pre-matching trajectory in each frame is set as 0, and the trajectory survival time of other trajectories is increased by 1.

3. The multi-target tracking method of claim 1, wherein: taking the trajectories matched to the target in the previous frame and the previous n frames as confirmed state trajectories of the previous frame, and taking other trajectories as unconfirmed state trajectories of the previous frame, the trajectories in the previous frame image in step S4 are the confirmed state trajectories of the previous frame, and the trajectory set of the previous frame image in step S6 is a set composed of all trajectories in the previous frame image.

4. A multi-target tracking terminal device, characterized by: The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-3.

5. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-3.

Images