Multi-object trajectory tracking method and apparatus

Abstract

The application discloses a kind of multi-target trajectory tracking method and device.Therein, the method includes: obtaining the multiple first tracking targets corresponding to the current frame video image of the video information to be detected, and the historical trajectory tracking result corresponding to the previous frame video image of the current frame video image;According to the historical trajectory tracking result, the multiple first tracking targets are processed using a multi-target tracking model, to obtain the first prediction box corresponding to the multiple first tracking targets respectively;According to the historical trajectory tracking result, the multiple first tracking targets are processed using Kalman filtering algorithm, to obtain the second prediction box corresponding to the multiple first tracking targets respectively;Based on the first prediction box and the second prediction box, the target trajectory tracking result corresponding to the multiple first tracking targets respectively is obtained.The application solves the technical problems of low tracking efficiency and poor recognition ability in multi-target tracking scenarios in related technologies.

Description

Technical Field

[0001] This invention relates to the field of multi-target tracking, and more specifically, to a method and apparatus for multi-target trajectory tracking. Background Technology

[0002] Currently, multi-target tracking technology has been widely used in various fields such as autonomous driving, intelligent monitoring, and pedestrian tracking. Deep learning algorithms are often used for continuous tracking in these technologies. However, in practical applications, various complex situations arise, such as target trajectory occlusion and overlap, poor image clarity, dense targets, and moving targets. These issues lead to unsatisfactory multi-target tracking results in complex scenes, resulting in low tracking accuracy and a high probability of false positives and false negatives.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a multi-target trajectory tracking method and apparatus to at least solve the technical problems of low tracking efficiency and poor recognition ability in multi-target tracking scenarios in related technologies.

[0005] According to one aspect of the present invention, a multi-target trajectory tracking method is provided, comprising: acquiring a plurality of first tracking targets corresponding to a current frame video image in video information to be detected, and historical trajectory tracking results corresponding to a previous frame video image of the current frame video image; processing the plurality of first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the plurality of first tracking targets respectively; processing the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the plurality of first tracking targets respectively; and obtaining target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction boxes and the second prediction boxes respectively.

[0006] Optionally, obtaining the historical trajectory tracking result corresponding to the previous frame video image of the current frame video image includes: obtaining a plurality of second tracking targets corresponding to the previous frame video image, and historical detection boxes and historical prediction boxes corresponding to the plurality of second tracking targets respectively, wherein the historical detection boxes correspond to second tracking targets for which no historical trajectory has been generated, and the historical prediction boxes correspond to second tracking targets for which a historical trajectory has been generated; converting the historical detection boxes, and generating the historical trajectory tracking result based on the converted historical detection boxes and the historical prediction boxes.

[0007] Optionally, the step of processing the plurality of first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the plurality of first tracking targets respectively includes: determining whether the plurality of first tracking targets have historical trajectories based on the historical trajectory tracking results; if the plurality of first tracking targets all have historical trajectories, then processing the plurality of first tracking targets using the multi-target tracking model to generate the first prediction boxes corresponding to the plurality of first tracking targets respectively.

[0008] Optionally, the method further includes: if there is a first tracking target among the plurality of first tracking targets that does not have the historical trajectory, then the first tracking target among the plurality of first tracking targets that does not have the historical trajectory is designated as a third tracking target, and the first tracking targets among the plurality of first tracking targets other than the third tracking target are designated as a fourth tracking target; the third tracking target is processed using the multi-target tracking model to generate a first detection box corresponding to the third tracking target; the fourth tracking target is processed using the multi-target tracking model to generate a first prediction box corresponding to the fourth tracking target.

[0009] Optionally, when all of the plurality of first tracking targets have the historical trajectory, obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box includes: using the first prediction box as the target trajectory tracking results corresponding to the plurality of first tracking targets respectively; when there is a third tracking target among the plurality of first tracking targets that does not have the historical trajectory, obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box includes: obtaining the target tracking trajectory corresponding to the fourth tracking target among the plurality of first tracking targets other than the third tracking target based on the first prediction box and the second prediction box.

[0010] Optionally, obtaining the target tracking trajectory corresponding to the fourth tracking target (excluding the third tracking target) among the plurality of first tracking targets based on the first prediction box and the second prediction box includes: processing the first prediction box and the second prediction box corresponding to the fourth tracking target to obtain a first intersection-union ratio (IU / R) corresponding to the fourth tracking target; determining whether the first IU / R is less than a first preset threshold; if the first IU / R is less than the first preset threshold, then obtaining the target trajectory tracking result corresponding to the fourth tracking target based on the first prediction box and the second prediction box of the fourth tracking target.

[0011] Optionally, the method further includes: if the first intersection-union ratio (CIU) is greater than or equal to the first preset threshold, processing the second predicted bounding box of the fourth tracking target and the first detected bounding box of the third tracking target to obtain a second CIU corresponding to the fourth tracking target; determining whether the second CIU is less than a second preset threshold; if the second CIU is less than the second preset threshold, updating the first predicted bounding box of the fourth tracking target using the first detected bounding box of the third tracking target, and obtaining the target trajectory tracking result corresponding to the fourth tracking target based on the first detected bounding box of the third tracking target and the updated first predicted bounding box of the fourth tracking target.

[0012] Optionally, the method further includes: if the second intersection-union ratio is greater than or equal to the second preset threshold, updating the first detection box of the third tracking target using the first prediction box of the fourth tracking target; and obtaining the target trajectory tracking result corresponding to the fourth tracking target based on the updated first detection box of the third tracking target.

[0013] Optionally, the step of processing the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the plurality of first tracking targets respectively includes: processing the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain motion state information corresponding to the plurality of first tracking targets respectively; and obtaining the second prediction boxes corresponding to the plurality of first tracking targets respectively based on the motion state information.

[0014] Optionally, obtaining the multi-target tracking model includes: obtaining historical video information; determining historical target location information and historical target identification information in the historical video information, and using the historical target location information and historical target identification information as a multi-target training set; and obtaining the multi-target tracking model based on the multi-target training set and employing a multi-target tracking algorithm.

[0015] According to another aspect of the present invention, a multi-target trajectory tracking device is provided, comprising: a first acquisition module, configured to acquire a plurality of first tracking targets corresponding to a current frame video image in video information to be detected, and historical trajectory tracking results corresponding to a previous frame video image of the current frame video image; a first generation module, configured to process the plurality of first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the plurality of first tracking targets respectively; a second generation module, configured to process the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the plurality of first tracking targets respectively; and a third generation module, configured to obtain target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction boxes and the second prediction boxes respectively.

[0016] According to another aspect of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium storing a plurality of instructions adapted for loading by a processor and executing any one of the multi-target trajectory tracking methods described herein.

[0017] In this embodiment of the invention, multiple first tracking targets corresponding to the current frame video image in the video information to be detected are obtained, along with historical trajectory tracking results corresponding to the previous frame video image. Based on the historical trajectory tracking results, a multi-target tracking model is used to process the multiple first tracking targets to obtain first prediction boxes corresponding to each of the multiple first tracking targets. Based on the historical trajectory tracking results, a Kalman filter algorithm is used to process the multiple first tracking targets to obtain second prediction boxes corresponding to each of the multiple first tracking targets. Based on the first prediction boxes and the second prediction boxes, target trajectory tracking results corresponding to each of the multiple first tracking targets are obtained. This achieves the goal of combining multiple algorithms to improve dynamic tracking capabilities, realizing the technical effects of improving tracking accuracy and tracking efficiency, and thus solving the technical problems of low tracking efficiency and poor recognition ability in multi-target tracking scenarios in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 A flowchart of an optional multi-target trajectory tracking method provided according to an embodiment of the present invention;

[0020] Figure 2 A flowchart of another optional multi-target trajectory tracking method provided according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of an optional multi-target trajectory tracking device provided according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0025] The multi-target recognition algorithm (transformer) is an end-to-end target tracking method with high target tracking efficiency.

[0026] The Kalman filter algorithm is an algorithm that uses the equations governing the state of a linear system to make an optimal estimate of the system state using the system's input and output observation data.

[0027] Multi-target tracking algorithms detect targets and obtain their positions in images based on image information.

[0028] In related technologies, multi-target tracking algorithms are used to continuously track targets in video sequences and obtain the motion trajectories of multiple targets. With the development of deep learning, deep learning-based multi-target tracking methods have been widely used in fields such as autonomous driving, intelligent surveillance, and pedestrian tracking, enabling these fields to develop rapidly. However, deep learning-based multi-target tracking methods still suffer from low accuracy, false positives, and false negatives in specific scenarios such as complex scenes, overlapping target trajectories, small target pixel values, and dense target populations. Therefore, researching high-precision target tracking algorithms is of great significance.

[0029] In related technologies, multi-object tracking algorithms mainly include detection-based tracking and joint detection and feature extraction tracking, both of which have relatively low tracking efficiency. With the application of multi-object recognition algorithms in computer vision, end-to-end multi-object tracking frameworks have developed rapidly in recent years, and attention-based multi-object tracking methods have been widely researched and applied. Multi-object recognition algorithms implement query-key mechanisms for multi-object tracking. Using the embedding vector of the detected target in the previous frame as the query key for the current frame, the algorithm queries its position and scale information in the current image, providing end-to-end target tracking with high efficiency.

[0030] Multi-target tracking methods based on multi-target recognition algorithms predict and track the target's position in the next frame end-to-end, without fully considering the target's motion characteristics. This means the position of the same target in the next frame is related to its current motion trend. Consequently, target tracking efficiency is unsatisfactory, and the tracking trajectory repeatedly switches.

[0031] To address the aforementioned problems, this invention provides a method embodiment for multi-target trajectory tracking. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a multi-target trajectory tracking method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0033] Step S102: Obtain multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and the historical trajectory tracking results corresponding to the previous frame video image of the current frame video image.

[0034] Step S104: Based on the above historical trajectory tracking results, a multi-target tracking model is used to process the above multiple first tracking targets to obtain the first prediction boxes corresponding to the above multiple first tracking targets respectively.

[0035] Step S106: Based on the above historical trajectory tracking results, the Kalman filter algorithm is used to process the above multiple first tracking targets to obtain the second prediction boxes corresponding to the above multiple first tracking targets respectively.

[0036] Step S108: Based on the first prediction box and the second prediction box, obtain the target trajectory tracking results corresponding to the multiple first tracking targets respectively.

[0037] Through the above steps, the goal of combining multiple algorithms can be achieved to improve dynamic tracking capabilities, thereby increasing tracking accuracy and efficiency. This solves the technical problems of low tracking efficiency and poor recognition ability in multi-target tracking scenarios in related technologies.

[0038] In the multi-target trajectory tracking method provided in this embodiment of the invention, a combination of a multi-target tracking model and a Kalman filter algorithm is used to improve the target trajectory tracking capability. First, multiple first tracking targets corresponding to the current frame video image in the video information to be detected, as well as the historical trajectory tracking results corresponding to the previous frame video image, are obtained. Based on the historical trajectory tracking results, the multi-target tracking model is used to process the first tracking targets to obtain a first predicted bounding box. Based on the historical trajectory tracking results, the Kalman filter algorithm is used to process the first tracking targets to obtain a second predicted bounding box. Based on the first and second predicted bounding boxes, target trajectory tracking results corresponding to the multiple first tracking targets are generated.

[0039] Optionally, the first tracking target can be of various types, such as pedestrians, vehicles, and biological entities.

[0040] It should be noted that, since the target in real-world applications changes constantly with movement, affecting the tracking performance, combining the first and second prediction boxes can help improve the efficiency of multi-target tracking.

[0041] In an optional embodiment, obtaining the historical trajectory tracking result corresponding to the previous frame video image of the current frame video image includes: obtaining a plurality of second tracking targets corresponding to the previous frame video image, and historical detection boxes and historical prediction boxes corresponding to the plurality of second tracking targets respectively, wherein the historical detection boxes correspond to second tracking targets for which no historical trajectory has been generated, and the historical prediction boxes correspond to second tracking targets for which a historical trajectory has been generated; converting the historical detection boxes, and generating the historical trajectory tracking result based on the converted historical detection boxes and historical prediction boxes.

[0042] It is understandable that historical trajectory tracking results are input to the multi-target tracking model and the Kalman filter algorithm. To obtain historical trajectory tracking results, firstly, multiple second-target objects corresponding to the previous frame of video image are acquired. In the processing of the previous frame, historical detection boxes and historical prediction boxes corresponding to the multiple second-target objects are obtained. The historical detection boxes are then transformed, and based on the transformed historical detection boxes and historical prediction boxes, the historical trajectory tracking results are obtained.

[0043] Optionally, there are several ways to transform historical detection boxes. For example, the historical detection boxes can be transformed and then incorporated into the historical prediction boxes. In this way, the next frame will generate a new trajectory for the second tracking target (i.e., the historical detection box) that has not generated a historical trajectory based on the historical trajectory tracking results.

[0044] In an optional embodiment, the process of using a multi-target tracking model to process the plurality of first tracking targets based on the historical trajectory tracking results to obtain first prediction boxes corresponding to each of the plurality of first tracking targets includes: determining whether each of the plurality of first tracking targets has a historical trajectory based on the historical trajectory tracking results; if all of the plurality of first tracking targets have historical trajectories, then processing the plurality of first tracking targets using the multi-target tracking model to generate the first prediction boxes corresponding to each of the plurality of first tracking targets. It can be understood that, based on the historical trajectory tracking results, it is possible to determine whether any of the plurality of first tracking targets have historical trajectories. In the case where all of the plurality of first tracking targets have first tracking targets with historical trajectories, corresponding first prediction boxes are generated for the plurality of first tracking targets.

[0045] In an optional embodiment, the method further includes: if there is a first tracking target among the plurality of first tracking targets that does not have the historical trajectory, then the first tracking target among the plurality of first tracking targets that does not have the historical trajectory is designated as a third tracking target, and the first tracking targets among the plurality of first tracking targets other than the third tracking target are designated as a fourth tracking target; the third tracking target is processed using the multi-target tracking model to generate a first detection box corresponding to the third tracking target; the fourth tracking target is processed using the multi-target tracking model to generate a first prediction box corresponding to the fourth tracking target.

[0046] It is understandable that in practical applications, there may be various situations among multiple first-target tracking targets, including the case where there is a first-target tracking target without a historical trajectory. In this scenario, the first-target tracking target without a historical trajectory is designated as the third-target tracking target, and the first-target tracking target with a historical trajectory is designated as the fourth-target tracking target. After processing by the multi-target tracking model, a first detection box is generated for the third-target tracking target, and a first prediction box is generated for the fourth-target tracking target.

[0047] It should be noted that for any one of the multiple first tracking targets, the first tracking target may exist in either the first detection box or the first prediction box. The first tracking target with historical trajectory corresponds to the first detection box, and the first tracking target without historical trajectory corresponds to the first prediction box.

[0048] In an optional embodiment, when all of the plurality of first tracking targets have the aforementioned historical trajectories, obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the aforementioned first prediction box and the aforementioned second prediction box includes: using the aforementioned first prediction box as the target trajectory tracking results corresponding to the plurality of first tracking targets respectively; when there is a third tracking target among the plurality of first tracking targets that does not have the aforementioned historical trajectory, obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the aforementioned first prediction box and the aforementioned second prediction box includes: obtaining the target tracking trajectory corresponding to the fourth tracking target among the plurality of first tracking targets other than the aforementioned third tracking target based on the aforementioned first prediction box and the aforementioned second prediction box.

[0049] It is understandable that the distinction was made between two scenarios: one where multiple first tracking targets all have the aforementioned historical trajectories, and another where a third tracking target among the multiple first tracking targets does not have the aforementioned historical trajectories.

[0050] In one optional embodiment, obtaining the target trajectory tracking results corresponding to the fourth tracking target based on the first prediction box and the second prediction box includes: processing the first prediction box and the second prediction box of the fourth tracking target to obtain a first intersection-union ratio (IU) corresponding to the fourth tracking target; determining whether the first IU is less than a first preset threshold; if the first IU is less than the first preset threshold, then obtaining the target trajectory tracking results corresponding to the fourth tracking target based on the first prediction box and the second prediction box of the fourth tracking target.

[0051] It is understandable that, in order to improve tracking capabilities, the target trajectory tracking result is obtained by calculating the intersection-union ratio (IU / R). For the fourth tracking target with a historical trajectory, the corresponding first and second predicted boxes are processed to obtain the first IU / R of the first and second predicted boxes. It is then determined whether the first IU / R meets a first preset threshold. If the first IU / R is less than the first preset threshold, it is considered that the first predicted box obtained by the multi-target tracking model has completed correct tracking in the current frame. Based on the first and second predicted boxes, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0052] Optionally, the first intersection-union ratio (IUGR) can be of various types, for example: A represents the first predicted bounding box output by the multi-object tracking model, B represents the second predicted bounding box output by the Kalman filter algorithm, A∩B represents the area of ​​the union of A and B, and A∪B represents the area of ​​the intersection of A and B. The first IUGR is obtained in the following way: Wherein, IOU1 is the first intersection-union ratio. The first preset threshold is set to 1. When the first intersection-union ratio is less than 1, it is considered that the first predicted box obtained by the multi-target tracking model has completed correct tracking in the current frame. Based on the first predicted box and the second predicted box, the above target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0053] It should be noted that the physical meaning of the intersection-union ratio (IU) is to calculate the ratio of the intersection to the union of two bounding boxes (such as the first and second predicted boxes mentioned above). It is often used to test the target tracking capability. When the first and second predicted boxes perfectly overlap, the first IU is 1. Judging the relationship between the first IU and a preset first threshold (e.g., set to 1) can well reflect the accuracy of the output results of the multi-target tracking model.

[0054] In an optional embodiment, the method further includes: if the first intersection-union ratio (CIU) is greater than or equal to the first preset threshold, processing the second prediction box of the fourth tracking target and the first detection box of the third tracking target to obtain the second CIU corresponding to the fourth tracking target; determining whether the second CIU is less than the second preset threshold; if the second CIU is less than the second preset threshold, updating the first prediction box of the fourth tracking target using the first detection box of the third tracking target, and obtaining the target trajectory tracking result corresponding to the fourth tracking target based on the first detection box of the third tracking target and the updated first prediction box of the fourth tracking target.

[0055] It is understandable that, due to the potential for target loss in real-world scenarios, trajectory correction is performed for the fourth tracking target that was not correctly tracked in order to improve robustness and enhance multi-target tracking capabilities. If the first intersection-union ratio (IU) is greater than or equal to the first preset threshold, it is considered that the first predicted bounding box obtained by the multi-target tracking model failed to track correctly in the current frame and needs correction. Based on this judgment, further processing is performed. A second IU is obtained based on the second predicted bounding box of the fourth tracking target and the first detection bounding box of the third tracking target. It is then determined whether the second IU is less than the preset second threshold. If the second IU is less than the preset second threshold, it is considered that the algorithm has made a recognition error, and the correct tracking result of the fourth tracking target has been found in the third tracking target (which was originally considered to have no historical trajectory). Therefore, the first detection bounding box corresponding to the third tracking target is associated with the first predicted bounding box corresponding to the fourth tracking target as an update to the first predicted bounding box. Based on the first detection bounding box and the updated first predicted bounding box of the fourth tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0056] Optionally, the second intersection-union ratio (IUU) can be of various types, for example: C represents the second predicted bounding box output by the Kalman filter algorithm, D represents the first detected bounding box output by the multi-object tracking model, C∩D represents the area of ​​the union of C and D, and C∪D represents the area of ​​the intersection of C and D. The second IUU is obtained in the following way: Here, IOU2 represents the second intersection-union ratio. A second preset threshold is set to 0.3. When the second intersection-union ratio is less than 0.3, it is considered that the correct tracking result of the fourth tracking target has been found among the third tracking targets, and correction is required. Therefore, based on the first detection box and the updated first prediction box of the fourth tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0057] In an optional embodiment, the method further includes: if the second intersection-union ratio is greater than or equal to the second preset threshold, updating the first detection box of the third tracking target using the first prediction box of the fourth tracking target; and obtaining the target trajectory tracking result corresponding to the fourth tracking target based on the updated first detection box of the third tracking target.

[0058] It is understandable that in practical applications, there may be situations where tracking is lost and cannot be corrected, requiring re-tracking. If the second intersection-union ratio (IUU) is greater than or equal to the second preset threshold, it is considered that the correct tracking result for the fourth tracking target was not found in the third updated target, and the existing tracking trajectory cannot continue, requiring re-tracking. Using the first predicted bounding box as the first detection box is to re-establish a new trajectory in the next frame. After updating the first detection box, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0059] In an optional embodiment, the above-mentioned processing of the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the plurality of first tracking targets respectively includes: processing the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain motion state information corresponding to the plurality of first tracking targets respectively; and obtaining second prediction boxes corresponding to the plurality of first tracking targets respectively based on the motion state information.

[0060] It is understandable that the Kalman filter algorithm is used to improve the perception of motion trends. To combine this with the first predicted bounding box obtained from the multi-object tracking model, a second predicted bounding box is needed. First, using the same historical trajectory tracking results as the multi-object tracking model as input, the Kalman filter algorithm is applied to multiple first-tracked targets to obtain their corresponding motion state information. Based on this motion state information, second predicted bounding boxes are obtained for each of the multiple first-tracked targets.

[0061] Optionally, the above motion state information can take various forms. For example, the motion state of the target can be described using a state space approach. For ease of understanding, a specific example can be provided: A state space of (x, y, r, h, Δx, Δy, Δr, Δh) can be established to describe the motion state of the target. Here, (x, y) represents the coordinates of the target's center position, r is the aspect ratio of the second prediction box, h represents the height of the second prediction box, and Δx, Δy, Δr, and Δh are the time derivatives of x, y, r, and h in the image coordinate system, respectively. The Kalman filter algorithm predicts the motion state of the target in subsequent frames of the video information to be detected. The prediction result is represented as (x, y, r, n). Through transformation, the information of the second prediction box can be obtained, represented as (x, y, w, h), where w represents the width of the second prediction box.

[0062] In one optional embodiment, obtaining the multi-target tracking model includes: obtaining historical video information; determining historical target location information and historical target identification information in the historical video information, and using the historical target location information and historical target identification information as a multi-target training set; and obtaining the multi-target tracking model based on the multi-target training set and employing a multi-target tracking algorithm.

[0063] It is understandable that the above multi-target tracking model is trained based on historical video information using a multi-target tracking algorithm.

[0064] It should be noted that the type and complexity of the targets in the historical video information, as well as the number of training iterations, all affect the tracking performance of the multi-target tracking model. If the target type in the historical video information is a pedestrian, then the first target that the multi-target tracking model can track and identify will also be a pedestrian.

[0065] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation method. Figure 2 A flowchart of another optional multi-target trajectory tracking method provided by an embodiment of the present invention is shown below. Figure 2 The specific steps are as follows:

[0066] Step 1: Multi-target tracking model training. From the current frame video image in the video information to be detected, the positions and labels of multiple first tracking targets in the current frame video image are labeled to construct a dataset for training the multi-target tracking model. The tracking dataset of the first tracking targets constructed using a multi-target recognition algorithm is used to train the model, resulting in a multi-target recognition model.

[0067] Step 2: Multi-target tracking model result prediction. The multi-target recognition algorithm can simultaneously process the first tracking target with a historical trajectory and the newly emerging first tracking target. The target with a historical trajectory is designated as the fourth tracking target, and the target without a historical trajectory is designated as the third tracking target. The newly emerging first tracking target (fourth tracking target) will have its tracking trajectory created in the next frame. The decoder of the multi-target recognition algorithm uses the input query key to obtain the historical trajectory tracking results, performs attention calculations, and predicts the detection and prediction results of the first tracking target, corresponding to the first detection box and the first prediction box. The query key consists of the embedding vector for predicting the tracking result of the first tracking target and the vector used to predict the detection result of the first tracking target. The embedding vector used to predict the tracking result of the first tracking target is obtained from the detection and prediction results of the second tracking target (corresponding to the historical detection box and historical prediction box) output by the decoder in the previous frame of the video image, and the corresponding feature vector input from the previous layer network. The vector used for the detection result of the third tracking target (first detection box) is constructed through random initialization. The prediction result of the multi-target recognition model can be expressed as:

[0068] N = N object +N track

[0069] Where, N object N represents the first detection box. track Indicates the first prediction box. N object The subsequent tracking trajectory will be created in the next frame, and the subsequent trajectory tracking results will be predicted by a multi-target recognition model. The position information of the first tracking target is described using (x, y, w, h), where (x, y) represents the center position coordinates of the first predicted box corresponding to the first tracking target, w is the width of the first predicted box corresponding to the first tracking target, and h represents the height of the first predicted box corresponding to the first tracking target.

[0070] Step 3, Motion Estimation: The historical trajectory tracking results are input into the Kalman filter algorithm to continuously predict the position of the first tracked target in the next frame. The motion state of the first tracked target is described using a state space (x, y, r, h, Δx, Δy, Δr, Δh). Here, (x, y) represents the target center coordinates, r is the aspect ratio of the second prediction box, h represents the height of the second prediction box, and Δx, Δy, Δr, and Δh are the time derivatives of x, y, r, and h in the image coordinate system, respectively. The Kalman filter algorithm predicts the target's motion state in subsequent frames of the video information to be detected, and the prediction result is represented as (x, y, r, h). Through transformation, the information of the second prediction box can be obtained, represented as (x, y, w, h), where w represents the width of the second prediction box.

[0071] Step 4, Tracking Result Correction: For each constructed tracking trajectory, calculate the first intersection-union ratio (IUR) of the first predicted bounding box obtained by the multi-target recognition model and the second predicted bounding box obtained by the Kalman filter algorithm. The calculation formula can be expressed as:

[0072]

[0073] Where A represents the first predicted bounding box output by the multi-target tracking model, B represents the second predicted bounding box output by the Kalman filter algorithm, A∩B represents the area of ​​the union of A and B, and A∪B represents the area of ​​the intersection of A and B. IOU1 is the first intersection-union ratio. A first preset threshold is set to 1. When the first intersection-union ratio is less than 1, it is considered that the first predicted bounding box obtained by the multi-target tracking model has completed correct tracking in the current frame. Based on the first and second predicted bounding boxes, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0074] In real-world scenarios, target loss may occur. Trajectory correction is performed on the fourth tracked target that was not correctly tracked. The second intersection-union ratio (IUR) is obtained based on the second predicted bounding box of the fourth tracked target and the first detected bounding box of the third tracked target. C represents the second predicted bounding box output by the Kalman filter algorithm, D represents the first detected bounding box output by the multi-target tracking model, C∩D represents the area of ​​the union of C and D, and C∪D represents the area of ​​the intersection of C and D. The second IUR is obtained as follows: Here, IOU2 represents the second intersection-union ratio. A second preset threshold is set to 0.3. When the second intersection-union ratio is less than 0.3, it is considered that a correct tracking result for the fourth tracking target has been found among the third tracking targets, and correction is required. The first detection box corresponding to the third tracking target is associated with the first predicted box corresponding to the fourth tracking target as an update to the first predicted box. Based on the first detection box and the updated first predicted box of the fourth tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

[0075] In practical applications, there are situations where tracking is lost and cannot be corrected, requiring re-tracking. If the second intersection-union ratio (IU) is greater than or equal to the second preset threshold, it is considered that the correct tracking result for the fourth tracking target was not found in the third updated target, and the existing tracking trajectory cannot continue, requiring re-tracking. When the second IU is greater than or equal to 0.3, the tracking trajectory of the fourth tracking target will end, and a new corresponding tracking trajectory will be established in the next frame.

[0076] The above optional implementation methods achieve at least one of the following effects: A multi-target tracking model is constructed using a highly efficient multi-target tracking algorithm. While the multi-target tracking model predicts the target tracking result, a Kalman filter algorithm is used to predict the target trajectory's position in the next frame. This adds motion features to the end-to-end tracking method of the multi-target tracking model, which is beneficial for improving tracking capabilities. The target tracking information predicted by the Kalman filter algorithm is used to correct the predicted target tracking result by the multi-target tracking model. While ensuring target tracking efficiency, the accuracy of multi-target tracking is improved, and the frequent switching of tracking trajectories is reduced.

[0077] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0078] This embodiment also provides a multi-target trajectory tracking device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module" and "device" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0079] According to embodiments of the present invention, an apparatus embodiment for implementing a multi-target trajectory tracking method is also provided. Figure 3 This is a schematic diagram of a multi-target trajectory tracking device according to an embodiment of the present invention, such as... Figure 3 As shown, the above-mentioned multi-target trajectory tracking device includes: a first acquisition module 302, a first generation module 304, a second generation module 306, and a third generation module 308. The device will be described below.

[0080] The first acquisition module 302 is used to acquire multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and the historical trajectory tracking result corresponding to the previous frame video image of the current frame video image.

[0081] The first generation module 304 is connected to the first acquisition module 302 and is used to process the multiple first tracking targets according to the above historical trajectory tracking results using a multi-target tracking model to obtain the first prediction boxes corresponding to the multiple first tracking targets respectively.

[0082] The second generation module 306 is connected to the first generation module 304 and is used to process the multiple first tracking targets using the Kalman filter algorithm based on the above historical trajectory tracking results to obtain the second prediction boxes corresponding to the multiple first tracking targets respectively.

[0083] The third generation module 308, connected to the second generation module 306, is used to obtain the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box.

[0084] In a multi-target trajectory tracking device provided by this invention, a first acquisition module 302 is configured to acquire multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and the historical trajectory tracking results corresponding to the previous frame video image of the current frame video image; a first generation module 304, connected to the first acquisition module 302, is configured to process the multiple first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the multiple first tracking targets respectively; a second generation module 306, connected to the first generation module 304, is configured to process the multiple first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the multiple first tracking targets respectively; and a third generation module 308, connected to the second generation module 306, is configured to obtain target trajectory tracking results corresponding to the multiple first tracking targets based on the first prediction boxes and the second prediction boxes. This achieves the goal of combining multiple algorithms to improve dynamic tracking capabilities, realizing the technical effects of improving tracking accuracy and tracking efficiency, and thus solving the technical problems of low tracking efficiency and poor recognition ability in multi-target tracking scenarios in related technologies.

[0085] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0086] It should be noted that the first acquisition module 302, the first generation module 304, the second generation module 306, and the third generation module 308 mentioned above correspond to steps S102 to S108 in the embodiments. The instances and application scenarios implemented by the above modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run on a computer terminal.

[0087] It should be noted that the optional or preferred implementation methods of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0088] The aforementioned multi-target trajectory tracking device may further include a processor and a memory. The first acquisition module 302, the first generation module 304, the second generation module 306, the third generation module 308, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0089] The processor contains a core that retrieves the corresponding program unit from memory. One or more cores may be configured. Memory may include non-persistent memory in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip.

[0090] This invention provides a non-volatile storage medium storing a program that, when executed by a processor, implements a multi-target tracking method.

[0091] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and historical trajectory tracking results corresponding to the previous frame video image; processing the multiple first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to each of the multiple first tracking targets; processing the multiple first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to each of the multiple first tracking targets; and obtaining target trajectory tracking results corresponding to each of the multiple first tracking targets based on the first prediction boxes and the second prediction boxes. The device in this document can be a server, PC, etc.

[0092] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: acquiring multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and historical trajectory tracking results corresponding to the previous frame video image of the current frame video image; processing the multiple first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the multiple first tracking targets respectively; processing the multiple first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the multiple first tracking targets respectively; and obtaining target trajectory tracking results corresponding to the multiple first tracking targets based on the first prediction boxes and the second prediction boxes respectively.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0099] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0100] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0101] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-target trajectory tracking method, characterized in that, include: Obtain multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and the historical trajectory tracking results corresponding to the previous frame video image of the current frame video image; Based on the historical trajectory tracking results, a multi-target tracking model is used to process the multiple first tracking targets to obtain the first prediction boxes corresponding to the multiple first tracking targets respectively; Based on the historical trajectory tracking results, the Kalman filter algorithm is used to process the multiple first tracking targets to obtain the second prediction boxes corresponding to the multiple first tracking targets respectively; Based on the first prediction box and the second prediction box, the target trajectory tracking results corresponding to the plurality of first tracking targets are obtained respectively; The step of obtaining target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box includes: determining whether the plurality of first tracking targets have historical trajectories based on the historical trajectory tracking results; if there is a first tracking target among the plurality of first tracking targets that does not have a historical trajectory, then designating the first tracking target among the plurality of first tracking targets that does not have a historical trajectory as a third tracking target, and designating the first tracking targets among the plurality of first tracking targets other than the third tracking target as a fourth tracking target; processing the third tracking target using the multi-target tracking model to generate a first detection box corresponding to the third tracking target; and processing the fourth tracking target using the multi-target tracking model. The system generates a first prediction box corresponding to the fourth tracking target; processes the first prediction box and the second prediction box corresponding to the fourth tracking target to obtain a first intersection-union ratio (IU / R) corresponding to the fourth tracking target; if the first IU / R is greater than or equal to a first preset threshold, processes the second prediction box of the fourth tracking target and the first detection box of the third tracking target to obtain a second IU / R corresponding to the fourth tracking target; if the second IU / R is less than a second preset threshold, updates the first prediction box of the fourth tracking target using the first detection box of the third tracking target, and obtains the target trajectory tracking result corresponding to the fourth tracking target based on the first detection box of the third tracking target and the updated first prediction box of the fourth tracking target.

2. The method according to claim 1, characterized in that, The step of obtaining the historical trajectory tracking result corresponding to the previous frame of the current frame video image includes: Obtain multiple second tracking targets corresponding to the previous frame video image, and historical detection boxes and historical prediction boxes corresponding to the multiple second tracking targets respectively, wherein the historical detection boxes correspond to second tracking targets that have not generated historical trajectories, and the historical prediction boxes correspond to second tracking targets that have generated historical trajectories; The historical detection boxes are transformed, and the historical trajectory tracking results are generated based on the transformed historical detection boxes and historical prediction boxes.

3. The method according to claim 1, characterized in that, The step of processing the plurality of first tracking targets using a multi-target tracking model based on the historical trajectory tracking results to obtain first prediction boxes corresponding to the plurality of first tracking targets includes: Based on the historical trajectory tracking results, it is determined whether each of the plurality of first tracking targets has a historical trajectory; If all of the multiple first tracking targets have the historical trajectory, then the multi-target tracking model is used to process the multiple first tracking targets to generate the first prediction box corresponding to each of the multiple first tracking targets.

4. The method according to claim 3, characterized in that, When all of the plurality of first tracking targets have the historical trajectory, the step of obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box includes: using the first prediction box as the target trajectory tracking results corresponding to the plurality of first tracking targets respectively; If, among the plurality of first tracking targets, there is a third tracking target that does not have the historical trajectory, obtaining the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box includes: obtaining the target trajectory tracking results corresponding to the fourth tracking target (excluding the third tracking target) among the plurality of first tracking targets based on the first prediction box and the second prediction box.

5. The method according to claim 4, characterized in that, The step of obtaining the target trajectory tracking result corresponding to the fourth tracking target (excluding the third tracking target) among the plurality of first tracking targets based on the first prediction box and the second prediction box includes: The first prediction box and the second prediction box corresponding to the fourth tracking target are processed to obtain the first intersection-union ratio (IU) corresponding to the fourth tracking target. Determine whether the first intersection-union ratio is less than a first preset threshold; If the first intersection-union ratio is less than the first preset threshold, then based on the first prediction box and the second prediction box of the fourth tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

6. The method according to claim 1, characterized in that, The method further includes: If the second crossover ratio is greater than or equal to the second preset threshold, the first detection box of the third tracking target is updated using the first prediction box of the fourth tracking target; Based on the first detection box of the updated third tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

7. The method according to claim 1, characterized in that, The step of processing the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results to obtain second prediction boxes corresponding to the plurality of first tracking targets includes: Based on the historical trajectory tracking results, the Kalman filter algorithm is used to process the multiple first tracking targets to obtain the motion state information corresponding to the multiple first tracking targets respectively; Based on the motion state information, the second prediction boxes corresponding to the plurality of first tracking targets are obtained respectively.

8. The method according to any one of claims 1 to 7, characterized in that, The process of obtaining the multi-target tracking model includes: Retrieve historical video information; Determine the historical target location information and historical target identification information in the historical video information, and use the historical target location information and historical target identification information as a multi-target training set; Based on the multi-target training set, a multi-target tracking algorithm is used to obtain the multi-target tracking model.

9. A multi-target trajectory tracking device, characterized in that, include: The first acquisition module is used to acquire multiple first tracking targets corresponding to the current frame video image in the video information to be detected, and the historical trajectory tracking result corresponding to the previous frame video image of the current frame video image; The first generation module is used to process the plurality of first tracking targets using a multi-target tracking model based on the historical trajectory tracking results, and obtain the first prediction boxes corresponding to the plurality of first tracking targets respectively; The second generation module is used to process the plurality of first tracking targets using a Kalman filter algorithm based on the historical trajectory tracking results, and obtain second prediction boxes corresponding to the plurality of first tracking targets respectively; The third generation module is used to obtain the target trajectory tracking results corresponding to the plurality of first tracking targets based on the first prediction box and the second prediction box; The third generation module is further configured to determine, based on the historical trajectory tracking results, whether each of the plurality of first tracking targets has a historical trajectory; if there is a first tracking target among the plurality of first tracking targets that does not have a historical trajectory, then the first tracking target among the plurality of first tracking targets that does not have a historical trajectory is designated as the third tracking target, and the first tracking targets among the plurality of first tracking targets other than the third tracking target are designated as the fourth tracking target; the multi-target tracking model is used to process the third tracking target to generate a first detection box corresponding to the third tracking target; the multi-target tracking model is used to process the fourth tracking target to generate a first prediction box corresponding to the fourth tracking target. The first predicted bounding box and the second predicted bounding box corresponding to the fourth tracking target are processed to obtain the first intersection-union ratio (IU / R) corresponding to the fourth tracking target. If the first IU / R is greater than or equal to a first preset threshold, the second predicted bounding box of the fourth tracking target and the first detection bounding box of the third tracking target are processed to obtain the second IU / R corresponding to the fourth tracking target. If the second IU / R is less than a second preset threshold, the first predicted bounding box of the fourth tracking target is updated using the first detection bounding box of the third tracking target. Based on the first detection bounding box of the third tracking target and the updated first predicted bounding box of the fourth tracking target, the target trajectory tracking result corresponding to the fourth tracking target is obtained.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions, which are adapted to be loaded by a processor and executed by the multi-target trajectory tracking method according to any one of claims 1 to 8.

Images