Target tracking method and apparatus

Abstract

The application provides a multi-target tracking method and device, which comprises the following steps: target detection is performed on a pre-acquired target tracking image to obtain a target detection result corresponding to the target tracking image, the target detection result comprising target detection boxes of multiple different target categories; a target tracker is used to perform target prediction on a currently tracked target tracking image to obtain a target prediction result, the target prediction result comprising target prediction boxes of multiple different target categories; and the target prediction boxes of the multiple different target categories and the target detection boxes of the multiple different target categories are associated and matched to realize multi-target tracking. The application performs target detection and target prediction on the target tracking image respectively, uniformly associates the target detection boxes of different categories obtained by detection and the target prediction boxes of different categories obtained by prediction, and can ensure that the time consumption of the matching algorithm is significantly reduced while the effect of time sequence association is not affected.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target tracking method and apparatus. Background Technology

[0002] With the widespread application of image processing in civilian and commercial fields, multi-target tracking plays an increasingly important role in areas such as intelligent video surveillance, autonomous driving, and unmanned supermarkets. Therefore, target tracking, especially multi-target tracking, faces higher requirements. In monocular ADAS systems, the accuracy and stability of target tracking (especially temporal stability) are crucial for upper-layer ranging, velocity measurement, and alarm logic. The attributes of target detection and tracking include at least the bounding box and grounding point. Actual ADAS system deployments involve not only vehicles but also the cloud. Especially for vehicles with low computing power, uploading a very short video clip (e.g., 10 seconds) to the cloud allows for computational results to be obtained with less time than on the vehicle, enabling updates and corrections to the vehicle's data.

[0003] However, current research on monocular ADAS on low-end computing devices in vehicles is relatively limited. Multi-object tracking research mainly focuses on online tracking within the tracking-by-detection framework, such as SORT and DeepSort. Additionally, there is a recent trend towards integrated detection and tracking, as seen in JDE and FairMoT. However, these studies are primarily focused on online single-class, single-attribute multi-object tracking, such as bounding box attributes for pedestrians, pose attributes for human bodies, and bounding box attributes for vehicles.

[0004] In addition, deep learning can be used for tracking, such as detection, appearance feature extraction, matching, and even end-to-end detection and tracking. However, deep learning relies on large-scale data, and the video stream annotation required for tracking is far more expensive than the bounding box annotation for object detection. Furthermore, its generalization ability is significantly weaker than SORT and similar methods when the training data does not cover the scene, and it does not have a built-in temporal filtering smoothing effect. Summary of the Invention

[0005] This invention provides a target tracking method and apparatus to address the shortcomings of poor accuracy and stability in multi-target tracking in the prior art, and can significantly improve the accuracy and temporal stability of target tracking.

[0006] This invention provides a target tracking method, comprising: performing target detection on a pre-acquired target tracking image to obtain a target detection result corresponding to the target tracking image, wherein the target detection result includes multiple target detection boxes of different target categories; performing target prediction on the currently tracked target tracking image using a multi-target tracker to obtain a target prediction result, wherein the target prediction result includes multiple target prediction boxes of different target categories; and associating and matching the multiple target prediction boxes of different target categories with the multiple target detection boxes of different target categories to achieve multi-target tracking.

[0007] According to a target tracking method provided by the present invention, the step of associating and matching multiple target prediction boxes and multiple target detection boxes of different target categories includes: obtaining corresponding metric values ​​based on each target prediction box and each target detection box; matching multiple target detection boxes and multiple target prediction boxes of different target categories in the target tracking image based on the metric values ​​and a preset association algorithm to obtain target matching results corresponding to each target category; and obtaining a target tracking result based on the target matching results being displayed as matching pairs, and the number of consecutive frames of the target tracking image corresponding to the matching pair meeting a preset number of frames.

[0008] According to a target tracking method provided by the present invention, after obtaining target matching results corresponding to each target category, the method includes: based on the target matching results being displayed as matching pairs, comparing the metric value with a preset threshold, and determining whether the categories of the target prediction box and the target detection box are consistent; if the metric value is greater than or equal to the preset threshold and the categories are consistent, then updating the multi-target tracker using the target detection results; otherwise, dissolving the pairing relationship of the corresponding matching pairs, updating the target matching result of the corresponding target prediction box to an unmatched prediction box, and updating the target matching result of the corresponding target detection box to an unmatched detection box.

[0009] According to a target tracking method provided by the present invention, after obtaining the target matching results corresponding to each target category, the method further includes: initializing the multi-target tracker based on the target matching results being displayed as unmatched detection boxes; and recording the number of consecutive unmatches based on the target matching results being displayed as unmatched prediction boxes.

[0010] According to a target tracking method provided by the present invention, the target detection result further includes detection ground points corresponding one-to-one with the target detection boxes of the plurality of different target categories. The step of updating the multi-target tracker using the target detection result includes: updating the multi-target tracker using target detection boxes that match the target prediction box and the detection ground points corresponding to the target detection boxes, so as to perform forward Kalman filtering on the target prediction result using the updated multi-target tracker.

[0011] According to a target tracking method provided by the present invention, when the pre-acquired target tracking image is an offline video in the cloud, after associating and matching the target prediction boxes of the plurality of different target categories and the target detection boxes of the plurality of different target categories, the method further includes: based on the target matching result being displayed as a matching pair, performing backward Kalman filtering on the target prediction results of each time point constituting the same trajectory.

[0012] According to a target tracking method provided by the present invention, the step of performing backward Kalman filtering on the target prediction results of each time point constituting the same trajectory based on the target matching results being displayed as matching pairs includes: performing optimal state estimation on the targets at each time point constituting the same trajectory based on the target matching results being displayed as matching pairs, and obtaining corresponding optimal estimation results; obtaining a Gaussian distribution sequence based on each time point and the optimal estimation results; and performing trajectory smoothing by iterating backward through the backward Kalman filter according to the Gaussian distribution sequence and the pre-acquired target detection boxes at each time point.

[0013] The present invention also provides a target tracking device, comprising: a target detection module, which performs target detection on a pre-acquired target tracking image to obtain a target detection result corresponding to the target tracking image, the target detection result including multiple target detection boxes of different target categories; a target prediction module, which uses a multi-target tracker to perform target prediction on the currently tracked target tracking image to obtain a target prediction result, the target prediction result including multiple target prediction boxes of different target categories; and a tracking module, which associates and matches the multiple target prediction boxes of different target categories with the multiple target detection boxes of different target categories to achieve multi-target tracking.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the target tracking methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the target tracking methods described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the target tracking methods described above.

[0017] The target tracking method and apparatus provided by this invention perform target detection and target prediction on the target tracking image respectively, and uniformly associate the detected target bounding boxes of different categories and the predicted target bounding boxes of different categories. This ensures that the time consumption of the matching algorithm is significantly reduced without affecting the effect of temporal association. In addition, by decomposing the attributes into two different functions of temporal association and temporal filtering, the interference caused by using the predicted ground point attributes for temporal association can be significantly avoided, and the ground point after online filtering can be obtained. Moreover, all categories share the same multi-target tracker to uniformly associate targets of different categories, which can significantly reduce the time consumption of the matching algorithm without affecting the effect of temporal association. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the target tracking method provided by the present invention;

[0020] Figure 2 This is the second flowchart of the target tracking method provided by the present invention;

[0021] Figure 3 This is a schematic diagram of the target tracking device provided by the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 A flowchart of a target tracking method according to the present invention is shown, the method comprising:

[0025] S11, Perform target detection on the pre-acquired target tracking image to obtain the target detection result of the corresponding target tracking image. The target detection result includes target detection boxes of multiple different target categories.

[0026] S12, use a multi-target tracker to predict the target tracking image of the currently tracked target and obtain the target prediction result, which includes target prediction boxes of multiple different target categories;

[0027] S13, associate and match multiple target prediction boxes of different target categories with multiple target detection boxes of different target categories to achieve multi-target tracking.

[0028] It should be noted that S1N in this specification does not represent the order of the target tracking methods. The following will explain in detail... Figure 2 The target tracking method of the present invention is described.

[0029] Step S11: Perform target detection on the pre-acquired target tracking image to obtain the target detection result of the corresponding target tracking image. The target detection result includes target detection boxes of multiple different target categories.

[0030] In this embodiment, target detection is performed on the pre-acquired target tracking image, including: using a multi-target detector to perform target detection on the current frame target tracking image to obtain target detection results. It should be noted that the target detection results include multiple target detection boxes for different target categories and the corresponding detection ground points for each target detection box. Targets include other vehicle targets and non-vehicle targets, such as pedestrians. It should be noted that some existing target detection models can be used for target detection on the target tracking image, and no further limitations are made here.

[0031] In one optional embodiment, before performing target detection on the pre-acquired target tracking image, the method includes: acquiring a cloud video frame stream; decoding the video frame stream; and sampling the decoded video frame stream at a preset frame rate to obtain the target tracking image. It should be noted that sampling at a preset frame rate is used to ensure sampling uniformity as much as possible.

[0032] In one optional embodiment, before performing target detection on the pre-acquired target tracking image, the method includes: acquiring a real-time image frame stream captured by the vehicle-mounted camera as the target tracking image. Furthermore, after acquiring the real-time image frame stream captured by the vehicle-mounted camera, the method further includes: sampling the real-time image frame stream based on a preset frame rate.

[0033] It should be noted that by acquiring video frame streams from the vehicle or the cloud, this method can be used for real-time vehicle tracking on low-end computing devices. Furthermore, by adding an algorithm module, it can significantly improve tracking accuracy and stability by leveraging the offline nature of cloud video with minimal computational cost. It can also be used for multi-target tracking schemes that track multiple attributes of multiple types of obstacles, which is very important for the practical application and promotion of monocular ADAS.

[0034] Step S12: Use a multi-target tracker to predict the target image of the currently tracked target and obtain the target prediction result, which includes target prediction boxes of multiple different target categories.

[0035] In this embodiment, target prediction is performed on the currently tracked target image using a multi-target tracker, including: using the multi-target tracker and combining historical target prediction data to perform target prediction on the currently tracked target image, and obtaining target prediction results; wherein, the target prediction results include multiple target prediction boxes of different target categories and prediction ground points corresponding to each target prediction box.

[0036] Step S13: Associate and match multiple target prediction boxes of different target categories with multiple target detection boxes of different target categories to achieve multi-target tracking.

[0037] In this embodiment, the association and matching of multiple target prediction boxes and multiple target detection boxes of different target categories includes: obtaining corresponding metric values ​​based on each target prediction box and each target detection box; matching multiple target detection boxes and multiple target prediction boxes of different target categories in the target tracking image based on the metric values ​​and a preset association algorithm to obtain target matching results for each target category; and obtaining target tracking results based on the target matching results being displayed as matching pairs, and the number of consecutive frames of the target tracking image corresponding to the matching pair meeting a preset number of frames.

[0038] In an optional embodiment, after obtaining the target matching results for each target category, the process includes: based on the target matching results being displayed as matching pairs, comparing the metric value with a preset threshold, and determining whether the categories of the target prediction box and the target detection box are consistent; if the metric value is greater than or equal to the preset threshold and the categories are consistent, then updating the multi-target tracker using the target detection results; otherwise, removing the pairing relationship of the corresponding matching pair, updating the target matching result of the corresponding target prediction box to an unmatched prediction box, and updating the target matching result of the corresponding target detection box to an unmatched detection box.

[0039] It should be noted that since obstacle targets have multiple categories, and all categories share a single multi-target tracker, removing matching pairs from different categories after unified temporal association significantly reduces the matching algorithm's overhead. Furthermore, in actual processing, the inter-frame motion distance of the same target is much smaller than the distance between obstacles of different categories, thus having virtually no impact on temporal association.

[0040] Furthermore, updating the multi-target tracker using target detection results includes updating the multi-target tracker using target detection boxes that match the target prediction boxes and the corresponding detection ground points, so that the updated multi-target tracker can be used to perform forward Kalman filtering on the target prediction results. It should be added that, since targets have multiple attributes, considering the purposes of tracking and subsequent velocity measurement, the roles of the target prediction boxes and prediction ground points are decomposed, and association matching is performed only on the target prediction boxes and target detection boxes. In addition, monocular ranging and velocity measurement heavily rely on the stability of the ground point timing; therefore, by updating the multi-target tracker using target detection results that match the target prediction boxes, temporal filtering (i.e., forward Kalman filtering) is performed on both the target prediction boxes and prediction ground points, so that the tracking is not hindered by the ground point in terms of temporal association, while still obtaining the temporally filtered ground point.

[0041] In another optional embodiment, after obtaining the target matching results for each target category, the method further includes: initializing a multi-target tracker based on the target matching results displaying unmatched detection boxes; and recording the number of consecutive unmatched instances based on the target matching results displaying unmatched prediction boxes. More specifically, if the number of matching pairs corresponding to consecutive frames of target tracking images meets a preset frame number, the corresponding target matching result can be determined as an output target tracking result; if the target matching results corresponding to a certain number of consecutive frames of target tracking images display unmatched detection boxes, these are used to initialize the multi-target tracker; if the target matching results corresponding to a certain number of consecutive frames of target tracking images display unmatched prediction boxes, the corresponding tracker is deleted, and reference is made. Figure 2 .

[0042] In addition, the multi-target tracker can use a forward Kalman filter. Before using the multi-target tracker to perform forward Kalman prediction on the tracking image of the currently tracked target, the following steps are also taken: parameter tuning of the forward Kalman filter, mainly including process noise Q, measurement noise R and initial state covariance matrix P.

[0043] It should be noted that, without annotation, the above three parameters are generally adjusted according to certain principles on a test set covering the expected scene, and the tracking effect is observed. Generally, P is a relatively large value to reflect greater uncertainty about the initial motion state, allowing for rapid adjustment to the correct motion state after the first few frames. Measurement noise R reflects the degree of trust in the detection in Bayesian fusion and can be adjusted based on the detector's accuracy. In reality, detector accuracy is affected by factors such as the distance of the target from the vehicle, lighting conditions, day / night cycles, rain, snow, and fog. Adaptively incorporating these factors, and even the detector's positioning accuracy score, into R can significantly improve the filtering effect. Process noise Q reflects the degree of trust in motion modeling in Bayesian fusion. The smaller the value of Q, the less uncertainty there is in motion modeling, and the more stable the filtered bounding boxes are in time. However, linear filtering inevitably has an over-shooting problem for sudden changes in motion state. In practice, factors affecting the accuracy of motion modeling include the uniformity of the sampling frames and the sampling frame rate. For example, the higher the sampling frame rate, the smaller the time difference between frames, and the higher the accuracy of approximating with uniform motion. Therefore, the value of Q can be adjusted to be smaller accordingly.

[0044] In one alternative embodiment, the forward Kalman filter can use a uniform motion model, which is approximately accurate enough when the sampling frame rate is high enough. Although there is an over-shooting problem when the motion state changes suddenly, it will gradually adjust to be correct after the motion stabilizes. If an accelerated model is used, although it responds faster to changes in motion state, it is easy to incorporate noise into the acceleration, which is reflected in the second order in the prediction, resulting in very large fluctuations and instability.

[0045] In an optional embodiment, when the pre-acquired target tracking image is an offline video from the cloud, after associating and matching multiple target prediction boxes and multiple target detection boxes of different target categories, the method further includes: displaying the target matching results as matching pairs, and performing backward Kalman filtering on the target prediction results of each time-series point constituting the same trajectory. It should be noted that in offline scenarios, by smoothly decomposing the RTS (Rauch-Tung-Striebel) into forward Kalman filtering and backward Kalman filtering, only backward Kalman filtering can be used on top of online tracking to achieve a smoothing effect, thereby improving the accuracy and temporal stability of the acquired target tracking results.

[0046] Furthermore, theoretically, performing backward Kalman filtering on the target prediction results that have undergone forward Kalman filtering is equivalent to performing RTS smoothing on each trajectory segment. That is, the state of each point in the trajectory is optimally estimated using the target detection boxes of all time points on the entire trajectory. In practice, since the temporal correlation of tracking is still online, the temporal filtering of tracking has changed from online filtering to offline smoothing. Online filtering and offline smoothing add future temporal target detection boxes to the current and previous temporal target detection boxes to optimally estimate the state at the current moment, thus solving the over-shooting problem of online filtering.

[0047] Given prior and current target detection boxes, online filtering can only predict the current state based on the motion trend reflected in the prior detection boxes. If the target motion changes in the target tracking image, the prediction will be inaccurate. However, changes in target motion in the target tracking image will be reflected in future data. With past, present, and future target detection boxes, offline RTS smoothing can accurately estimate the current state, even under changing motion conditions. By adding RTS smoothing, the offline nature of the cloud is utilized to provide a more time-stable state estimate without overshooting issues, resulting in more stable and accurate boxes and grounding points. This eliminates the need to redesign offline multi-class, multi-attribute, and multi-target tracking schemes. This improvement alone is evident in the FCW and HMW of monocular ADAS. Fine-tuning the alarm parameters improves the HMW alarm accuracy by 8 percentage points.

[0048] Furthermore, based on the target matching results showing as matching pairs, backward Kalman filtering is performed on the target prediction results of each time point constituting the same trajectory. This includes: based on the target matching results showing as matching pairs, performing optimal state estimation on the targets at each time point constituting the same trajectory to obtain the corresponding optimal estimation results; obtaining a Gaussian distribution sequence based on each time point and the optimal estimation results; and iterating backward through the backward Kalman filter according to the Gaussian distribution sequence and the pre-acquired target detection boxes at each time point to perform trajectory smoothing.

[0049] In summary, the embodiments of the present invention perform target detection and target prediction on the target tracking image separately, and uniformly associate the detected target bounding boxes of different categories and the predicted target bounding boxes of different categories. This ensures that the time consumption of the matching algorithm is significantly reduced without affecting the effect of temporal association. In addition, by decomposing the attributes into two different functions of temporal association and temporal filtering, the interference caused by using the predicted ground point attributes for temporal association can be significantly avoided, while obtaining the ground point after online filtering. Moreover, all categories share the same multi-target tracker to uniformly associate targets of different categories, which can significantly reduce the time consumption of the matching algorithm without affecting the effect of temporal association.

[0050] The target tracking device provided by the present invention is described below. The target tracking device described below can be referred to in correspondence with the target tracking method described above.

[0051] Figure 3 A schematic diagram of a target tracking device is shown, the device comprising:

[0052] The target detection module 31 performs target detection on the pre-acquired target tracking image to obtain the target detection result of the corresponding target tracking image. The target detection result includes target detection boxes of multiple different target categories.

[0053] The target prediction module 32 uses a multi-target tracker to predict the target of the currently tracked second target tracking image and obtains the target prediction result, which includes target prediction boxes of multiple different target categories.

[0054] The tracking module 33 associates and matches multiple target prediction boxes of different target categories with multiple target detection boxes of different target categories to achieve multi-target tracking.

[0055] In this embodiment, the target detection module 31 includes a target detection unit that uses a multi-target detector to perform target detection on the target tracking image of the current frame to obtain target detection results. It should be noted that the target detection results include multiple target detection boxes of different target categories and the detection ground points corresponding to each target detection box. Targets include other vehicle targets and non-vehicle targets, such as pedestrians.

[0056] The image acquisition module includes: a video frame stream acquisition unit, which acquires a cloud video frame stream before performing target detection on a pre-acquired target tracking image; a decoding unit, which decodes the video frame stream; and a sampling unit, which samples the decoded video frame stream at a preset frame rate to obtain the target tracking image.

[0057] In one optional embodiment, the image acquisition module includes: an image acquisition unit, which acquires a real-time image frame stream captured by a camera on the vehicle side as a target tracking image before performing target detection on a pre-acquired target tracking image. More specifically, the image acquisition module further includes: a sampling unit, which samples the real-time acquired image frame stream based on a preset frame rate.

[0058] In this embodiment, the target prediction module 32 includes: a target prediction unit, which uses a multi-target tracker and combines historical target prediction data to perform target prediction on the currently tracked target tracking image to obtain a target prediction result; wherein, the target prediction result includes multiple target prediction boxes of different target categories and prediction ground points corresponding to each target prediction box.

[0059] In this embodiment, the tracking module 33 includes: a measurement unit, which obtains corresponding measurement values ​​based on each target prediction box and each target detection box; a matching unit, which matches multiple target detection boxes and multiple target prediction boxes of different target categories in the target tracking image based on the measurement values ​​and a preset association algorithm, to obtain target matching results corresponding to each target category; and a target tracking unit, which obtains target tracking results based on the target matching results being displayed as matching pairs and the number of consecutive frames of the target tracking image corresponding to the matching pairs meeting a preset number of frames.

[0060] In an optional embodiment, the tracking module 33 further includes: a comparison and judgment unit, which compares the metric value with a preset threshold based on the target matching result being displayed as a matching pair, and determines whether the categories of the target prediction box and the target detection box are consistent; an update unit, which updates the multi-target tracker using the target detection result if the metric value is greater than or equal to the preset threshold and the categories are consistent; otherwise, a pairing removal unit, which removes the pairing relationship of the corresponding matching pair, updates the target matching result of the corresponding target prediction box to an unmatched prediction box, and updates the target matching result of the corresponding target detection box to an unmatched detection box.

[0061] Furthermore, the update unit includes: an update subunit that updates the multi-target tracker using the target detection box that matches the target prediction box and the detection ground point corresponding to the target detection box, so as to perform forward Kalman filtering on the target prediction result using the updated multi-target tracker.

[0062] In another optional embodiment, the tracking module 33 further includes: an initialization unit, which initializes a multi-target tracker based on the target matching result being displayed as an unmatched detection box; and a data recording unit, which records the number of consecutive unmatched occurrences based on the target matching result being displayed as an unmatched prediction box. More specifically, the tracking module 33 further includes: a target tracking result determination unit, which determines whether there are consecutive frames of target tracking images where the number of matching pairs matches a preset frame number; if so, it determines the corresponding target matching result as an output target tracking result; and a tracker deletion unit, which initializes the multi-target tracker if there are consecutive frames of target tracking images where the target matching result is displayed as an unmatched detection box; and deletes the corresponding tracker if there are consecutive frames of target tracking images where the target matching result is displayed as an unmatched prediction box.

[0063] In an optional embodiment, the device further includes a parameter tuning module for tuning the forward Kalman filter, which mainly includes process noise Q, measurement noise R, and initial state covariance matrix P.

[0064] It should be noted that, without annotation, the above three parameters are generally adjusted according to certain principles on a test set covering the expected scene, and the tracking effect is observed. Generally, P is a relatively large value to reflect greater uncertainty about the initial motion state, allowing for rapid adjustment to the correct motion state after the first few frames. Measurement noise R reflects the degree of trust in the detection in Bayesian fusion and can be adjusted based on the detector's accuracy. In reality, detector accuracy is affected by factors such as the distance of the target from the vehicle, lighting conditions, day / night cycles, rain, snow, and fog. Adaptively incorporating these factors, and even the detector's positioning accuracy score, into R can significantly improve the filtering effect. Process noise Q reflects the degree of trust in motion modeling in Bayesian fusion. The smaller the value of Q, the less uncertainty there is in motion modeling, and the more stable the filtered bounding boxes are in time. However, linear filtering inevitably has an over-shooting problem for sudden changes in motion state. In practice, factors affecting the accuracy of motion modeling include the uniformity of the sampling frames and the sampling frame rate. For example, the higher the sampling frame rate, the smaller the time difference between frames, and the higher the accuracy of approximating with uniform motion. Therefore, the value of Q can be adjusted to be smaller accordingly.

[0065] In an optional embodiment, when the acquired target tracking image is an offline video in the cloud, the device further includes: a backward Kalman filter module, which, after associating and matching multiple target prediction boxes of different target categories and multiple target detection boxes of different target categories, displays the target matching results as paired, and performs backward Kalman filtering on the target detection results of each time point constituting the same trajectory.

[0066] Furthermore, the backward Kalman filter module includes: an optimal estimation unit, which, based on the target matching results displayed as matching pairs, performs optimal state estimation on the targets at each time point constituting the same trajectory to obtain the corresponding optimal estimation results; a Gaussian distribution sequence acquisition unit, which obtains a Gaussian distribution sequence based on each time point and the optimal estimation results; and a target tracking unit, which iterates backward into the backward Kalman filter according to the Gaussian distribution sequence and the pre-acquired target detection boxes at each time point to perform trajectory smoothing.

[0067] This invention, through target detection and prediction of target tracking images, unifies the association between detected bounding boxes and predicted bounding boxes of different categories, ensuring that the time consumption of the matching algorithm is significantly reduced without affecting the temporal association effect. Furthermore, by decomposing attributes into two different functions—temporal association and temporal filtering—interference caused by using predicted grounding point attributes for temporal association is significantly avoided, while still obtaining grounding points after online filtering. All categories share the same multi-target tracker for unified association of targets of different categories, significantly reducing the time consumption of the matching algorithm without affecting the temporal association effect.

[0068] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 41, a communications interface 42, a memory 43, and a communication bus 44, wherein the processor 41, the communications interface 42, and the memory 43 communicate with each other via the communication bus 44. The processor 41 can call logical instructions in the memory 43 to execute a target tracking method, which includes: performing target detection on a pre-acquired target tracking image to obtain a target detection result for the corresponding target tracking image, the target detection result including multiple target detection boxes of different target categories; using a multi-target tracker to perform target prediction on the currently tracked target tracking image to obtain a target prediction result, the target prediction result including multiple target prediction boxes of different target categories; and associating and matching the multiple target prediction boxes of different target categories with the multiple target detection boxes of different target categories to achieve multi-target tracking.

[0069] Furthermore, the logical instructions in the aforementioned memory 43 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target tracking method provided by the above methods. The method includes: performing target detection on a pre-acquired target tracking image to obtain a target detection result for the corresponding target tracking image, the target detection result including multiple target detection boxes of different target categories; using a multi-target tracker to perform target prediction on the currently tracked target tracking image to obtain a target prediction result, the target prediction result including multiple target prediction boxes of different target categories; and associating and matching the multiple target prediction boxes of different target categories with the multiple target detection boxes of different target categories to achieve multi-target tracking.

[0071] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target tracking method provided by the above methods. The method includes: performing target detection on a pre-acquired target tracking image to obtain a target detection result for the corresponding target tracking image, the target detection result including multiple target detection boxes of different target categories; performing target prediction on the currently tracked target tracking image using a multi-target tracker to obtain a target prediction result, the target prediction result including multiple target prediction boxes of different target categories; and associating and matching the multiple target prediction boxes of different target categories with the multiple target detection boxes of different target categories to achieve multi-target tracking.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target tracking method, characterized in that, include: Target detection is performed on a pre-acquired target tracking image to obtain a target detection result corresponding to the target tracking image. The target detection result includes multiple target detection boxes of different target categories. A multi-target tracker is used to predict the target in the currently tracked target image to obtain a target prediction result, which includes multiple target prediction boxes of different target categories. Multiple target prediction boxes and multiple target detection boxes of different target categories are associated and matched to achieve multi-target tracking; The step of associating and matching the predicted bounding boxes of the plurality of different target categories with the detected bounding boxes of the plurality of different target categories includes: Based on each of the target prediction boxes and each of the target detection boxes, the corresponding metric values ​​are obtained; Based on the metric value and a preset association algorithm, the target detection boxes and target prediction boxes of multiple different target categories in the target tracking image are matched to obtain the target matching results for each target category. The target tracking result is obtained based on the target matching result being displayed as a matching pair, and the number of consecutive frames of the target tracking image corresponding to the matching pair meeting the preset number of frames; After obtaining the target matching results corresponding to each target category, the process includes: Based on the target matching result being displayed as a matching pair, the metric value is compared with a preset threshold, and it is determined whether the categories of the target prediction box and the target detection box are consistent. If the metric value is greater than or equal to the preset threshold and the categories are consistent, the multi-target tracker is updated using the target detection result; otherwise, the pairing relationship of the corresponding matching pair is terminated, and the target matching result of the corresponding target prediction box is updated to an unmatched prediction box, and the target matching result of the corresponding target detection box is updated to an unmatched detection box. The target detection results also include detection ground points corresponding one-to-one with the target detection boxes of the multiple different target categories. Updating the multi-target tracker using the target detection results includes: The multi-target tracker is updated using the target detection box that matches the target prediction box and the detection ground point corresponding to the target detection box, so that the target prediction result is subjected to forward Kalman filtering using the updated multi-target tracker.

2. The target tracking method according to claim 1, characterized in that, After obtaining the target matching results for each target category, the process also includes: Based on the target matching result being displayed as a non-matching detection box, the multi-target tracker is initialized; Based on the target matching result, which is displayed as a non-matching prediction box, the number of consecutive non-matches is recorded.

3. The target tracking method according to claim 1, characterized in that, When the pre-acquired target tracking image is an offline video from the cloud, after associating and matching the target prediction boxes and target detection boxes of the multiple different target categories, the method further includes: Based on the target matching results displayed as matching pairs, backward Kalman filtering is performed on the target prediction results of each time point constituting the same trajectory.

4. The target tracking method according to claim 3, characterized in that, The step of displaying the target matching results as matching pairs and performing backward Kalman filtering on the target prediction results of each time point constituting the same trajectory includes: Based on the target matching results displayed as matching pairs, the optimal state estimation is performed on the targets at each time point that constitute the same trajectory to obtain the corresponding optimal estimation results; Based on each time point and the optimal estimation results, a Gaussian distribution sequence is obtained; Based on the Gaussian distribution sequence and the target detection boxes at each time point obtained in advance, the back-stepping Kalman filter is iterated in reverse to perform trajectory smoothing.

5. A target tracking device, characterized in that, include: The target detection module performs target detection on the pre-acquired target tracking image to obtain the target detection result corresponding to the target tracking image. The target detection result includes multiple target detection boxes of different target categories. The target prediction module uses a multi-target tracker to predict the target tracking image of the currently tracked target and obtains the target prediction result, which includes multiple target prediction boxes of different target categories. The tracking module associates and matches the predicted bounding boxes of multiple target categories with the detected bounding boxes of multiple target categories to achieve multi-target tracking; The tracking module includes: The metric unit obtains the corresponding metric value based on each of the target prediction boxes and each of the target detection boxes; The matching unit, based on the metric value and a preset association algorithm, matches multiple target detection boxes and multiple target prediction boxes of different target categories in the target tracking image to obtain target matching results for each target category. The target tracking unit obtains the target tracking result based on the target matching result being displayed as a matching pair, and the number of consecutive frames of the target tracking image corresponding to the matching pair meeting the preset number of frames. The tracking module further includes: The comparison and judgment unit, after obtaining the target matching results corresponding to each target category, displays the target matching results as matching pairs, compares the metric value with a preset threshold, and determines whether the categories of the target prediction box and the target detection box are consistent. The update unit updates the multi-target tracker using the target detection result if the metric value is greater than or equal to the preset threshold and the categories are consistent; otherwise, it removes the pairing relationship of the corresponding matching pair, updates the target matching result of the corresponding target prediction box to the unmatched prediction box, and updates the target matching result of the corresponding target detection box to the unmatched detection box. The target detection result also includes detection ground points corresponding one-to-one with the target detection boxes of the multiple different target categories, and the update unit includes: The update subunit updates the multi-target tracker using the target detection box that matches the target prediction box and the detection ground point corresponding to the target detection box, so as to use the updated multi-target tracker to perform forward Kalman filtering on the target prediction result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target tracking method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target tracking method as described in any one of claims 1 to 4.

Images