Target tracking method, target tracking system, and electronic device

Abstract

A target tracking method, a target tracking system and an electronic device are disclosed. The target tracking method comprises: sequentially obtaining N frames of images. N is greater than or equal to 2 and is an integer. The target tracking method further comprises: performing target detection on the Nth frame of image to generate a target region; performing optical flow calculation and Kalman filtering prediction on the target region of the (N-1)th frame of image to respectively generate an optical flow predicted target region and a Kalman filtering predicted target region of the Nth frame of image; performing first matching on the target region and the optical flow predicted target region of the Nth frame of image; if the two do not match, performing second matching on the target region and the Kalman filtering predicted target region of the Nth frame of image which do not match; determining a prediction condition for the Kalman filtering predicted target region which does not match; if the prediction condition is met, updating the Kalman filtering model and performing optical flow point resampling by using the optical flow predicted target region of the Nth frame of image corresponding to the Kalman filtering predicted target region.

Description

Technical Field

[0001] This invention relates to the field of video surveillance, and more specifically, to a target tracking method, a target tracking system, and an electronic device. Background Technology

[0002] In the field of video surveillance, target tracking is an indispensable and crucial foundational technology for intelligent recognition applications. The quality of tracking directly impacts the accuracy of intelligent recognition; conversely, stable tracking can reduce the consumption of redundant computational resources. Target tracking includes multi-target tracking and single-target tracking. For example... Figure 1a and Figure 1b As shown, Figure 1a and Figure 1b The diagram illustrates multi-target tracking across two frames.

[0003] Multi-object tracking is applied in almost every aspect of the vision field, such as security, autonomous driving, and healthcare. In security, we often need tracking to count the number of people in a specific area. In autonomous driving, we often need tracking to predict the trajectories of pedestrians or vehicles. In healthcare, we often need tracking to obtain information about cell movement.

[0004] Among existing technologies, SORT (Simple Online and Realtime Tracking) is a commonly used target tracking algorithm. The SORT tracking algorithm uses Faster R-CNN to detect targets in video sequences, uses a Kalman filter to predict the possible location of the target in the next frame, and employs the Hungarian method to calculate the cost matrix of Intersection Over Union (IOU) between the bounding box of each newly detected target and all predicted bounding boxes of targets in the existing trajectory, thus achieving data association. IOU is the ratio of the intersection to the union of the predicted bounding boxes and the ground truth bounding boxes of the target, i.e., the overlap, used to evaluate the accuracy of object localization. The IOU algorithm is used to calculate the IOU.

[0005] The SORT tracking algorithm is widely used due to its low complexity, simple framework, and fast execution speed. However, its reliance on IOU matching between consecutive frames and its heavy dependence on detection results make it prone to tracking ID number jumps due to missed detections, thus hindering effective tracking. Furthermore, its reliance on IOU matching also makes it susceptible to tracking object ID swapping.

[0006] Target tracking also includes sparse optical flow, which is specifically designed for image registration of sparse points in an image. This involves finding the corresponding points in the current image for a given set of points (usually corner points) in a reference image. Due to its relatively low computational cost, it is frequently used for both image registration and target tracking. Summary of the Invention

[0007] Embodiments of this disclosure provide a target tracking method, a target tracking system, and an electronic device.

[0008] In a first aspect, embodiments of this disclosure provide a target tracking method, comprising: sequentially acquiring N frames of images, wherein N is greater than or equal to 2 and N is an integer, wherein the target tracking method further comprises: performing target detection on the Nth frame image using a target detection algorithm to generate a target region; performing optical flow calculation on the target region of the (N-1)th frame image to generate an optical flow predicted target region of the Nth frame image; performing Kalman filter prediction on the target region of the (N-1)th frame image to generate a Kalman filter predicted target region of the Nth frame image; performing a first matching between the target region of the Nth frame image and the optical flow predicted target region of the Nth frame image; in response to the target region of the Nth frame image matching the optical flow predicted target region of the Nth frame image, updating the Kalman filter model and performing optical flow point resampling for the target region of the Nth frame image that passed the first matching; in response to the target region of the Nth frame image not matching the optical flow predicted target region of the Nth frame image, performing a second matching between the target region of the Nth frame image that did not pass the first matching and the Kalman filter predicted target region of the Nth frame image; and in response to the Nth frame image not matching the target region of the Nth frame image not passing the first matching, performing a second matching between the target region of the Nth frame image that failed the first matching and the Kalman filter predicted target region of the Nth frame image; and performing a second matching between the target region of the Nth frame image that did not pass the first matching and the Kalman filter predicted target region of the Nth frame image. The target region of the frame image is matched with the Kalman filter predicted target region of the Nth frame image. For the target region of the Nth frame image that passes the second match, the Kalman filter model is updated and the optical flow points are resampled. In response to the target region of the Nth frame image that fails the first match not matching the Kalman filter predicted target region of the Nth frame image, a Kalman motion model is created and initialized for the target regions of the Nth frame image that fail both the first and second matches, and optical flow points are sampled. For the Kalman filter predicted target region of the Nth frame image that fails the second match, a prediction condition is determined. In response to the determination result that the prediction condition is not met, the Kalman motion model is deleted. In response to the determination result that the prediction condition is met, the Kalman filter model is updated and the optical flow points are resampled using the optical flow predicted target region of the Nth frame image corresponding to the Kalman filter predicted target region of the Nth frame image that fails the second match. The Kalman filter predicted target region of the Nth frame image that fails the second match and the optical flow predicted target region of the corresponding Nth frame image correspond to the same target.

[0009] In some embodiments of this disclosure, the target tracking method further includes: acquiring a first frame image; performing target detection on the first frame image using the target detection algorithm to generate a target region; performing optical flow point sampling on the target region of the first frame image; and creating and initializing a Kalman motion model for the target region of the first frame image.

[0010] In some embodiments of this disclosure, the target detection algorithm is YOLOv5, SSD, or FasterRcnn.

[0011] In some embodiments of this disclosure, the step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image includes: using forward optical flow calculation and reverse optical flow calculation to calculate the optical flow vector of the sampling points in the target region of the (N-1)th frame image, wherein the average value of the optical flow vectors less than a first distance threshold is taken as the optical flow prediction vector of the optical flow prediction target region of the Nth frame image.

[0012] In some embodiments of this disclosure, the step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image includes: calculating the optical flow vector of the optical flow prediction target region of the Nth frame image, and deleting the optical flow prediction target regions whose optical flow vectors are greater than a second distance threshold.

[0013] In some embodiments of this disclosure, the first matching of the target region of the Nth frame image with the optical flow prediction target region of the Nth frame image includes: using the IOU algorithm to compare the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

[0014] In some embodiments of this disclosure, the step of performing a second match between the target region of the Nth frame image that failed the first match and the Kalman filter predicted target region of the Nth frame image includes: using the IOU algorithm to compare the overlap between the target region of the Nth frame image that failed the first match and the Kalman filter predicted target region of the Nth frame image.

[0015] In some embodiments of this disclosure, the prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the first distance threshold.

[0016] In some embodiments of this disclosure, the prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the second distance threshold.

[0017] In some embodiments of this disclosure, the step of performing target detection on the Nth frame image using a target detection algorithm to generate a target region includes: using a CNN target detector, performing target detection on the Nth frame image using the target detection algorithm to generate the target region.

[0018] In a second aspect, embodiments of this disclosure provide a target tracking system, comprising: a camera configured to sequentially acquire N frames of images, where N is greater than or equal to 2 and N is an integer; a target detector connected to the camera and configured to perform target detection on the Nth frame image using a target detection algorithm to generate a target region; an optical flow prediction module connected to the camera and the target detector and configured to perform optical flow calculation on the target region of the (N-1)th frame image to generate an optical flow predicted target region of the Nth frame image; and a Kalman prediction module connected to the camera and the target detector and configured to perform Kalman prediction on the target region of the (N-1)th frame image. Filtering prediction is performed to generate a Kalman filter predicted target region for the Nth frame image; a first matching module, connected to the target detector, the optical flow prediction module, and the Kalman prediction module, is configured to perform a first match between the target region of the Nth frame image from the target detector and the optical flow predicted target region of the Nth frame image from the optical flow prediction module; in response to the matching of the target region of the Nth frame image with the optical flow predicted target region of the Nth frame image, the first matching module is further configured to output the target region of the Nth frame image that has passed the first match to update the Kalman filter model and perform optical flow point resampling; in response to the target region of the Nth frame image... If the target region of the Nth frame image does not match the optical flow prediction target region, the first matching module is further configured to output the target region of the Nth frame image that failed the first matching to the second matching module. The second matching module is configured to perform a second matching between the target region of the Nth frame image that failed the first matching and the Kalman filter prediction target region of the Nth frame image from the Kalman prediction module. In response to the target region of the Nth frame image that failed the first matching matching matching with the Kalman filter prediction target region of the Nth frame image, the second matching module is further configured to output the target region of the Nth frame image that passed the second matching to update the Kalman filter model and The optical flow point resampling is performed; in response to the target region of the Nth frame image that fails the first match not matching the Kalman filter predicted target region of the Nth frame image, the second matching module is further configured to output the target regions of the Nth frame image that fail both the first and second matches to create and initialize the Kalman motion model and perform optical flow point sampling; the second matching module is further configured to output the Kalman filter predicted target regions of the Nth frame image that fail the second match to the determination module, the determination module is configured to perform prediction condition determination; in response to the determination result being that the prediction condition is not met, the determination module is further configured to delete the Kalman motion model;In response to a determination result that meets the prediction conditions, the determination module is further configured to update the Kalman filter model and perform optical flow point resampling using the optical flow prediction target region from the optical flow prediction module of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that failed the second match; wherein the Kalman filter prediction target region of the Nth frame image that failed the second match and the optical flow prediction target region of the corresponding Nth frame image correspond to the same target.

[0019] In some embodiments of this disclosure, the camera is further configured to acquire a first frame image; the target detector is further configured to receive the first frame image and perform target detection on the first frame image using the target detection algorithm to generate a target region; the optical flow prediction module is further configured to perform optical flow point sampling on the target region of the first frame image from the target detector; and the Kalman prediction module is further configured to create and initialize a Kalman motion model on the target region of the first frame image from the target detector.

[0020] In some embodiments of this disclosure, the target detection algorithm is YOLOv5, SSD, or FasterRcnn.

[0021] In some embodiments of this disclosure, the optical flow prediction module is further configured to: calculate the optical flow vector of the sampling points in the target region of the (N-1)th frame image using forward and reverse optical flow calculations; wherein, the average value of the optical flow vectors less than a first distance threshold is taken as the optical flow prediction vector of the target region of the Nth frame image.

[0022] In some embodiments of this disclosure, the optical flow prediction module is further configured to: calculate the optical flow vector of the optical flow prediction target region of the Nth frame image, and delete the optical flow prediction target region whose optical flow vector is greater than a second distance threshold.

[0023] In some embodiments of this disclosure, the first matching module is further configured to: use the IOU algorithm to compare the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

[0024] In some embodiments of this disclosure, the second matching module is further configured to: use the IOU algorithm to compare the overlap between the target region of the Nth frame image that failed the first matching and the Kalman filter predicted target region of the Nth frame image.

[0025] In some embodiments of this disclosure, the prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the first distance threshold.

[0026] In some embodiments of this disclosure, the prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the second distance threshold.

[0027] In some embodiments of this disclosure, the object detector is a CNN object detector.

[0028] In a third aspect, embodiments of this disclosure provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the target tracking method according to any one of the embodiments in the first aspect.

[0029] In the target tracking method, target tracking system, and electronic device provided in the embodiments of this disclosure, the SORT tracking algorithm is combined with the sparse optical flow method to effectively solve the problems of tracking ID transformation and tracking failure caused by detection loss due to SORT's single IOU matching. At the same time, it can also effectively remedy when the detection effect is poor, thereby improving the stability of tracking and making the tracking effect less dependent on the detection effect. Attached Figure Description

[0030] The following figures are merely illustrative examples based on various disclosed embodiments and are not intended to limit the scope of the invention.

[0031] Figure 1a and Figure 1b The diagram illustrates multi-target tracking across two frames.

[0032] Figure 2 A flowchart of a target tracking method provided for embodiments of this disclosure.

[0033] Figure 3 A block diagram of a target tracking system provided for embodiments of this disclosure.

[0034] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown. Detailed Implementation

[0035] This disclosure will now be described in more detail with reference to the following embodiments. It should be noted that the following description of some embodiments presented herein is for illustrative and descriptive purposes only. It is not exhaustive or limited to the precise forms disclosed.

[0036] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in one or more embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0037] Embodiments of this disclosure provide a target tracking method, target tracking system, and electronic device that substantially eliminate one or more problems caused by the limitations and disadvantages of the prior art. In one aspect, this disclosure provides a target tracking method, comprising: sequentially acquiring N frames of images, where N is greater than or equal to 2 and N is an integer, wherein the target tracking method further comprises: performing target detection on the Nth frame image using a target detection algorithm to generate a target region; performing optical flow calculation on the target region of the (N-1)th frame image to generate an optical flow predicted target region of the Nth frame image; performing Kalman filter prediction on the target region of the (N-1)th frame image to generate a Kalman filter predicted target region of the Nth frame image; performing a first matching between the target region of the Nth frame image and the optical flow predicted target region of the Nth frame image; in response to the matching between the target region of the Nth frame image and the optical flow predicted target region of the Nth frame image, updating the Kalman filter model and resampling optical flow points for the target region of the Nth frame image that passed the first matching; in response to the mismatch between the target region of the Nth frame image and the optical flow predicted target region of the Nth frame image, performing a second matching between the target region of the Nth frame image that did not pass the first matching and the Kalman filter predicted target region of the Nth frame image; and in response to the mismatch between the Nth frame image that did not pass the first matching... The target region of the image is matched with the Kalman filter predicted target region of the Nth frame image. For the target region of the Nth frame image that passes the second match, the Kalman filter model is updated and the optical flow points are resampled. In response to the target region of the Nth frame image that fails the first match not matching the Kalman filter predicted target region of the Nth frame image, a Kalman motion model is created and initialized for the target regions of the Nth frame image that fail both the first and second matches, and optical flow points are sampled. For the Kalman filter predicted target region of the Nth frame image that fails the second match, a prediction condition is determined. In response to the determination result that the prediction condition is not met, the Kalman motion model is deleted. In response to the determination result that the prediction condition is met, the Kalman filter model is updated and the optical flow points are resampled using the optical flow predicted target region of the Nth frame image corresponding to the Kalman filter predicted target region of the Nth frame image that fails the second match. The Kalman filter predicted target region of the Nth frame image that fails the second match and the optical flow predicted target region of the corresponding Nth frame image correspond to the same target.

[0038] In a first aspect, embodiments of the present disclosure provide a target tracking method. Figure 2 A flowchart illustrating a target tracking method provided for embodiments of this disclosure. Figure 2 As shown, embodiments of this disclosure provide a target tracking method, including sequentially acquiring N frames of images, where N is greater than or equal to 2 and N is an integer, wherein the target tracking method further includes:

[0039] S1 performs target detection on the Nth frame image using a target detection algorithm to generate target regions (e.g., target bounding boxes);

[0040] Specifically, in some embodiments of this disclosure, a target detector is used to perform target detection on the Nth frame image using a target detection algorithm to generate a target region. Then, each target is sampled at intervals using a pre-set sampling step size S to obtain sampling points. In specific implementations, the size of S can be adjusted according to specific computing resources and target size. In some embodiments of this disclosure, S is set to 10 by default. For cases where S is less than 10, the midpoint of the target rectangle is taken. For example, if the width of the target region is 8 pixels, only points with S=4 are sampled (i.e., only the midpoint of the target region is taken).

[0041] In addition, in some embodiments of this disclosure, a scaling array R[] is obtained by calculating the distance between two points in the t-th frame and the distance between two corresponding points in the (t+1)-th frame. The scaling ratio of the target region in the current frame is obtained by taking the median value of R, thereby completing the prediction of the target's position and size in the current frame. Here, t is an integer greater than or equal to 1 and less than N. For example, the distance between two sampling points in the t-th frame is 10 pixels, while in the (t+1)-th frame, the distance between these two sampling points is 4 pixels, thus determining the scaling ratio of the target region in the current frame.

[0042] In some embodiments of this disclosure, the target detector uses target detection algorithms such as YOLOv5, SSD, or Faster RCNN.

[0043] In some embodiments of this disclosure, the sampling method can be implemented in an edge computing device. The sampling step size S is varied depending on the specific computational load and real-time requirements. Simultaneously, to ensure speed, optical flow prediction is performed using two scales or a single scale to balance tracking performance and algorithm execution speed.

[0044] S2 performs optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image (optical flow prediction result); and performs Kalman filter prediction on the target region of the (N-1)th frame image to generate the Kalman filter prediction target region of the Nth frame image (Kalman filter prediction result).

[0045] In some embodiments of this disclosure, the specific steps for performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region (optical flow prediction result) of the Nth frame image are as follows: Tracking prediction is performed using the sampling points in step S1. First, given an image sequence and a given time t, the position of the sampling point at time t+1 is located. Let S = (I[t], I[t+1], ..., I[t+k]) be an image sequence. To ensure the accuracy of optical flow prediction, this disclosure uses two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) to calculate the optical flow vector of the sampling point. The position of the sampling point at time t is x[t]. A tracking algorithm is used to track forward (i.e., calculate the optical flow vector in the forward direction), T[k,f] = (x[t], x[t+1]), where f represents the forward optical flow calculation. Subsequently, using the calculated position x[t+1], the optical flow vector is traced backward (i.e., the optical flow vector is calculated in reverse) to x[t], generating a verification trajectory T[k,b]=(x`[t],x`[t+1]), where b represents the backward optical flow calculation. Ideally, the predicted optical flow point (the target region for optical flow prediction) should coincide with the sampling point at time t. Based on this, the error of the prediction point is obtained:

[0046]

[0047] The error array FB[] is obtained by calculating the Euclidean distance between corresponding points of two optical flows. The median of FB[] is calculated as the first distance threshold, and the optical flow vectors of prediction points (optical flow prediction target areas) that are greater than the first distance threshold are deleted. Then, the average value of the optical flow vectors of prediction points that are less than the first distance threshold is taken as the optical flow prediction vector of the current frame to predict the next movement direction of the target.

[0048] In some embodiments of this disclosure, the first distance threshold is not limited to the median value of FB[]. In other embodiments of this disclosure, the first distance threshold can be set as needed.

[0049] Alternatively, in some embodiments of this disclosure, the step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image includes: calculating the optical flow vector of the optical flow prediction target region of the Nth frame image, and deleting the optical flow prediction target regions whose optical flow vectors are greater than a second distance threshold.

[0050] In some embodiments of this disclosure, if the optical flow vector of the optical flow prediction target region of the calculated Nth frame image is greater than the second distance threshold, it is considered that the moving speed of the sampling point is too large in the Nth and N-1th frame images, which is unreasonable, and therefore the prediction point (optical flow prediction target region) is deleted. Prediction points whose optical flow vector of the optical flow prediction target region of the calculated Nth frame image is less than the second distance threshold are retained.

[0051] In some embodiments of this disclosure, the second distance threshold can be set as needed.

[0052] In some embodiments of this disclosure, the calculation of the optical flow vector at the sampling point is not limited to the two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) used in the embodiments of this disclosure. In other embodiments of this disclosure, other methods can also be used to calculate the optical flow vector at the sampling point, which can be set as needed and are not limited here.

[0053] In some embodiments of this disclosure, optical flow calculation is performed on the target region of the (N-1)th frame image to generate the optical flow predicted target region of the Nth frame image (optical flow prediction result), and Kalman filtering prediction is performed on the target region of the (N-1)th frame image to generate the Kalman filter predicted target region of the Nth frame image (Kalman filter prediction result) simultaneously. However, this disclosure is not limited thereto. In other embodiments of this disclosure, the two steps may not occur simultaneously.

[0054] S3 performs a first match between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image; if the target region of the Nth frame image matches the optical flow prediction target region of the Nth frame image, the method proceeds to step S4; if the target region of the Nth frame image does not match the optical flow prediction target region of the Nth frame image, the method proceeds to step S5.

[0055] In some embodiments of this disclosure, performing a first match between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image includes: comparing the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image (i.e., performing IOU (Intersection Over Union) matching between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image).

[0056] Specifically, in some embodiments of this disclosure, a cost matrix is ​​calculated based on the overlap (i.e., intersection-union ratio) between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image. Then, a weighted Hungarian matching algorithm is used to process the calculated cost matrix, thereby performing a first matching between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

[0057] For example, the target region of the Nth frame image includes three target regions A1, B1, and C1, and correspondingly, the optical flow prediction target region of the Nth frame image includes three target regions A2, B2, and C2. The intersection-over-union (IOU) ratios of target region A1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are calculated; the IOU ratios of target region B1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are also calculated; the IOU ratios of target region C1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are also calculated. These nine values ​​form the cost matrix. Then, the calculated cost matrix is ​​processed using a weighted Hungarian matching algorithm to obtain the optimal matching pair between the target region and the optical flow prediction target region of the Nth frame image.

[0058] In some embodiments of this disclosure, the weighted Hungarian matching algorithm is used to obtain the optimal matching pair, so that for each pair of target regions formed by each target region in the target region of the Nth frame image and the corresponding target region in the optical flow prediction target region of the Nth frame image, the optimal solution can be obtained.

[0059] S4 updates the Kalman filter model and performs optical flow point resampling for the target region of the matched Nth frame image (the target region of the Nth frame image through the first match);

[0060] In some embodiments of this disclosure, for the target region of the matched Nth frame image, the Kalman filter model is updated and optical flow points are resampled. That is, optical flow is calculated for the target region of the matched Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0061] S5 performs a second match between the unmatched target region of the Nth frame image (the target region of the Nth frame image that did not pass the first match) and the Kalman filter predicted target region of the Nth frame image; if the unmatched target region of the Nth frame image matches the Kalman filter predicted target region of the Nth frame image, the method proceeds to step S6; if the unmatched target region of the Nth frame image does not match the Kalman filter predicted target region of the Nth frame image, the method proceeds to steps S7 and S8.

[0062] In some embodiments of this disclosure, performing a second matching between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image includes: comparing the overlap between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image (i.e., performing IOU matching between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image).

[0063] Specifically, in some embodiments of this disclosure, a cost matrix is ​​calculated based on the overlap (i.e., intersection-union ratio) between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image. Then, a weighted Hungarian matching algorithm is used to process the calculated cost matrix, thereby performing a second matching between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image.

[0064] For example, the target region of the unmatched Nth frame image includes three target regions A1, B1, and C1. Correspondingly, the Kalman filter predicted target region of the Nth frame image includes three target regions A2, B2, and C2. The Intersection over Union (IOU) ratios of target region A1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are calculated; the IOU ratios of target region B1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are also calculated; the IOU ratios of target region C1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are also calculated. These nine values ​​constitute the cost matrix. Then, the calculated cost matrix is ​​processed using a weighted Hungarian matching algorithm to obtain the optimal matching pair between the unmatched target regions of the Nth frame image and the Kalman filter predicted target regions of the Nth frame image.

[0065] In some embodiments of this disclosure, the second matching is mainly to ensure that the IOU matching of the Kalman filter prediction is still retained even when the optical flow point prediction fails, thereby ensuring the tracking recall rate.

[0066] In some embodiments of this disclosure, the relevant algorithms used in the sparse optical flow method can be implemented using NVIDIA's CUDA platform. TMImplemented in the Compute Unified Device Architecture (CWA). This CWA enables dense optical flow prediction, which is more computationally intensive and resource-intensive than sparse optical flow methods, to calculate the optical flow vectors of all pixels within the target region and perform optical flow prediction. Furthermore, to overcome the problem of optical flow failure at high speeds, this CWA employs multi-scale optical flow prediction to ensure both accuracy and speed.

[0067] S6 updates the Kalman filter model and performs optical flow point resampling for the target region of the matched Nth frame image (the target region of the Nth frame image through the second match);

[0068] In some embodiments of this disclosure, for the target region of the matched Nth frame image, the Kalman filter model is updated and optical flow points are resampled. That is, optical flow is calculated for the target region of the matched Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0069] S7 creates and initializes a Kalman motion model and performs optical flow point sampling for the target region of the Nth frame image that is not matched (i.e., the target region of the Nth frame image that has not passed the first and second matching operations. Since the two matching operations have not matched, the target is considered to be a newly emerging target).

[0070] S8 performs a prediction condition determination for the Kalman filter prediction target region of the Nth frame image that is not matched (the Kalman filter prediction target region of the Nth frame image that does not pass the second match); if the determination result is that the prediction condition is not met, the method proceeds to step S9; if the determination result is that the prediction condition is met, the method proceeds to step S10.

[0071] In some embodiments of this disclosure, the prediction condition determination for the Kalman filter prediction target region of the unmatched Nth frame image includes: determining the prediction condition for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the unmatched Nth frame image, wherein the Kalman filter prediction target region of the unmatched Nth frame image and the optical flow prediction target region of the Nth frame image correspond to the same target.

[0072] Specifically, in the optical flow prediction results of the optical flow prediction target region of the Nth frame image, if the optical flow vector of the sampling point calculated by two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) is greater than a first distance threshold, where the first distance threshold is the median of the error FB[] of the prediction point, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to not meet the prediction conditions; if the optical flow vector of the sampling point calculated by two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) is less than the first distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to meet the prediction conditions.

[0073] Alternatively, in some embodiments of this disclosure, the optical flow vector of the optical flow prediction target region of the Nth frame image is calculated, and optical flow prediction target regions whose optical flow vectors are greater than a second distance threshold are deleted. In some embodiments of this disclosure, if the calculated optical flow vector of the optical flow prediction target region of the Nth frame image is greater than the second distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to not meet the prediction conditions; if the calculated optical flow vector of the optical flow prediction target region of the Nth frame image is less than the second distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to meet the prediction conditions.

[0074] S9 removes the Kalman motion model;

[0075] In some embodiments of this disclosure, if the determination result is that the determination condition is not met, the Kalman motion model is deleted. However, this disclosure is not limited thereto. In other embodiments of this disclosure, if the determination result is that the determination condition is not met, the lifetime of the Kalman motion model is incremented by 1. When the lifetime of the Kalman motion model exceeds a certain threshold, the Kalman motion model is deleted.

[0076] S10 uses the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match (the Kalman filter prediction target region of the Nth frame image that did not pass the second match) to update the Kalman filter model and perform optical flow point resampling.

[0077] In some embodiments of this disclosure, the Kalman filter model is updated and optical flow point resampling is performed using the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that is not matched. That is, optical flow calculation is performed on the optical flow prediction target region of the Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0078] In some embodiments of this disclosure, the object detector is a CNN (convolutional neural network) object detector.

[0079] In some optional implementations of this embodiment, the object detector can be obtained by training a pre-defined convolutional neural network (CNN). This CNN can be an untrained or incompletely trained multi-layer CNN. For example, the CNN may include convolutional layers, pooling layers, fully connected layers, and loss layers. Furthermore, a non-first convolutional layer in the CNN can be connected to at least one convolutional layer preceding it. For example, the non-first convolutional layer can be connected to all preceding convolutional layers to achieve first convolutional layer selection; the non-first convolutional layer can also be connected to some preceding convolutional layers.

[0080] In some embodiments of this disclosure, the target tracking method further includes acquiring a first frame image; performing target detection on the first frame image using a target detection algorithm to generate a target region (e.g., a target bounding box); performing optical flow point sampling on the target region and creating and initializing a Kalman motion model.

[0081] In some embodiments of this disclosure, the above steps S1 to S10 are performed on each of the N frames of images, where N is greater than or equal to 2 and N is an integer, thereby completing the tracking of multiple targets in the video stream.

[0082] In some embodiments of this disclosure, a second matching is performed to ensure sufficient correlation between the tracking of preceding and following frames.

[0083] In some embodiments of this disclosure, optical flow compensation is performed on the Kalman filter predicted target region of the unmatched Nth frame image, and the Kalman filter model is updated using the compensated Kalman filter predicted target region, thereby ensuring the continuity of the motion model update and improving tracking stability.

[0084] In the target tracking method provided in the embodiments of this disclosure, the SORT tracking algorithm is combined with the sparse optical flow method to effectively solve the problems of tracking ID transformation and tracking failure caused by detection loss due to SORT's single IOU matching. At the same time, it can also effectively remedy when the detection effect is poor, thereby improving the stability of tracking and making the tracking effect less dependent on the detection effect.

[0085] It should be noted that the methods of one or more embodiments of this disclosure can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this disclosure, and the multiple devices will interact with each other to complete the method described.

[0086] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] In a second aspect, embodiments of this disclosure provide a target tracking system. Figure 3 A block diagram of a target tracking system provided for embodiments of this disclosure. (As shown) Figure 3 As shown, embodiments of this disclosure provide a target tracking system, including: a camera 1 configured to sequentially acquire N frames of images, where N is greater than or equal to 2 and N is an integer; and a target detector 2 connected to the camera 1 and configured to perform target detection on the Nth frame of the image using a target detection algorithm to generate a target region (e.g., a target bounding box).

[0088] Specifically, in some embodiments of this disclosure, target detector 2 is used to perform target detection on the Nth frame image using a target detection algorithm to generate a target region. Then, each target is sampled at intervals using a pre-set sampling step size S to obtain sampling points. In specific implementations, the size of S can be adjusted according to specific computing resources and target size. In some embodiments of this disclosure, S is set to 10 by default. For cases where S is less than 10, the midpoint of the target rectangle is taken. For example, if the width of the target region is 8 pixels, only points with S=4 are sampled (i.e., only the midpoint of the target region is taken).

[0089] In addition, in some embodiments of this disclosure, a scaling array R[] is obtained by calculating the distance between two points in the t-th frame and the distance between two corresponding points in the (t+1)-th frame. The scaling ratio of the target region in the current frame is obtained by taking the median value of R, thereby completing the prediction of the target's position and size in the current frame. Here, t is an integer greater than or equal to 1 and less than N. For example, the distance between two sampling points in the t-th frame is 10 pixels, while in the (t+1)-th frame, the distance between these two sampling points is 4 pixels, thus determining the scaling ratio of the target region in the current frame.

[0090] In some embodiments of this disclosure, the target detector 2 uses target detection algorithms such as YOLOv5, SSD, or Faster RCN.

[0091] In some embodiments of this disclosure, the sampling method can be implemented in an edge computing device. The sampling step size S is varied depending on the specific computational load and real-time requirements. Simultaneously, to ensure speed, optical flow prediction is performed using two scales or a single scale to balance tracking performance and algorithm execution speed.

[0092] The target tracking system provided in the embodiments of this disclosure further includes an optical flow prediction module 3, which is connected to the camera 1 and the target detector 2, and is configured to perform optical flow calculation on the target region of the (N-1)th frame image to generate an optical flow predicted target region (optical flow prediction result) of the Nth frame image; and a Kalman prediction module 4, which is connected to the camera 1 and the target detector 2, and is configured to perform Kalman filter prediction on the target region of the (N-1)th frame image to generate a Kalman filter predicted target region (Kalman filter prediction result) of the Nth frame image.

[0093] In some embodiments of this disclosure, optical flow calculation is performed on the target region of the (N-1)th frame image to generate the optical flow predicted target region of the Nth frame image (optical flow prediction result), and Kalman filtering prediction is performed on the target region of the (N-1)th frame image to generate the Kalman filter predicted target region of the Nth frame image (Kalman filter prediction result) simultaneously. However, this disclosure is not limited thereto. In other embodiments of this disclosure, the two steps may not occur simultaneously.

[0094] In some embodiments of this disclosure, the specific steps for performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region (optical flow prediction result) of the Nth frame image are as follows: Tracking prediction is performed using the sampling points in step S1. First, given an image sequence and a given time t, the position of the sampling point at time t+1 is located. Let S = (I[t], I[t+1], ..., I[t+k]) be an image sequence. To ensure the accuracy of optical flow prediction, this disclosure uses two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) to calculate the optical flow vector of the sampling point. The position of the sampling point at time t is x[t]. A tracking algorithm is used to track forward (i.e., calculate the optical flow vector in the forward direction), T[k,f] = (x[t], x[t+1]), where f represents the forward optical flow calculation. Subsequently, using the calculated position x[t+1], the optical flow vector is traced backward (i.e., the optical flow vector is calculated in reverse) to x[t], generating a verification trajectory T[k,b]=(x`[t],x`[t+1]), where b represents the backward optical flow calculation. Ideally, the predicted optical flow point (the target region for optical flow prediction) should coincide with the sampling point at time t. Based on this, the error of the prediction point is obtained:

[0095]

[0096] The error array FB[] is obtained by calculating the Euclidean distance between corresponding points of two optical flows. The median of FB[] is calculated as the first distance threshold, and the optical flow vectors of prediction points (optical flow prediction target areas) that are greater than the first distance threshold are deleted. Then, the average value of the optical flow vectors of prediction points that are less than the first distance threshold is taken as the optical flow prediction vector of the current frame to predict the next movement direction of the target.

[0097] In some embodiments of this disclosure, the first distance threshold is not limited to the median value of FB[]. In other embodiments of this disclosure, the first distance threshold can be set as needed.

[0098] Alternatively, in some embodiments of this disclosure, the step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image specifically includes: calculating the optical flow vector of the optical flow prediction target region of the Nth frame image, and deleting the optical flow prediction target regions whose optical flow vectors are greater than a second distance threshold.

[0099] In some embodiments of this disclosure, if the optical flow vector of the optical flow prediction target region of the calculated Nth frame image is greater than the second distance threshold, it is considered that the moving speed of the sampling point is too large in the Nth and N-1th frame images, which is unreasonable, and therefore the prediction point (optical flow prediction target region) is deleted. Prediction points whose optical flow vector of the optical flow prediction target region of the calculated Nth frame image is less than the second distance threshold are retained.

[0100] In some embodiments of this disclosure, the second distance threshold can be set as needed.

[0101] In some embodiments of this disclosure, the calculation of the optical flow vector at the sampling point is not limited to the two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) used in the embodiments of this disclosure. In other embodiments of this disclosure, other methods can also be used to calculate the optical flow vector at the sampling point, which can be set as needed and are not limited here.

[0102] The target tracking system provided in the embodiments of this disclosure further includes a first matching module 5, which is connected to the target detector 2, the optical flow prediction module 3 and the Kalman prediction module 4, and is configured to perform a first matching between the target region of the Nth frame image from the target detector 2 and the optical flow predicted target region of the Nth frame image from the optical flow prediction module 3.

[0103] In some embodiments of this disclosure, the first matching module 5 is configured to: compare the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image (i.e., perform IOU (Intersection Over Union) matching between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image).

[0104] Specifically, in some embodiments of this disclosure, a cost matrix is ​​calculated based on the overlap (i.e., intersection-union ratio) between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image. Then, a weighted Hungarian matching algorithm is used to process the calculated cost matrix, thereby performing a first matching between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

[0105] For example, the target region of the Nth frame image includes three target regions A1, B1, and C1, and correspondingly, the optical flow prediction target region of the Nth frame image includes three target regions A2, B2, and C2. The intersection-over-union (IOU) ratios of target region A1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are calculated; the IOU ratios of target region B1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are also calculated; the IOU ratios of target region C1 with the three target regions A2, B2, and C2 in the optical flow prediction target region of the Nth frame image are also calculated. These nine values ​​form the cost matrix. Then, the calculated cost matrix is ​​processed using a weighted Hungarian matching algorithm to obtain the optimal matching pair between the target region and the optical flow prediction target region of the Nth frame image.

[0106] In some embodiments of this disclosure, the weighted Hungarian matching algorithm is used to obtain the optimal matching pair, so that for each pair of target regions formed by each target region in the target region of the Nth frame image and the corresponding target region in the optical flow prediction target region of the Nth frame image, the optimal solution can be obtained.

[0107] In some embodiments of this disclosure, in response to the target region of the Nth frame image being matched with the optical flow prediction target region of the Nth frame image, the first matching module 5 is further configured to output the target region of the Nth frame image that has passed the first match (the target region of the Nth frame image that has passed the first match) to update the Kalman filter model and perform optical flow point resampling.

[0108] In some embodiments of this disclosure, for the target region of the matched Nth frame image, the Kalman filter model is updated and optical flow points are resampled. That is, optical flow is calculated for the target region of the matched Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0109] In some embodiments of this disclosure, in response to a mismatch between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image, the first matching module 5 is further configured to output the target region of the Nth frame image that did not pass the first matching to the second matching module 6. The second matching module 6 is configured to perform a second matching between the unmatched target region of the Nth frame image (the target region of the Nth frame image that did not pass the first matching) and the Kalman filter prediction target region of the Nth frame image from the Kalman prediction module 4.

[0110] In some embodiments of this disclosure, the second matching module 6 is configured to: compare the overlap between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image (i.e., perform IOU matching between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image).

[0111] Specifically, in some embodiments of this disclosure, a cost matrix is ​​calculated based on the overlap (i.e., intersection-union ratio) between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image. Then, a weighted Hungarian matching algorithm is used to process the calculated cost matrix, thereby performing a second matching between the target region of the unmatched Nth frame image and the Kalman filter predicted target region of the Nth frame image.

[0112] For example, the target region of the unmatched Nth frame image includes three target regions A1, B1, and C1. Correspondingly, the Kalman filter predicted target region of the Nth frame image includes three target regions A2, B2, and C2. The Intersection over Union (IOU) ratios of target region A1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are calculated; the IOU ratios of target region B1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are also calculated; the IOU ratios of target region C1 with the three target regions A2, B2, and C2 in the Kalman filter predicted target region of the Nth frame image are also calculated. These nine values ​​constitute the cost matrix. Then, the calculated cost matrix is ​​processed using a weighted Hungarian matching algorithm to obtain the optimal matching pair between the unmatched target regions of the Nth frame image and the Kalman filter predicted target regions of the Nth frame image.

[0113] In some embodiments of this disclosure, the second matching is mainly to ensure that the IOU matching of the Kalman filter prediction is still retained even when the optical flow point prediction fails, thereby ensuring the tracking recall rate.

[0114] In some embodiments of this disclosure, the relevant algorithms used in the sparse optical flow method can be implemented using NVIDIA's CUDA platform. TMImplemented in the Compute Unified Device Architecture (CWA). This CWA enables dense optical flow prediction, which is more computationally intensive and resource-intensive than sparse optical flow methods, to calculate the optical flow vectors of all pixels within the target region and perform optical flow prediction. Furthermore, to overcome the problem of optical flow failure at high speeds, this CWA employs multi-scale optical flow prediction to ensure both accuracy and speed.

[0115] In some embodiments of this disclosure, in response to the target region of the Nth frame image that did not pass the first match being matched with the Kalman filter predicted target region of the Nth frame image, the second matching module 6 is further configured to output the target region of the Nth frame image that has been matched (the target region of the Nth frame image that passed the second match) to update the Kalman filter model and perform the optical flow point resampling.

[0116] In some embodiments of this disclosure, for the target region of the matched Nth frame image, the Kalman filter model is updated and optical flow points are resampled. That is, optical flow is calculated for the target region of the matched Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0117] In some embodiments of this disclosure, in response to the target region of the Nth frame image that fails the first match not matching the Kalman filter predicted target region of the Nth frame image, the second matching module 6 is further configured to output the target regions of the Nth frame image that fail both the first and second matches, in order to create and initialize the Kalman motion model and perform optical flow point sampling.

[0118] In some embodiments of this disclosure, the second matching module 6 is further configured to output the Kalman filter prediction target region of the Nth frame image that is not matched (the Kalman filter prediction target region of the Nth frame image that does not pass the second matching) to the determination module 7, which is configured to perform prediction condition determination.

[0119] In some embodiments of this disclosure, the determination module 7 is configured to: determine the prediction conditions for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that is not matched, wherein the Kalman filter prediction target region of the Nth frame image and the optical flow prediction target region of the Nth frame image correspond to the same target.

[0120] Specifically, in the optical flow prediction results of the optical flow prediction target region of the Nth frame image, if the optical flow vector of the sampling point calculated by two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) is greater than a first distance threshold, where the first distance threshold is the median of the error FB[] of the prediction point, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to not meet the prediction conditions; if the optical flow vector of the sampling point calculated by two optical flow calculations (i.e., forward optical flow calculation and reverse optical flow calculation) is less than the first distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to meet the prediction conditions.

[0121] Alternatively, in some embodiments of this disclosure, the optical flow vector of the optical flow prediction target region of the Nth frame image is calculated, and optical flow prediction target regions whose optical flow vectors are greater than a second distance threshold are deleted. In some embodiments of this disclosure, if the calculated optical flow vector of the optical flow prediction target region of the Nth frame image is greater than the second distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to not meet the prediction conditions; if the calculated optical flow vector of the optical flow prediction target region of the Nth frame image is less than the second distance threshold, then the optical flow prediction target region of the Nth frame image corresponding to the unmatched Kalman filter prediction target region of the Nth frame image is considered to meet the prediction conditions.

[0122] In some embodiments of this disclosure, in response to a determination result that does not meet the prediction conditions, the determination module 7 is further configured to delete the Kalman motion model.

[0123] In some embodiments of this disclosure, if the determination result is that the determination condition is not met, the Kalman motion model is deleted. However, this disclosure is not limited thereto. In other embodiments of this disclosure, if the determination result is that the determination condition is not met, the lifetime of the Kalman motion model is incremented by 1. When the lifetime of the Kalman motion model exceeds a certain threshold, the Kalman motion model is deleted.

[0124] In some embodiments of this disclosure, in response to the determination result being in accordance with the prediction conditions, the determination module 7 is further configured to update the Kalman filter model and perform optical flow point resampling using the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second matching (Kalman filter prediction target region of the Nth frame image that did not pass the second matching).

[0125] In some embodiments of this disclosure, the Kalman filter model is updated and optical flow point resampling is performed using the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that is not matched. That is, optical flow calculation is performed on the optical flow prediction target region of the Nth frame image to generate the optical flow prediction target region (optical flow prediction result) of the N+1th frame image; and Kalman filter prediction is performed on the target region of the matched Nth frame image to generate the Kalman filter prediction target region (Kalman filter prediction result) of the N+1th frame image.

[0126] In some embodiments of this disclosure, the object detector 2 is a CNN (convolutional neural network) object detector.

[0127] In some optional implementations of this embodiment, the object detector 2 can be obtained by training a preset convolutional neural network (CNN). This CNN can be an untrained or incompletely trained multi-layer CNN. For example, the CNN may include convolutional layers, pooling layers, fully connected layers, and loss layers. Furthermore, a non-first convolutional layer in the CNN can be connected to at least one convolutional layer preceding it. For example, the non-first convolutional layer can be connected to all preceding convolutional layers to achieve first convolutional layer selection; the non-first convolutional layer can also be connected to some preceding convolutional layers.

[0128] In some embodiments of this disclosure, the camera 1 is further configured to acquire a first frame image; the target detector 2 is further configured to receive the first frame image and perform target detection on the first frame image using the target detection algorithm to generate a target region; the optical flow prediction module 3 is further configured to perform optical flow point sampling on the target region of the first frame image from the target detector 2; and the Kalman prediction module 4 is further configured to create and initialize a Kalman motion model on the target region of the first frame image from the target detector 2.

[0129] In some embodiments of this disclosure, the above operation is performed on each of the N frames of images, where N is greater than or equal to 2 and N is an integer, thereby completing the tracking of multiple targets in the video stream.

[0130] In some embodiments of this disclosure, a second matching is performed to ensure sufficient correlation between the tracking of preceding and following frames.

[0131] In some embodiments of this disclosure, optical flow compensation is performed on the Kalman filter predicted target region of the unmatched Nth frame image, and the Kalman filter model is updated using the compensated Kalman filter predicted target region, thereby ensuring the continuity of the motion model update and improving tracking stability.

[0132] In the target tracking system provided in the embodiments of this disclosure, the SORT tracking algorithm is combined with the sparse optical flow method to effectively solve the problems of tracking ID transformation and tracking failure caused by detection loss due to SORT's single IOU matching. At the same time, it can also effectively remedy when the detection effect is poor, thereby improving the stability of tracking and making the tracking effect less dependent on the detection effect.

[0133] In a third aspect, one or more embodiments of this disclosure provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target tracking method as described in any of the foregoing embodiments.

[0134] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown. Figure 4 As shown, the electronic device 1000 may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected within the electronic device via the bus 1050.

[0135] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the target tracking method as described in any of the above embodiments.

[0136] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the target tracking method described in any of the above embodiments is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0137] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be installed as components in electronic devices (not shown in the figure) or externally connected to electronic devices to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0138] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this electronic device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0139] Bus 1050 is used to transfer information between various components of an electronic device (e.g., processor 1010, memory 1020, input / output interface 1030, and communication interface 1040).

[0140] It should be noted that although the above-described electronic device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the electronic device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described electronic device may only include the components necessary for implementing the target tracking method described in the embodiments of this disclosure, and does not necessarily need to include all the components shown in the figures.

[0141] This disclosure also provides a non-transitory computer-readable storage medium storing computer-executable instructions thereon, wherein the instructions, when executed by a processor, perform the target tracking method described above.

[0142] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; under the inventive concept of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and electronic devices according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two blocks shown sequentially may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0144] The components involved in the embodiments of this disclosure can be implemented in software or in hardware. The described components can also be housed in a processor; for example, each component can be a software program located in a computer or mobile smart device, or a separately configured hardware device. The names of these components do not, in certain circumstances, constitute a limitation on the component itself.

[0145] Additionally, to simplify the description and discussion, and to avoid obscuring one or more embodiments of this disclosure, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring one or more embodiments of this disclosure, and this also takes into account the fact that details regarding implementations of these block diagram-based apparatuses are highly dependent on the platform on which one or more embodiments of this disclosure will be implemented (i.e., such details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that one or more embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0146] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0147] One or more embodiments of this disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A target tracking method, comprising: Obtain N frames of images sequentially, where N is greater than or equal to 2 and is an integer. The target tracking method further includes: The Nth frame image is used to detect targets using a target detection algorithm to generate target regions. Optical flow calculation is performed on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image; and Kalman filtering prediction is performed on the target region of the (N-1)th frame image to generate the Kalman filtering prediction target region of the Nth frame image. Perform a first match between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image; In response to the target region of the Nth frame image being matched with the optical flow prediction target region of the Nth frame image, the Kalman filter model is updated and optical flow points are resampled for the target region of the Nth frame image that has passed the first match; In response to a mismatch between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image, a second match is performed between the target region of the Nth frame image that failed the first match and the Kalman filter prediction target region of the Nth frame image. In response to the target region of the Nth frame image that fails the first match being matched with the Kalman filter predicted target region of the Nth frame image, the Kalman filter model is updated and the optical flow point is resampled for the target region of the Nth frame image that passes the second match. In response to the target region of the Nth frame image that failed the first match not matching the Kalman filter predicted target region of the Nth frame image, For the target region of the Nth frame image that fails both the first and second matching, a Kalman motion model is created and initialized, and optical flow point sampling is performed; For the Kalman filter prediction target region of the Nth frame image that fails the second match, a prediction condition is determined to delete unreasonable optical flow prediction target regions. The prediction condition is applied to the optical flow prediction target region of the Nth frame image that corresponds to the Kalman filter prediction target region of the Nth frame image that fails the second match. In response to the determination result that the prediction conditions are not met, the Kalman motion model is deleted; In response to the determination result being in accordance with the prediction conditions, the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match is used to update the Kalman filter model and perform optical flow point resampling; Wherein, the Kalman filter prediction target region of the Nth frame image that did not pass the second matching corresponds to the optical flow prediction target region of the corresponding Nth frame image and is the same target.

2. The target tracking method according to claim 1 further includes: Get the first frame image; The target detection algorithm is used to detect targets in the first frame image to generate target regions. Optical flow point sampling is performed on the target region of the first frame image, and For the target region of the first frame image, a Kalman motion model is created and initialized.

3. The target tracking method according to claim 2, wherein, The target detection algorithm is yolov5, SSD or FasterRcnn.

4. The target tracking method according to claim 2, wherein, The step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image includes: The optical flow vectors of the sampling points in the target region of the (N-1)th frame image are calculated using forward optical flow calculation and reverse optical flow calculation. The average value of the optical flow vectors that are less than a first distance threshold is taken as the optical flow prediction vector of the target region of the Nth frame image.

5. The target tracking method according to claim 2, wherein, The step of performing optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image includes: Calculate the optical flow vector of the optical flow prediction target region in the Nth frame image, and delete the optical flow prediction target regions whose optical flow vectors are greater than the second distance threshold.

6. The target tracking method according to claim 1, wherein, The first matching of the target region of the Nth frame image with the optical flow prediction target region of the Nth frame image includes: The IOU algorithm is used to compare the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

7. The target tracking method according to claim 1, wherein, The step of performing a second match between the target region of the Nth frame image that failed the first match and the Kalman filter predicted target region of the Nth frame image includes: The IOU algorithm is used to compare the overlap between the target region of the Nth frame image that failed the first match and the Kalman filter predicted target region of the Nth frame image.

8. The target tracking method according to claim 4, wherein, The prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the first distance threshold.

9. The target tracking method according to claim 5, wherein, The prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the second distance threshold.

10. The target tracking method according to claim 1, wherein, The step of performing target detection on the Nth frame image using a target detection algorithm to generate a target region includes: Using a CNN object detector and the object detection algorithm, object detection is performed on the Nth frame image to generate the target region.

11. A target tracking system, comprising: A camera is configured to sequentially acquire N frames of images, where N is greater than or equal to 2 and is an integer. A target detector, connected to the camera, is configured to perform target detection on the Nth frame image using a target detection algorithm to generate a target region; An optical flow prediction module, which is connected to the camera and the target detector, is configured to perform optical flow calculation on the target region of the (N-1)th frame image to generate the optical flow prediction target region of the Nth frame image; A Kalman prediction module, which is connected to the camera and the target detector, is configured to perform Kalman filtering prediction on the target region of the (N-1)th frame image to generate the Kalman filtering predicted target region of the Nth frame image; A first matching module is connected to the target detector, the optical flow prediction module, and the Kalman prediction module, and is configured to perform a first match between the target region of the Nth frame image from the target detector and the optical flow predicted target region of the Nth frame image from the optical flow prediction module. In response to the target region of the Nth frame image being matched with the optical flow prediction target region of the Nth frame image, the first matching module is further configured to output the target region of the Nth frame image that has passed the first match, so as to update the Kalman filter model and perform optical flow point resampling; In response to a mismatch between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image, the first matching module is further configured to output the target region of the Nth frame image that failed the first matching to the second matching module, and the second matching module is configured to perform a second matching between the target region of the Nth frame image that failed the first matching and the Kalman filter prediction target region of the Nth frame image from the Kalman prediction module; In response to the target region of the Nth frame image that fails the first match being matched with the Kalman filter predicted target region of the Nth frame image, the second matching module is further configured to output the target region of the Nth frame image that passes the second match, so as to update the Kalman filter model and perform the optical flow point resampling; In response to the target region of the Nth frame image that failed the first match not matching the Kalman filter predicted target region of the Nth frame image, The second matching module is also configured to output the target region of the Nth frame image that fails to pass the first and second matching, in order to create and initialize the Kalman motion model and perform optical flow point sampling; The second matching module is further configured to output the Kalman filter prediction target region of the Nth frame image that failed the second matching to the determination module. The determination module is configured to perform prediction condition determination to delete unreasonable optical flow prediction target regions. The prediction condition is applied to the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that failed the second matching. In response to the determination result being that the prediction conditions are not met, the determination module is further configured to delete the Kalman motion model; In response to the determination result being in accordance with the prediction conditions, the determination module is further configured to update the Kalman filter model and perform optical flow point resampling using the optical flow prediction target region from the optical flow prediction module of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that failed the second match; Wherein, the Kalman filter prediction target region of the Nth frame image that did not pass the second matching corresponds to the optical flow prediction target region of the corresponding Nth frame image and is the same target.

12. The target tracking system according to claim 11, wherein, The camera is also configured to acquire a first frame image; The target detector is also configured to receive the first frame image and perform target detection on the first frame image using the target detection algorithm to generate a target region; The optical flow prediction module is further configured to perform optical flow point sampling on the target region of the first frame image from the target detector, and The Kalman prediction module is also configured to create and initialize a Kalman motion model for the target region of the first frame image from the target detector.

13. The target tracking system according to claim 12, wherein, The target detection algorithm is yolov5, SSD or FasterRcnn.

14. The target tracking system according to claim 12, wherein, The optical flow prediction module is also configured to: The optical flow vectors of the sampling points in the target region of the (N-1)th frame image are calculated using forward and reverse optical flow calculations; wherein, the average value of the optical flow vectors less than a first distance threshold is taken as the optical flow prediction vector of the target region of the Nth frame image.

15. The target tracking system according to claim 12, wherein, The optical flow prediction module is also configured to: Calculate the optical flow vector of the optical flow prediction target region in the Nth frame image, and delete the optical flow prediction target regions whose optical flow vectors are greater than the second distance threshold.

16. The target tracking system according to claim 11, wherein, The first matching module is also configured as follows: The IOU algorithm is used to compare the overlap between the target region of the Nth frame image and the optical flow prediction target region of the Nth frame image.

17. The target tracking system according to claim 11, wherein, The second matching module is also configured as follows: The IOU algorithm is used to compare the overlap between the target region of the Nth frame image that failed the first match and the Kalman filter predicted target region of the Nth frame image.

18. The target tracking system according to claim 14, wherein, The prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the first distance threshold.

19. The target tracking system according to claim 15, wherein, The prediction condition is that for the optical flow prediction target region of the Nth frame image corresponding to the Kalman filter prediction target region of the Nth frame image that did not pass the second match, its optical flow vector is less than the second distance threshold.

20. The target tracking system according to claim 11, wherein, The object detector is a CNN object detector.

21. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the computer program is executed by the processor, it implements the target tracking method according to any one of claims 1 to 10.

Images