Object Tracking and Counting Method, Device, Computer Equipment, and Storage Medium

Abstract

The present application relates to an object tracking and counting method, device, computer device, and storage medium. The method includes: obtaining an object video stream, first performing object detection on the object video stream, and then performing tracking detection based on a conventional tracking algorithm to track the object. For the detected object bounding box data and object tracking bounding box data, Kalman filtering is used for tracking. Finally, linear tracking is performed based on the data obtained from Kalman tracking to count the number of objects moving to the target position. Throughout the process, object detection is used to identify the object, the conventional tracking algorithm is used to determine the position of the object in the next step, and Kalman filtering is used to obtain the movement speed of the object. It can accurately perform linear tracking on the object. The entire process does not need to rely on complex trained deep learning models and continuous feature-based tracking. The implementation process is simple, and efficient and accurate object tracking and counting can be achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and particularly to an object tracking and counting method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of image processing technologies, current image processing technologies have been widely applied in production and life, bringing convenience to people. For example, image processing technologies can be applied to object tracking and counting. An image acquisition device is used to acquire object images, and a conventional tracking and counting method can be used to count the number of objects within a certain period of time.

[0003] The above object tracking and counting method is most commonly applied to the express delivery industry. In the transfer yard of the express delivery industry, estimating the package flow of the belt conveyor has great application significance. For example, it can be used to calculate the working efficiency of the transfer yard, improve the energy efficiency of equipment and personnel, etc. The most important task in estimating the package flow of the belt conveyor is to count the number of packages within a certain period of time. The method based on machine vision has been widely applied due to its convenient installation, high accuracy, and low cost.

[0004] However, traditional object tracking and counting solutions cannot balance counting accuracy and counting efficiency, and the implementation process is complex. Summary of the Invention

[0005] Based on this, to address the above technical problems, it is necessary to provide an accurate and efficient object tracking and counting method, apparatus, computer device, and storage medium.

[0006] An object tracking and counting method, the method includes:

[0007] Obtain an object video stream;

[0008] Perform object detection on the object video stream to obtain object bounding box data of the detection output;

[0009] Track the object according to the object bounding box data of the detection output by using a preset tracking algorithm to obtain object tracking bounding box data;

[0010] Use Kalman filtering based on the object bounding box data of the detection output and the object tracking bounding box data to obtain Kalman tracking bounding box data;

[0011] Perform linear tracking according to the Kalman tracking bounding box data to count the number of objects moving to the target position.

[0012] In one embodiment, performing object detection on the object video stream to obtain object bounding box data of the detection output includes:

[0013] Perform object detection on the object video stream to obtain the initial bounding box data of the detected objects;

[0014] Obtain historical bounding box data, which is the bounding box data of historical record objects during object detection;

[0015] Construct an IoU matrix between the initial bounding box data of the object and the historical bounding box data;

[0016] Perform bipartite graph matching on the IoU matrix;

[0017] Update the historical bounding box data according to the bipartite graph matching result and the preset constraint threshold to obtain the object bounding box data of the detection output.

[0018] In one embodiment, updating the historical bounding box data according to the bipartite graph matching result and the preset constraint threshold to obtain the object bounding box data of the detection output includes:

[0019] Obtain the matching result value corresponding to the bipartite graph matching result;

[0020] If the matching result value is less than the preset constraint threshold, it is determined that a new object is detected, and the bounding box data corresponding to the new object is added to the historical bounding box data to obtain the object bounding box data of the detection output;

[0021] If the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the object bounding box data of the detection output.

[0022] In one embodiment, performing bipartite graph matching on the IoU matrix includes:

[0023] Perform bipartite graph matching on the IoU matrix using the Hungarian algorithm.

[0024] In one embodiment, performing object detection on the object video stream includes:

[0025] Perform object tracking detection on the object video stream based on a neural network detection model.

[0026] In one embodiment, obtaining the Kalman tracking bounding box data according to the object bounding box data of the detection output and the object tracking bounding box data using Kalman filtering includes:

[0027] Based on the object bounding box data and object tracking bounding box data output by the detection, the (x, y, w, h, x*, y*, w*, h*) corresponding to each object is obtained through a Kalman filter, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box;

[0028] Assign (x, y, w, h) to the bounding box data when starting Kalman tracking for the historical record object, and assign (x*, y*, w*, h*) to the Kalman parameters to obtain the Kalman tracking bounding box data.

[0029] In one of the embodiments, linear tracking is performed based on the Kalman tracking bounding box data, and counting the number of objects moving to the target position includes:

[0030] Calculate the new bounding box data of the historical record object according to the bounding box data when starting Kalman tracking for the historical record object and the Kalman parameters;

[0031] In the next frame when starting Kalman tracking, re-use the new bounding box data of the historical record object as the bounding box data when starting Kalman tracking for the historical record object, and return to the step of calculating the new bounding box data of the historical record object according to the bounding box data when starting Kalman tracking for the historical record object and the Kalman parameters to linearly track the movement of the object;

[0032] Count the number of objects moving to the target position according to the new bounding box data of the historical record object at different time frames.

[0033] An object tracking and counting device, the device includes:

[0034] A video stream acquisition module for acquiring an object video stream;

[0035] An object detection module for performing object detection on the object video stream to obtain the object bounding box data output by the detection;

[0036] An object tracking module for tracking an object according to the object bounding box data output by the detection and using a preset tracking algorithm to obtain the object tracking bounding box data;

[0037] A Kalman tracking module for obtaining the Kalman tracking bounding box data according to the object bounding box data output by the detection and the object tracking bounding box data and using Kalman filtering;

[0038] A linear tracking module for performing linear tracking based on Kalman tracking bounding box data and counting the number of objects moving to the target position.

[0039] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0040] Obtain an object video stream;

[0041] Perform object detection on the object video stream to obtain the object bounding box data of the detection output;

[0042] Track the object according to the object bounding box data of the detection output using a preset tracking algorithm to obtain the object tracking bounding box data;

[0043] Use Kalman filtering based on the object bounding box data of the detection output and the object tracking bounding box data to obtain the Kalman tracking bounding box data;

[0044] Perform linear tracking based on the Kalman tracking bounding box data and count the number of objects moving to the target position.

[0045] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0046] Obtain an object video stream;

[0047] Perform object detection on the object video stream to obtain the object bounding box data of the detection output;

[0048] Track the object according to the object bounding box data of the detection output using a preset tracking algorithm to obtain the object tracking bounding box data;

[0049] Use Kalman filtering based on the object bounding box data of the detection output and the object tracking bounding box data to obtain the Kalman tracking bounding box data;

[0050] Perform linear tracking based on the Kalman tracking bounding box data and count the number of objects moving to the target position.

[0051] The above object tracking and counting method, device, computer device and storage medium obtain an object video stream, first perform object detection on the object video stream, and then perform tracking detection based on a conventional tracking algorithm to track the object. For the obtained object bounding box data and object tracking bounding box data, Kalman filtering is used for tracking, and finally linear tracking is performed based on the data obtained by Kalman tracking to count the number of objects moving to the target position. Throughout the process, objects are identified by object detection, the positions of the next objects are determined by a conventional tracking algorithm, and the movement speed of the objects is obtained by means of Kalman filtering. It can accurately perform linear tracking on objects. The entire process does not need to rely on a complex pre-trained deep learning model and continuous feature-based tracking, and the implementation process is simple, and efficient and accurate object tracking and counting can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 FIG. is an application environment diagram of the object tracking and counting method in an embodiment;

[0053] Figure 2 FIG. is a schematic flowchart of the object tracking and counting method in an embodiment;

[0054] Figure 3 FIG. is a schematic flowchart of the object tracking and counting method in another embodiment;

[0055] Figure 4 FIG. is a structural block diagram of the object tracking and counting device in an embodiment;

[0056] Figure 5 FIG. is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The object tracking and counting method provided by the present application can be applied, for example, as Figure 1In the application environment shown. Among them, the terminal 102 is used to count the packages on the belt conveyor 104 in the sorting device. The terminal 102 acquires the object video stream on the belt conveyor 104, performs object detection on the object video stream, and obtains the object bounding box data output by the detection; tracks the object according to the object bounding box data output by the detection and using a preset tracking algorithm. Among them, the preset tracking algorithm can be an existing or conventional tracking algorithm adopted according to actual needs, and obtains the object tracking bounding box data; performs Kalman filtering according to the object bounding box data output by the detection and the object tracking bounding box data to obtain the Kalman tracking bounding box data; performs linear tracking according to the Kalman tracking bounding box data to count the number of objects moving to the destination position (the other end of the belt conveyor). Among them, the terminal 102 can be but is not limited to various devices with data processing functions, such as computers, notebooks, host computers, etc.

[0059] In one embodiment, as Figure 2 shown, a method for object tracking and counting is provided. Taking the terminal in Figure 1 as an example, the method includes the following steps:

[0060] S100: Acquire the object video stream.

[0061] The terminal can acquire the object video stream by using the image acquisition function, or can directly receive the object video stream collected by an external image acquisition device. Specifically, the terminal can be built-in with an image acquisition component, such as a camera and other components, to capture the video image of the moving object and obtain the object video stream. In actual application in Figure 1 the scenario shown, the camera in the terminal acquires the video image data of the packages moving on the belt conveyor to obtain the package video stream. Generally speaking, the movement trajectory of the package on the belt conveyor is an approximate uniform linear motion (that is, from one end of the belt conveyor to the other end). When the package reaches the other end, the counter is incremented by 1 to complete the final package tracking and counting.

[0062] S200: Perform object detection on the object video stream to obtain the object bounding box data output by the detection.

[0063] The terminal performs object detection on the object video stream, identifies the objects that appear in each frame of the video stream data, and extracts the corresponding object bounding box data in the image frame data for the identified objects. Specifically, the object detection of the object video stream can be performed through a preset classification model to identify the objects that appear in the video image frame. There is only one recognition category in this preset classification model, that is, a preset target category is set in the preset classification model. For example, the target category is an express parcel. In this way, after performing object detection on the object video frame stream through the preset classification model, the bounding box data of the express parcel can be detected. The bounding box data can be understood as the data used to describe the position where the object contour is located, and specifically it can include the abscissa and ordinate of the center of the bounding box, the width of the bounding box, and the height of the bounding box. The detected object bounding box data refers to the bounding box data corresponding to the object when it is first detected during the object detection of the object video stream, and it can be understood as the first frame bounding box data corresponding to the object.

[0064] In practical applications, taking the package detection on a belt conveyor as an example, the package detection is performed on the package video stream on the belt conveyor to determine the packages that appear in each frame of the video image, and the bounding box data of these packages is extracted, that is, the data such as the outer shape volume of these packages is extracted, specifically including the abscissa and ordinate of the center of the package and the width and height of the package. Optionally, after performing object detection, the detected object bounding box data can be cached in a temporary cache space. All detected objects will be recorded. When the object reaches the destination position, the object will be deleted from the cache space. As time goes by, the data in this temporary cache space is continuously updated, and these data include all the detected objects since the recording.

[0065] S300: Track the object according to the detected object bounding box data by using a preset tracking algorithm to obtain the object tracking bounding box data.

[0066] The preset tracking algorithm is an object tracking algorithm set in advance, which can specifically be an existing or conventional tracking algorithm, such as the common CSRT (Discriminative Correlation Filter) object tracking algorithm. Based on the object bounding box data output by detection, the currently detected objects can be determined, and the preset tracking algorithm is used to track all these detected objects to obtain object tracking bounding box data. The object tracking bounding box data refers to the object tracking bounding box data obtained by real-time tracking of the object on the basis of the object bounding box data output by detection after using the preset tracking algorithm. Specifically, the object bounding box data output by detection can be understood as the first-frame bounding box data of the object obtained after detecting each object in the object video stream. For a single object A, after obtaining the first-frame bounding box data of object A, the preset tracking algorithm is used with this as the "starting point" to track object A. This tracking is a process of using conventional feature tracking to obtain the subsequent (at least the second frame) bounding box data of object A, that is, the object tracking bounding box data.

[0067] S400: Based on the object bounding box data output by detection and the object tracking bounding box data, use Kalman filtering to obtain Kalman tracking bounding box data.

[0068] For each detected object, based on the object bounding box data output by detection and the object tracking bounding box data obtained by using a conventional tracking algorithm, use Kalman filtering to calculate the motion data of the object. The motion data includes the bounding box data of the object in the current frame and the motion speed data of the object, and obtain Kalman tracking bounding box data. The bounding box data of the current frame refers to the bounding box data at the moment when Kalman tracking of the object is started. Optionally, Kalman filtering can be implemented through a Kalman filter. As described above, for the same object A, S200 obtains the first-frame bounding box data of object A, S300 uses the preset tracking algorithm with the first-frame bounding box data of object A as the "starting point" and uses feature tracking to track and obtain the second-frame bounding box data of object A, that is, obtain the object tracking bounding box data of object A. Based on the first-frame bounding box data of object A and the second-frame bounding box data of object A, the motion data of object A can be further analyzed. Here, use Kalman filtering to obtain Kalman tracking bounding box data. In other words, the Kalman tracking bounding box data contains the motion data of the object and the bounding box data corresponding to the object when Kalman tracking is started.

[0069] S500: Perform linear tracking based on the Kalman tracking bounding box data and count the number of objects that move to the target position.

[0070] In the Kalman tracking bounding box data, the bounding box data of all objects at the start of Kalman tracking and the motion speed data of the objects have been determined. Using a linear tracking method, the motion trajectories of the objects are continuously tracked. When an object moves to the target position, the number of objects moving to the target position is incremented by 1. Specifically, for a conveyor belt, based on the Kalman tracking bounding box data, the detected packages on the conveyor belt are linearly tracked, and the motion trajectories of these packages are tracked. When the bounding box data of a package indicates that it has reached the other end (target position) of the conveyor belt, the package count is incremented by 1.

[0071] The above object tracking and counting method obtains an object video stream. First, object detection is performed on the object video stream, and then tracking detection based on a conventional tracking algorithm is performed to track the object. For the obtained object bounding box data and object tracking bounding box data, Kalman filtering is used for tracking. Finally, linear tracking is performed based on the data obtained from Kalman tracking, and the number of objects moving to the target position is counted. Throughout the process, objects are identified through object detection, the positions of the objects in the next step are determined through a conventional tracking algorithm, and the motion speed of the objects is obtained using Kalman filtering. It can accurately perform linear tracking on objects. The entire process does not rely on a complex pre-trained deep learning model and continuous feature-based tracking. The implementation process is simple, and efficient and accurate object tracking and counting can be achieved.

[0072] As Figure 3 shown, in one embodiment, S200 includes:

[0073] S210: Perform object detection on the object video stream to obtain the initial object bounding box data of the detection output.

[0074] Perform object detection on the object video stream to identify the objects currently appearing in the video stream, and obtain the initial object bounding box data corresponding to these objects. Specifically, object detection can be performed on the object video stream through a neural network model, that is, the object video stream is input into the neural network model. There is a unique classification category set in the neural network model, and this category is the target category object. For example, if package tracking and counting are required, the package is set as the target classification category in the neural network model, and the initial object bounding box data detected and output by the neural network model is obtained. Further, the neural network model is a convolutional neural network model, and the category it sets is express package. That is, object detection is performed on the video stream through a pre-trained convolutional neural network model for identifying express packages, and the initial bounding box data of the express packages appearing in the current video stream is identified and obtained.

[0075] S220: Obtain historical bounding box data, where the historical bounding box data is the bounding box data of the object recorded during historical object detection.

[0076] A historical record object refers to an object detected from an object video stream in the historical record. For example, for a segment of an object video stream, objects such as Object 1, Object 2, Object 3, Object A, and Object B are detected in sequence. These historical record objects include Object 1, Object 2, Object 3, Object A, and Object B. The bounding box data of a historical record object during object detection refers to the current frame bounding box data corresponding to the object during object detection, or simply understood as the bounding box data of the first frame when the object is detected (captured). This first frame is calculated independently for each object, and the corresponding first frame moment is different for different objects. For each object, the moment when it is first detected from the object video stream is its first frame. Cache all the bounding box data of historical record objects during object detection, and directly obtain this cached data here to enter the next step of processing, in order to identify whether there is bounding box data of a "new object" in the currently detected output object initial bounding box data, that is, whether there is the bounding box data of the first frame of a "new object".

[0077] S230: Construct an IoU matrix between the object initial bounding box data and the historical bounding box data.

[0078] Since the same object in the video stream may be repeatedly detected at different frames (for example, the object stays briefly), in order to avoid subsequent repeated tracking and counting, it is necessary to construct an IoU matrix here to identify whether there is matching content between the detected output object initial bounding box data and the historical record bounding box data. Specifically, as described above, for each object detected each time (each frame), it is stored in a temporary cache space. The objects detected in the historical record are stored in this cache space. After obtaining the object initial bounding box data detected and output in S210, it is necessary to further identify whether there is matching content between the currently detected output object initial bounding box data and the historical bounding box data in the temporary cache space. Assume that the object initial bounding box data detected and output contains the bounding box data of M objects, and the current historical record objects are N objects, that is, the historical bounding box data is the bounding box data of N objects during object detection. Then the IoU matrix is M * N.

[0079] S240: Perform bipartite graph matching on the IoU matrix.

[0080] S250: Update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data detected and output.

[0081] The Xiongya algorithm can be used to perform bipartite graph matching on the IoU matrix, and a constraint is added, that is, a preset constraint threshold is obtained. According to the comparison result between the bipartite graph matching result and the preset constraint threshold, it is determined whether there is content in the initial bounding box data of the detected object that matches the historical bounding box data, that is, whether the detected object belongs to the object in the historical record (previously detected), and whether there is a new object, so as to update the bounding box data of the detected object during object detection and obtain the bounding box data of the detected object. Specifically, the preset constraint threshold is a preset value, which can be set according to the actual needs, for example, it can be set to 0.5.

[0082] In one embodiment, according to the bipartite graph matching result and the preset constraint threshold, updating the historical bounding box data of the detected object to obtain the bounding box data of the detected object includes:

[0083] Obtain the matching result value corresponding to the bipartite graph matching result; if the matching result value is less than the preset constraint threshold, it is determined that a new object is detected, and the bounding box data corresponding to the new object is added to the historical bounding box data to obtain the bounding box data of the detected object; if the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the bounding box data of the detected object.

[0084] If the matching result value is less than the preset constraint threshold, it indicates that there is unmatched data in the initial bounding box data of the detected object, that is, it is considered a newly emerged object. The initial bounding box data corresponding to the entire newly emerged object is updated to the historical bounding box data to obtain the object bounding box data of the detection output. For example, the historical bounding box data contains the bounding box data of objects A, B, and C during object detection, and the initial bounding box data of the detection output contains the bounding box data of object D. At this time, the matching result value is less than the preset constraint threshold, and the bounding box data of object D is updated to the historical bounding box data to obtain the bounding box data of A, B, C, and D during object detection, that is, the object bounding box data of the output is obtained. If the matching result is not less than the preset constraint threshold, it indicates that there is a high degree of matching between the initial bounding box data of the detected object and the historical bounding box data, that is, it is considered that there is no newly emerged object, and the currently detected object is determined to be an object that has been detected in history. For the same object detected this time, the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the object bounding box data of the detection output. For example, the historical bounding box data contains the bounding box data of objects A, B, and C during object detection, and the initial bounding box data of the detection output contains the bounding box data of object C. At this time, the matching result value is not less than the preset constraint threshold, the same object C is determined, and the bounding box data corresponding to object C during object detection in the historical bounding box data is updated, that is, the current bounding box data of object C is replaced with the bounding box data of object C, to obtain the object bounding box data of the detection output.

[0085] In one embodiment, according to the object bounding box data of the detection output and the object tracking bounding box data, using the Kalman filter, the Kalman tracking bounding box data obtained includes:

[0086] According to the object bounding box data of the detection output and the object tracking bounding box data, each object corresponding (x, y, w, h, x*, y*, w*, h*) is obtained through the Kalman filter, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; (x, y, w, h) is assigned to the bounding box data when starting Kalman tracking for the historical record object, and (x*, y*, w*, h*) is assigned to the Kalman parameters to obtain the Kalman tracking bounding box data.

[0087] For each object detected in the historical record, based on the object bounding box data and object tracking bounding box data output by the detection, the Kalman filter can be used to obtain the (x, y, w, h, x*, y*, w*, h*) of the object. This data is the Kalman tracking bounding box data of each object, where (x, y, w, h) is the bounding box data of the object in the historical record when starting Kalman tracking, including the center abscissa, ordinate of the bounding box, as well as the height and width of the bounding box; where (x*, y*, w*, h*) are the Kalman parameters used to characterize the motion change of the object. For an object moving at a constant speed, this data characterizes the constant speed of the object. Specifically, (x, y, w, h) can be understood as the observation variables of the Kalman filter, and (x, y, w, h, x*, y*, w*, h*) can be understood as the state variables of the Kalman filter. In the next step, linear tracking of the object in the historical record will be performed based on these two parts of data.

[0088] In one embodiment, linear tracking is performed according to the Kalman tracking bounding box data, and counting the number of objects moving to the target position includes:

[0089] Calculate the new bounding box data of the object in the historical record according to the bounding box data and Kalman parameters when starting Kalman tracking of the object in the historical record; in the next frame after starting Kalman tracking, re-use the new bounding box data of the object in the historical record as the bounding box data when starting Kalman tracking of the object in the historical record, and return the step of calculating the new bounding box data of the object in the historical record according to the bounding box data and Kalman parameters when starting Kalman tracking of the object in the historical record, so as to linearly track the motion of the object; count the number of objects moving to the target position according to the new bounding box data of the object in the historical record at different time frames.

[0090] Linear tracking can be understood as a process of continuously updating the tracking over time, where the Kalman parameters characterize the motion changes of an object. Based on the bounding box data and Kalman parameters at the start of Kalman tracking, new bounding box data can be calculated. When reaching the next frame of starting Kalman tracking, the new bounding box data is re - used as the bounding box data of the historical record object at the start of Kalman tracking. Return the step of calculating the new bounding box data of the historical record object based on the bounding box data and Kalman parameters when starting Kalman tracking for the historical record object. Through continuous linear tracking according to the motion of the object until it is found that the object has moved to the target position based on the latest bounding box data of the object, count the number of objects that have moved to the target position. By accumulating the number of objects that have moved to the target position at different time frames, count the number of objects that have moved to the target position. Taking the packages on the belt conveyor as an example, for each package, calculate the new bounding box data (x', y', w', h') using the bounding box data (x, y, w, h) and Kalman parameters (x*, y*, w*, h*) at the start of Kalman tracking. The specific calculation formula is: x' = x + x*; y' = y + y*; w' = w + w*; h' = h + h*. Re - use the newly calculated bounding box data as the bounding box data at the start of Kalman tracking until the latest obtained bounding box data indicates that the package has moved to the target position, that is, the x and y coordinates in the latest bounding box data of the package are greater than the coordinate values at the other end of the belt conveyor, then the number of packages increases by 1. Optionally, for the objects that have moved to the target position, they can be deleted from the temporary cache space to avoid unnecessary resource occupation.

[0091] Specifically, in this application, the object bounding box data refers to the bounding box data corresponding to the object when it is first detected in the object video stream, that is, the first - frame bounding box data of the object. As time goes by, the object is moving, and a preset tracking algorithm is used to track the object to obtain the second - frame bounding box data of the object. After obtaining at least 2 - frame bounding box data of the object, the Kalman filtering method can be used to analyze and obtain the motion data of the object. Taking the second - frame bounding box data as the starting point, based on the analyzed motion data of the object, Kalman tracking is performed on the object to obtain the third - frame bounding box data, the fourth - frame bounding box data, until it is detected that the object has moved to the target position. It can be understood that the second - frame bounding box data mentioned above is the bounding box data of the object at the start of Kalman tracking.

[0092] To further explain in detail the technical solution and its effect of the object tracking and counting method in this application, taking the packages on the belt conveyor as an example, the whole process will be described in detail using a computer programming language, which specifically includes the following implementation stages:

[0093] Premise assumption: The movement trajectory of the package on the belt conveyor is an approximately uniform linear motion (i.e., from one end of the belt conveyor to the other end), and the counter is incremented by 1 when the package reaches the other end. In the algorithm, a structure represents a package.

[0094] I. Initialization

[0095] Load the model parameters of the detection algorithm, initialize a counter to 0, and initialize the data in a buffer (temporary cache space) to be empty. The data in this buffer can only be the structure of the package.

[0096] II. Model Detection and Processing

[0097] The model used is yolov3, a detection model based on convolutional neural network, with only one category, i.e., the package. The output of the model is the bounding box data of all detected packages during object detection. The form of the bounding box data is (x, y, w, h), which represent the abscissa of the center of the bounding box data, the ordinate of the center, the width, and the height respectively. First, construct an iou matrix using the bounding box data of the currently detected packages and the bounding box data included in the historical records during object detection. Assume the number of currently detected packages is M and the number of historically detected packages is N, then the dimension of the iou matrix is M*N, and the number in the mn-th dimension of the matrix represents the iou between the m-th currently detected package and the n-th historically detected package. Secondly, use the Hungarian algorithm to perform bipartite graph matching on the iou matrix, and add a constraint that only ious greater than or equal to a certain threshold (e.g., 0.5) can be matched. For the packages output by the detection algorithm that are not successfully matched, they are considered as newly emerged packages, and a structure is initialized and added to the buffer; for the two successfully matched packages, they are considered as the same package.

[0098] III. Traditional Tracking

[0099] Track all the packages in the buffer using traditional tracking methods (such as CSRT) to obtain the new bounding box data of all the packages, and assign them to the bounding box data of all the packages in the buffer after traditional tracking.

[0100] IV. Kalman Tracking

[0101] For each package in the buffer, using the bounding box data during object detection and the bounding box data after traditional tracking, the Kalman filter is used to calculate the (x, y, w, h, x*, y*, w*, h*) of the package. (x, y, w, h) is assigned to the bounding box data of the package when starting Kalman tracking, and (x*, y*, w*, h*) is assigned to the Kalman parameters. Among them, the observation variables of the Kalman filter are (x, y, w, h), and the state variables are (x, y, w, h, x*, y*, w*, h*). x*, y*, w*, h* respectively represent the speed of the abscissa of the center of the package bounding box, the speed of the ordinate of the center, the speed of the width, and the speed of the height.

[0102] V. Linear Tracking

[0103] Calculate the new package bounding box data (x’, y’w’, h’) using the package bounding box data (x, y, w, h) when starting Kalman tracking and the Kalman parameters (x*, y*, w*, h*). Determine whether there is a package in the historical record package that reaches the other end of the conveyor belt. If it reaches, the counter is incremented by 1, and this package is deleted from the historical record package.

[0104] When actually applying the object tracking and counting method of this application, the above-mentioned stage two to stage five can be implemented by an independent processing module respectively, that is, including a detection module, a traditional tracking module, a Kalman tracking module, and a linear tracking module. These modules each implement the corresponding functions above, forming a complete processing flow. One module processes one video frame or one picture. Assuming the length of the processing flow is N video frames, then there are N modules. The first three modules are the detection module, the traditional tracking module, and the Kalman tracking module respectively, and the latter 4 - N modules are all linear tracking modules. In actual application, through a large number of experiments and attempts, generally N = 6 is taken, and the video stream is 6 frames per second, and the speed and accuracy reach the best, that is, one processing flow is 1 second, with 1 detection module, traditional tracking module, Kalman tracking module, and 3 linear tracking modules.

[0105] It should be understood that although each step in the above flowcharts is shown in sequence according to the arrow indication, these steps do not necessarily need to be executed in the order indicated by the arrow. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0106] In addition, as Figure 4 shown, the present application further provides an object tracking and counting device, which includes:

[0107] A video stream acquisition module 100 for acquiring an object video stream;

[0108] An object detection module 200 for performing object detection on the object video stream to obtain object bounding box data output by the detection;

[0109] An object tracking module 300 for tracking the object according to the object bounding box data output by the detection and using a preset tracking algorithm to obtain object tracking bounding box data;

[0110] A Kalman tracking module 400 for performing Kalman filtering according to the object bounding box data output by the detection and the object tracking bounding box data to obtain Kalman tracking bounding box data;

[0111] A linear tracking module 500 for performing linear tracking according to the Kalman tracking bounding box data to count the number of objects moving to the target position.

[0112] The above object tracking and counting device acquires an object video stream, first performs object detection on the object video stream, and then performs tracking detection based on a conventional tracking algorithm to track the object. Kalman filtering is used for tracking according to the obtained object bounding box data and object tracking bounding box data. Finally, linear tracking is performed based on the data obtained by Kalman tracking to count the number of objects moving to the target position. During the whole process, objects are recognized by object detection, the positions of the next objects are determined by a conventional tracking algorithm, and the moving speed of the objects is obtained by using Kalman filtering. It can accurately perform linear tracking on objects. The whole process does not need to rely on a complex trained deep learning model and continuous feature-based tracking, and the implementation process is simple, and efficient and accurate object tracking and counting can be achieved.

[0113] In one embodiment, the object detection module 200 is further configured to perform object detection on the object video stream to obtain object initial bounding box data output by the detection; obtain historical bounding box data, where the historical bounding box data is the bounding box data of the object recorded in history during object detection; construct an IoU matrix between the object initial bounding box data and the historical bounding box data; perform bipartite graph matching on the IoU matrix; update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data output by the detection.

[0114] In one embodiment, the object detection module 200 is further configured to obtain a matching result value corresponding to the bipartite graph matching result; if the matching result value is less than a preset constraint threshold, it is determined that a new object is detected, and then the bounding box data corresponding to the new object is added to the historical bounding box data to obtain the object bounding box data of the detection output; if the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the object bounding box data of the detection output.

[0115] In one embodiment, the object detection module 200 is further configured to perform bipartite graph matching on the IoU matrix using the Hungarian algorithm.

[0116] In one embodiment, the object detection module 200 is further configured to perform object tracking detection on the object video stream based on a neural network detection model.

[0117] In one embodiment, the Kalman tracking module 400 is further configured to obtain (x, y, w, h, x*, y*, w*, h*) corresponding to each object through a Kalman filter according to the object bounding box data of the detection output and the object tracking bounding box data, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; assign (x, y, w, h) to the bounding box data when starting Kalman tracking for the historical record object, and assign (x*, y*, w*, h*) to the Kalman parameters to obtain the Kalman tracking bounding box data.

[0118] In one embodiment, the linear tracking module 500 is configured to calculate new bounding box data of the historical record object according to the bounding box data when starting Kalman tracking for the historical record object and the Kalman parameters; in the next frame after starting Kalman tracking, re-use the new bounding box data of the historical record object as the bounding box data when starting Kalman tracking for the historical record object, and return the operation of calculating new bounding box data of the historical record object according to the bounding box data when starting Kalman tracking for the historical record object and the Kalman parameters to linearly track the movement of the object; count the number of objects moving to the target position according to the new bounding box data of the historical record object at different time frames.

[0119] For the specific limitations of the object tracking and counting device, reference may be made to the limitations of the object tracking and counting method in the foregoing text, which will not be elaborated herein. Each module in the above object tracking and counting device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0120] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 5 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as historical object tracking data and pre-configured data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an object tracking and counting method.

[0121] Those skilled in the art can understand that Figure 5 the structure shown in

[0122] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0123] Obtain an object video stream;

[0124] Perform object detection on the object video stream to obtain the object bounding box data of the detection output;

[0125] Track the object according to the object bounding box data of the detection output and using a preset tracking algorithm to obtain the object tracking bounding box data;

[0126] According to the object bounding box data of the detection output and the object tracking bounding box data, use Kalman filtering to obtain the Kalman tracking bounding box data;

[0127] Perform linear tracking according to the Kalman tracking bounding box data and count the number of objects moving to the target position.

[0128] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0129] Perform object detection on the object video stream to obtain the initial bounding box data of the detected objects; obtain historical bounding box data, where the historical bounding box data is the bounding box data of the objects recorded in history during object detection; construct an IoU matrix between the initial bounding box data of the objects and the historical bounding box data; perform bipartite graph matching on the IoU matrix; update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data of the detected output.

[0130] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0131] Obtain the matching result value corresponding to the bipartite graph matching result; if the matching result value is less than the preset constraint threshold, it is determined that a new object is detected, and the bounding box data corresponding to the new object is added to the historical bounding box data to obtain the object bounding box data of the detected output; if the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the object bounding box data of the detected output.

[0132] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0133] Perform bipartite graph matching on the IoU matrix using the Hungarian algorithm.

[0134] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0135] Perform object tracking detection on the object video stream based on a neural network detection model.

[0136] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0137] Based on the object bounding box data and object tracking bounding box data output by detection, the (x, y, w, h, x*, y*, w*, h*) corresponding to each object are obtained through a Kalman filter, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; (x, y, w, h) is assigned to the bounding box data when starting Kalman tracking for the historical record object, and (x*, y*, w*, h*) is assigned to the Kalman parameters to obtain the Kalman tracking bounding box data.

[0138] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0139] According to the bounding box data and Kalman parameters when starting Kalman tracking for the historical record object, calculate the new bounding box data of the historical record object; in the next frame when starting Kalman tracking, re-use the new bounding box data of the historical record object as the bounding box data when starting Kalman tracking for the historical record object, and return to the step of calculating the new bounding box data of the historical record object according to the bounding box data and Kalman parameters when starting Kalman tracking for the historical record object to linearly track the movement of the object; according to the new bounding box data of the historical record object at different time frames, count the number of objects moving to the target position.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0141] Obtain an object video stream;

[0142] Perform object detection on the object video stream to obtain the object bounding box data output by detection;

[0143] According to the object bounding box data output by detection, track the object using a preset tracking algorithm to obtain the object tracking bounding box data;

[0144] According to the object bounding box data and object tracking bounding box data output by detection, use Kalman filtering to obtain the Kalman tracking bounding box data;

[0145] Perform linear tracking according to the Kalman tracking bounding box data and count the number of objects moving to the target position.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0147] Perform object detection on the object video stream to obtain the initial bounding box data of the detected objects; obtain the historical bounding box data, where the historical bounding box data is the bounding box data of the historical record objects during object detection; construct an IoU matrix between the initial bounding box data of the objects and the historical bounding box data; perform bipartite graph matching on the IoU matrix; update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data of the detection output.

[0148] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0149] Obtain the matching result value corresponding to the bipartite graph matching result; if the matching result value is less than the preset constraint threshold, it is determined that a new object is detected, and the bounding box data corresponding to the new object is added to the historical bounding box data to obtain the object bounding box data of the detection output; if the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and the bounding box data corresponding to the same object in the historical bounding box data is updated to obtain the object bounding box data of the detection output.

[0150] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0151] Use the Hungarian algorithm to perform bipartite graph matching on the IoU matrix.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0153] Perform object tracking detection on the object video stream based on a neural network detection model.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0155] According to the object bounding box data of the detection output and the object tracking bounding box data, obtain (x, y, w, h, x*, y*, w*, h*) corresponding to each object through a Kalman filter, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; assign (x, y, w, h) to the bounding box data when starting Kalman tracking for the historical record objects and assign (x*, y*, w*, h*) to the Kalman parameters to obtain the Kalman tracking bounding box data.

[0156] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0157] Calculate the new bounding box data of the historical object based on the bounding box data and Kalman parameters when starting Kalman tracking for the historical object; in the next frame when starting Kalman tracking, re-use the new bounding box data of the historical object as the bounding box data when starting Kalman tracking for the historical object, and return the step of calculating the new bounding box data of the historical object based on the bounding box data and Kalman parameters when starting Kalman tracking for the historical object to linearly track the movement of the object; count the number of objects moving to the target position according to the new bounding box data of the historical object in frames at different times.

[0158] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0159] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0160] The above embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An object tracking and counting method, characterized in that, The method includes: Obtain the object video stream; Perform object detection on the object video stream to obtain the object bounding box data of the detection output; Track the object according to the object bounding box data of the detection output and using a preset tracking algorithm to obtain the object tracking bounding box data; Perform Kalman filtering based on the object bounding box data of the detection output and the object tracking bounding box data to obtain the Kalman tracking bounding box data; Perform linear tracking according to the Kalman tracking bounding box data and count the number of objects moving to the target position; Among them, the performing Kalman filtering based on the object bounding box data of the detection output and the object tracking bounding box data to obtain the Kalman tracking bounding box data includes: According to the object bounding box data of the detection output and the object tracking bounding box data, obtain (x, y, w, h, x*, y*, w*, h*) corresponding to each object through a Kalman filter, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; Assign (x, y, w, h) to the bounding box data when starting Kalman tracking for the historical record object and assign (x*, y*, w*, h*) to the Kalman parameters to obtain the Kalman tracking bounding box data.

2. The method according to claim 1, wherein The performing object detection on the object video stream to obtain the object bounding box data of the detection output includes: Perform object detection on the object video stream to obtain the initial object bounding box data of the detection output; Obtain the historical bounding box data, where the historical bounding box data is the bounding box data of the historical record object during object detection; Construct an IoU matrix between the initial object bounding box data and the historical bounding box data; Perform bipartite graph matching on the IoU matrix; Update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data of the detection output.

3. The method according to claim 2, wherein The updating the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data of the detection output includes: Obtain the matching result value corresponding to the bipartite graph matching result; If the matching result value is less than the preset constraint threshold, it is determined that a new object is detected, and add the bounding box data corresponding to the new object to the historical bounding box data to obtain the object bounding box data of the detection output; If the matching result value is not less than the preset constraint threshold, it is determined that the same object is detected, and update the bounding box data corresponding to the same object in the historical bounding box data to obtain the object bounding box data of the detection output.

4. The method according to claim 2, wherein The performing bipartite graph matching on the IoU matrix includes: Perform bipartite graph matching on the IoU matrix using the Hungarian algorithm.

5. The method according to claim 1, wherein The performing object detection on the object video stream includes: Perform object tracking detection on the object video stream based on a neural network detection model.

6. The method according to claim 1, characterized in that, The performing linear tracking according to the Kalman tracking bounding box data and counting the number of objects moving to the target position includes: Calculate the new bounding box data of the historical object based on the bounding box data and Kalman parameters when starting Kalman tracking for the historical object; In the next frame when starting Kalman tracking, use the new bounding box data of the historical object as the bounding box data when starting Kalman tracking for the historical object again, and return the step of calculating the new bounding box data of the historical object based on the bounding box data and Kalman parameters when starting Kalman tracking for the historical object, so as to linearly track the movement of the object; Count the number of objects that have moved to the target position according to the new bounding box data of the historical object at different time frames.

7. An object tracking and counting device, characterized in that, The device includes: A video stream acquisition module for acquiring an object video stream; An object detection module for performing object detection on the object video stream to obtain the object bounding box data of the detection output; An object tracking module for tracking an object according to the object bounding box data of the detection output using a preset tracking algorithm to obtain the object tracking bounding box data, including: obtaining (x, y, w, h, x*, y*, w*, h*) corresponding to each object through a Kalman filter according to the object bounding box data of the detection output and the object tracking bounding box data, where x represents the abscissa of the center of the bounding box, y represents the ordinate of the center of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; x* represents the speed of the abscissa of the center of the bounding box, y* represents the speed of the ordinate of the center of the bounding box, w* represents the speed of the width of the bounding box, and h* represents the speed of the height of the bounding box; assign (x, y, w, h) to the bounding box data when starting Kalman tracking for the historical object and assign (x*, y*, w*, h*) to the Kalman parameters to obtain the Kalman tracking bounding box data; A Kalman tracking module for obtaining the Kalman tracking bounding box data by using Kalman filtering according to the object bounding box data of the detection output and the object tracking bounding box data; A linear tracking module for performing linear tracking according to the Kalman tracking bounding box data and counting the number of objects that have moved to the target position.

8. The device according to claim 7, characterized in that, The device includes: The object detection module is further configured to perform object detection on the object video stream to obtain the initial object bounding box data of the detection output; obtain the historical bounding box data, where the historical bounding box data is the bounding box data of the historical object during object detection; construct an iou matrix between the initial object bounding box data and the historical bounding box data; perform bipartite graph matching on the iou matrix; update the historical bounding box data according to the bipartite graph matching result and a preset constraint threshold to obtain the object bounding box data of the detection output.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images