A method for tracking, perceiving and recognizing a single person in a multi-target scenario

Abstract

The present invention discloses a method for tracking, sensing and identifying a single person in a multi-target situation, including: application of the FairMOT multi-target tracking algorithm in personnel tracking; point cloud filtering technology for obtaining the target object through a ZED camera, thereby obtaining the three-dimensional coordinate points of the camera system; multi-sensor data fusion based on multiple Kalman filters of the position data obtained by the camera and the position data of the UWB sensor to obtain the final person target position; based on the ZED camera and the UWB sensor, an original tracking strategy is used to track the target ID, which also includes target prediction, retrieval and re-identification algorithms in the event of target loss. The personnel tracking algorithm of the present invention can achieve better tracking effects with a small number of sensors, and can be well applied in the field of target tracking. The multiple Kalman filter data fusion method adopted by the present invention can also provide a case reference for multi-sensor data fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of tracking and identification, and in particular to a method for tracking, sensing and identifying a single person in a multi-target situation. Background Art

[0002] With technological advancements and economic development, people's demands for a better quality of life are increasing. As a hot topic in cutting-edge research, the development of intelligent robots has gradually shifted from industrial robots used to improve factory productivity to service robots that enhance the quality of life. With rising living standards and social productivity, more and more robots have become essential assistants in our daily lives. From sweeping robots and companion robots for personal use in the home to delivery robots and hotel robots for professional use, a growing number of service robots are helping to improve people's quality of life. Among service robots, automatic following is a very common function and is in high demand in many scenarios. Currently, many companies have applied it to scenarios such as supermarket shopping carts and suitcases. As a perception and recognition technology for target monitoring, it has many applications in both industrial environments and daily life. Mobile robots have broad application prospects in military, transportation, and service sectors. As application demands continue to evolve, the need for mobile robots to perform more advanced tasks is increasing.

[0003] Among the many mobile robotics technologies, target recognition and positioning provide powerful support for mobile robots to perform advanced tasks. Target recognition involves classifying and detecting targets through methods such as computer vision, image processing, pattern recognition, machine learning, and deep learning. This process primarily extracts image features as key information, matches or classifies the incoming information with stored key information, and ultimately generates a recognition result. Computer vision, an interdisciplinary field that integrates knowledge from computer science, biology, physics, and even psychology, is a subfield of computer vision technology. The basic idea behind visual recognition and positioning technology is to use a camera to capture visual information about the indoor environment, potentially including facial information of the target object. Then, based on a specific algorithm, the target object is identified from the image and its location is determined.

[0004] Existing people tracking perception methods mostly use a combination of vision and lidar, but this technology only uses a combination of vision and UWB to achieve target tracking, which can greatly reduce costs. In addition, existing people tracking methods are not well compatible with crowded and changing environments. This technology can achieve people tracking and re-identification in complex environments such as crowded and multi-corner environments, with strong robustness and portability. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a method for tracking, perceiving and identifying a single person in a multi-target situation. It is mainly used in the presence of multiple target objects to specifically track a specified target and continuously obtain its world coordinate points.

[0006] To achieve the above objectives, the specific technical solutions of the method for tracking, sensing, and identifying a single person in a multi-target situation of the present invention are as follows:

[0007] A method for tracking, perceiving, and identifying a single person in a multi-target situation includes the following steps:

[0008] S1. Data acquisition:

[0009] The ZED camera acquires the target image, and the UWB sensor acquires the distance and angle information between the UWB beacon and the base station;

[0010] S2. Data preprocessing:

[0011] The images acquired by the ZED camera are processed through the FairMOT multi-target recognition and tracking algorithm to obtain the position of each target in the image and select it. At the same time, each target is given an ID. Then, the box with the ID of the tracked target is taken and the point cloud within the box is obtained from the point cloud data of the ZED camera. Some low-confidence data in the point cloud data is eliminated, and data outside the measurement range is also deleted. After through-filtering and statistical filtering, dense point cloud information of target points and sparse point cloud information of background points are obtained. The point cloud is clustered to eliminate background points belonging to sparse points from the point cloud set to obtain a dense point cloud set of the target point cloud. Finally, the center of mass point of the target point cloud data is calculated to obtain the world coordinate point of the target object being tracked by the ZED camera.

[0012] The data information obtained by the UWB sensor is pre-processed through low-pass filtering and sliding window filtering to obtain the distance and angle data between the base station and the beacon, and then the distance and angle data are converted into two-dimensional world coordinate points;

[0013] S3, Data Fusion:

[0014] After obtaining the data processed by the ZED camera and the UWB sensor, the position data obtained by the ZED camera and the position data of the UWB sensor are fused by multi-sensor data based on multiple Kalman filters to obtain the final target position of the person;

[0015] S4. Target tracking:

[0016] After obtaining the target point data, it is necessary to determine the tracking target and track it subsequently, so initialization operations are required.

[0017] Furthermore, the ZED camera in step S1 obtains the RGBD information and point cloud data of the current image, and is equipped with C++, Python and ROS interfaces; the UWB sensor returns the distance and angle information of the beacon from the base station in real time at a frequency of 100 Hz. The effective measurement range of the UWB sensor is -75° to 75°.

[0018] Furthermore, when the FairMOT algorithm is applied in step S2, the ZED camera obtains the current target image, uses the DCNv2 model to identify the personnel, obtains the image position coordinates of all target personnel, selects them, assigns an ID number to each target, and performs ID matching on the personnel targets in subsequent images.

[0019] Furthermore, in step S3, multi-sensor data fusion based on multiple Kalman filters is performed. Specifically, the world coordinate point obtained by the ZED camera is fused with the world coordinate point obtained by the UWB. In the x-dimension, the data of the ZED camera and the UWB sensor are subjected to multiple Kalman filtering. In the y-dimension, only the data of the ZED camera is taken and processed by Kalman filtering to obtain a reliable target world coordinate point.

[0020] Furthermore, the initialization operation of the personnel tracking technology in step S4 is specifically as follows:

[0021] S4-1. Initialize the tracking target ID. After receiving the first image data, if there are multiple targets, the target ID closest to the UWB data point will be set as the initial tracking target.

[0022] S4-2. Considering the special circumstances of target loss and re-identification, the tracking ID may change. It is necessary to create a container for tracking target IDs. All targets in this container are tracked with a certain priority. The IDs in this container are updated by adding new target ID information when the target is lost, recovered, or re-identified. The priority is that the target added to the tracking target ID container first has the highest priority, and the most recent target ID has the lowest priority. Tracking always selects the target with the highest priority first.

[0023] S4-3. Notify the target data processing unit of the target ID to be tracked.

[0024] Furthermore, during the target tracking process in step S4, when FairMOT frame loss or temporary target loss occurs, it is necessary to perform target prediction processing and infer the target image position in the frame loss situation through historical image point cloud information.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] Existing personnel tracking perception methods mostly use the combination of vision and lidar, while this technology only uses the combination of vision and UWB to achieve target tracking, which can greatly reduce costs. In addition, existing personnel tracking methods are not very compatible with crowded and changeable environments. This technology can achieve personnel tracking and re-identification in crowded, multi-corner and other complex environments, with strong robustness and portability. The present invention is mainly based on computer vision, converting information from the image coordinate system to the desired coordinate system, and then using multi-sensor fusion technology to output the position in the world coordinate system, and then using the target tracking strategy to continuously predict the dynamic motion trajectory of the target. The personnel tracking algorithm of the present invention can achieve better tracking effects with a small number of sensors, and can be well applied in the field of target tracking. The multiple Kalman filter data fusion method adopted by the present invention can also provide a case reference for multi-sensor data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 This is a block diagram of the architecture of the present invention;

[0029] Figure 2 Schematic diagram of the effective recognition range of the ZED camera and UWB sensor in the present invention;

[0030] Figure 3 This is an effect diagram of multi-sensor data fusion in the present invention;

[0031] Figure 4 Flowchart of the present invention. DETAILED DESCRIPTION

[0032] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a single person tracking, perception and recognition method in a multi-target situation of the present invention in combination with the accompanying drawings and specific preferred embodiments.

[0033] Example 1:

[0034] See also Figure 1-4 The present invention provides a technical solution: a method for tracking, sensing and identifying a single person in a multi-target situation, comprising the following steps:

[0035] S1. Data acquisition:

[0036] The ZED camera acquires the target image, and the UWB sensor acquires the distance and angle information between the UWB beacon and the base station. The ZED camera and UWB sensor are the only two sensor devices used in this technology. They acquire the RGBD image of the current target and the distance and angle between the UWB beacon and the base station, respectively. The distance accuracy of the data acquired by UWB is relatively high, at the centimeter level, while the angle information is less accurate and requires appropriate filtering during use.

[0037] S2. Data preprocessing:

[0038] The image acquired by the ZED camera is processed by the FairMOT multi-target recognition and tracking algorithm to obtain the position of each target in the image and select it out. At the same time, each target will get an ID. Then, the frame of the tracked target ID is taken to obtain the point cloud in the frame from the point cloud data of the ZED camera. Some low-confidence data in the point cloud data are eliminated. The point cloud data is calculated by the binocular image information of the camera. At this time, the image information will be affected by some factors such as lighting, the distance is too far, resulting in a small amount of data and difficulty in calculation. Therefore, the calculated point cloud data information will also have corresponding uncertainties, and these uncertainties are expressed by the confidence of the point cloud data. The low confidence here actually means that the point cloud data with too low calculation accuracy or cannot be measured (usually expressed as inf or nan) is eliminated to achieve the number. The purpose of the measurement is to clarify the image; at the same time, the data beyond the measurement range is also deleted; after straight-through filtering and statistical filtering, the dense point cloud information of the target point and the sparse point cloud information of the background point are obtained; the point cloud is clustered, and the background points belonging to the sparse points are removed from the point cloud set to obtain the dense point cloud set of the target point cloud; finally, the centroid point is calculated for the target point cloud data to obtain the world coordinate point of the tracking target object of the ZED camera; in simple terms, the FairMOT algorithm is applied to single target dynamic tracking, and the point cloud data of each target frame, the target tracking ID, and the position information and recognition confidence of the target in the image are obtained from the acquired image. The framed point cloud data is processed using point cloud filtering and clustering algorithms to obtain the positions of multiple target points in an image, which are converted to the desired coordinate system through coordinate transformation;

[0039] The data information obtained by the UWB sensor is pre-processed through low-pass filtering and sliding window filtering to obtain the distance and angle data between the base station and the beacon, and then the distance and angle data are converted into two-dimensional world coordinate points;

[0040] S3, Data Fusion:

[0041] After obtaining the data processed by the ZED camera and the UWB sensor, the position data obtained by the ZED camera and the position data of the UWB sensor are fused through multi-sensor data based on multiple Kalman filters to obtain the final position of the target person. The multi-Kalman filtering technology used here is to perform multiple Kalman filtering on the data in the first dimension of the two target points, and then perform multiple Kalman filtering fusion on the data in the second dimension. The purpose of this is that despite the filtering operation, the data in the second dimension of the UWB data still has a large fluctuation range. Therefore, when fusing the multi-sensor data, it is necessary to select reliable data from the UWB data separately to improve accuracy.

[0042] S4. Target tracking:

[0043] After obtaining the target point data, it is necessary to determine the tracking target and track it subsequently, so initialization operations are required.

[0044] Furthermore, the ZED camera in step S1 can obtain the RGBD information and point cloud data of the current image, and is equipped with C++, Python and ROS interfaces; the UWB sensor consists of two parts: a UWB beacon and a base station. The UWB sensor can return the distance and angle information between the beacon and the base station in real time at a frequency of 100 Hz. The effective measurement range of the UWB sensor is -75° to 75°.

[0045] Furthermore, when the FairMOT algorithm is applied in step S2, the ZED camera obtains the current target image, uses the DCNv2 model to identify the personnel, obtains the image position coordinates of all target personnel, selects them, assigns an ID number to each target, and performs ID matching on the personnel targets in subsequent images.

[0046] Furthermore, in step S3, multi-sensor data fusion based on multiple Kalman filters is performed, specifically:

[0047] After processing the image data from the ZED camera, the target point obtained has a very small error within ±10cm within a distance of 5m. However, the data obtained from the UWB sensor has a very small error in the x-direction, but a very large error in the y-direction, making it less reliable.

[0048] The data accuracy of the two sensors shows that the ZED camera has high data accuracy in both the x and y dimensions, but the x dimension must be measured within 5 meters. The UWB sensor has stable data in the x dimension, high accuracy, and is independent of the measured distance. However, its y dimension accuracy is low, making it unsuitable for use.

[0049] Therefore, we adopt the form of using ZED camera as the main and UWB sensor as the auxiliary to fuse the data of the two; fuse the world coordinate point obtained by ZED camera with the world coordinate point obtained by UWB, and in the x-dimension, perform multiple Kalman filter processing on the data of ZED camera and UWB sensor; in the y-dimension, only take the data of ZED camera and perform Kalman filter processing to obtain a reliable target world coordinate point. The fusion effect is as follows Figure 3 As shown, Figure 3 Middle: Line A shows the data obtained by processing the image acquired by the ZED camera using the FairMOT multi-target recognition and tracking algorithm. Line B shows the data obtained by the UWB sensor. Line C shows the data obtained by processing the data of the ZED camera and UWB sensor using multiple Kalman filters.

[0050] Furthermore, the initialization operation of the personnel tracking technology in step S4 is specifically as follows:

[0051] S4-1. Initialize the tracking target ID. After receiving the first image data, if there are multiple targets, the target ID closest to the UWB data point will be set as the initial tracking target.

[0052] S4-2. Considering the special circumstances of target loss and re-identification, the tracking ID may change. It is necessary to create a container for tracking target IDs. All targets in this container are tracked with a certain priority. The IDs in this container are updated by adding new target ID information when the target is lost, recovered, or re-identified. The priority is that the target added to the tracking target ID container first has the highest priority, and the most recent target ID has the lowest priority. Tracking always selects the target with the highest priority first.

[0053] S4-3. Notify the target data processing unit of the target ID to be tracked.

[0054] Furthermore, during the target tracking process in step S4, when FairMOT frames are lost or the target is temporarily lost, target prediction processing is required. The target image position under the frame loss condition is inferred based on the historical image point cloud information. That is, the tracking strategy is used to track the target while considering special tracking conditions and handling the situations of target loss, reappearance, and target selection.

[0055] To determine the initial tracking target, it is necessary to identify the tracking target holding the UWB beacon from multiple objects in the image. Therefore, based on the position data of the UWB sensor, the target closest to it is selected to start tracking;

[0056] During the tracking process, when FairMOT loses frames or the target is temporarily lost, the target's trajectory is predicted and the loss time is recorded. When the loss time reaches the threshold, the target is judged to be lost. When the target is recaptured, FairMOT will treat it as a new object and assign a new ID, so it is necessary to search for a new possible target again. Among the target points within a certain range around the UWB sensor coordinate point, the point closest to the UWB point is regarded as the new tracking target, and the new tracking target ID is placed in the tracking ID container. In the subsequent process, the tracking target is always all the target IDs in the tracking container.

[0057] Working principle:

[0058] This patent includes: the application of the FairMOT multi-target tracking algorithm in personnel tracking; point cloud filtering technology for obtaining the target object through the ZED camera, and then obtaining the three-dimensional coordinate points of the camera system; multi-sensor data fusion based on multiple Kalman filters from the position data obtained by the camera and the position data of the UWB sensor to obtain the final personnel target position; based on the ZED camera and UWB sensor, the original tracking strategy is used to track the target ID, which also includes target prediction, retrieval and re-identification algorithms in the event of target loss.

[0059] Specifically, such as Figure 1 As shown in the figure, this technical framework can be divided into three modules: data reception and preprocessing module, information fusion module and target tracking module.

[0060] 1. Data reception and preprocessing module

[0061] The function of this module is to obtain external information through the ZED camera and UWB sensor, and perform data preprocessing on the received information.

[0062] The ZED camera captures image information, while the UWB sensor obtains target distance and angle information. The ZED camera image is processed using the FairMOT multi-target recognition and tracking algorithm to determine the location of each target in the image and select it. Each target is also assigned an ID, and the ID is matched against it in subsequent images for visual tracking. After obtaining the box and ID for each target, the box containing the ID of the tracked target is extracted from the ZED camera's point cloud. Low-confidence data is first removed from the point cloud, and data outside the measurement range is also deleted. The processed point cloud then undergoes a series of point cloud filtering operations. After pass-through filtering and statistical filtering, the point cloud data produces a dense point cloud of target points and a sparse point cloud of background points. Finally, the point cloud is clustered to remove sparse background points from the point cloud, resulting in a dense point cloud of the target point cloud. The centroid of the target point cloud is calculated to obtain the world coordinates of the tracked target as detected by the ZED camera.

[0063] The UWB sensor pre-processes the data through low-pass filtering and sliding window filtering to obtain stable distance and angle data between the base station and the beacon, and then uses the distance and angle data to convert it into a two-dimensional world coordinate point.

[0064] This module gets the world target point obtained by two different sensors.

[0065] 2. Information Fusion Module

[0066] The function of this module is to fuse the world coordinate point obtained by FairMOT with the world coordinate point obtained by UWB to obtain a reliable target world coordinate point.

[0067] The two coordinate points output by the data reception and preprocessing module are fused, and the resulting target point is used as the output of the information fusion module. Before data fusion, the coordinate points in the two different coordinate systems must be converted to the same coordinate system, and then the target point is fused. The two world coordinate points are fused using Kalman filtering technology to obtain a stable and reliable target position point.

[0068] 3. Target Tracking Module

[0069] This module infers the current state of the target based on the ID and recognition status of multiple targets output by FairMOT in the image stream, and tracks the target.

[0070] Initialization process: After the first image data is received, if there are multiple targets, the target ID closest to the UWB data point will be set as the initial tracking target.

[0071] Create a container for tracking target IDs and track all targets in this container with a certain priority. The ID updates in this container will be added when lost, recovered, or re-identified.

[0072] FairMOT's strategy for frame loss: During the recognition process, if a target frame is lost, the target is predicted and its position is inferred from historical image point cloud information. This prediction algorithm is also used in the case of target loss.

[0073] Target loss determination: In the current algorithm, when all target IDs in the tracking target ID container do not appear in the current image frame, this is called frame loss or temporary loss of view. If this situation occurs for several consecutive frames, the target is considered lost.

[0074] Strategy when the target is lost: When the target is lost, it is necessary to predict the target's position while waiting for the target to return to the field of view and identify and retrieve it.

[0075] After the target is lost, the tracking ID is updated: When the target returns to the field of view, its ID may no longer be the previous ID number. If the target re-identification fails and the ID is changed to a new ID, it is necessary to first determine whether the location point matches the data obtained by another sensor. If so, the target's new ID is added to the tracking target container. Otherwise, the search for the target will continue.

[0076] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for tracking, perceiving, and identifying a single person in a multi-target situation, characterized in that: This method is based on the FairMOT multi-target tracking algorithm and uses a ZED camera and UWB sensor to identify and track a single target in a multi-unit situation. It includes the following steps: S1. Data acquisition: The ZED camera acquires the target image, and the UWB sensor acquires the distance and angle information between the UWB beacon and the base station; S2. Data preprocessing: The images captured by the ZED camera are processed using the FairMOT multi-target recognition and tracking algorithm to determine the position of each target in the image. These targets are then boxed out and assigned an ID. A box containing the target ID is then taken and the point cloud within the box is obtained from the ZED camera's point cloud data. Low-confidence data is removed from the point cloud data, as is data outside the measurement range. After pass-through filtering and statistical filtering, a dense point cloud of target points and a sparse point cloud of background points are obtained. Perform clustering on the point cloud to remove sparse background points from the point cloud set, obtaining a dense point cloud set of the target point cloud. Finally, calculate the centroid of the target point cloud data to obtain the world coordinate point of the tracking target object of the ZED camera. The data information obtained by the UWB sensor is pre-processed through low-pass filtering and sliding window filtering to obtain the distance and angle data between the base station and the beacon, and then the distance and angle data are converted into two-dimensional world coordinate points; S3, Data Fusion: After obtaining the data processed by the ZED camera and the UWB sensor, the position data obtained by the ZED camera and the position data of the UWB sensor are fused by multi-sensor data based on multiple Kalman filters to obtain the final target position of the person; S4. Target tracking: After obtaining the target point data, it is necessary to determine the tracking target and track it subsequently, so initialization operations are required; In step S3, multi-sensor data fusion based on multiple Kalman filters is performed. Specifically, the world coordinate point obtained by the ZED camera is fused with the world coordinate point obtained by the UWB. In the x-dimension, the data of the ZED camera and the UWB sensor are processed by multiple Kalman filters. In the y-dimension, only the data of the ZED camera is taken and processed by Kalman filtering to obtain a reliable target world coordinate point. The initialization operation of the personnel tracking technology in step S4 is specifically as follows: S4-1. Initialize the tracking target ID. After receiving the first image data, if there are multiple targets, the target ID closest to the UWB data point will be set as the initial tracking target. S4-2. Considering the special circumstances of target loss and re-identification, the tracking ID may change. It is necessary to create a container for tracking target IDs. All targets in this container are tracked with a certain priority. The IDs in this container are updated by adding new target ID information when the target is lost, recovered, or re-identified. The priority is that the target added to the tracking target ID container first has the highest priority, and the most recent target ID has the lowest priority. Tracking always selects the target with the highest priority first. S4-3. Notify the target data processing unit of the target ID to be tracked.

2. The method for tracking, sensing and identifying a single person in a multi-target situation according to claim 1, characterized in that: The ZED camera in step S1 obtains the RGBD information and point cloud data of the current image and is equipped with C++, Python, and ROS interfaces. The UWB sensor returns the distance and angle information of the beacon from the base station in real time at a frequency of 100 Hz. The effective measurement range of the UWB sensor is -75° to 75°.

3. The method for tracking, sensing and identifying a single person in a multi-target situation according to claim 1, characterized in that: When the FairMOT algorithm is applied in step S2, the ZED camera acquires the current target image, uses the DCNv2 model to identify people, obtains the image position coordinates of all target people, selects them, assigns an ID number to each target, and performs ID matching on the people in subsequent images.

4. The method for tracking, sensing and identifying a single person in a multi-target situation according to claim 1, characterized in that: During the target tracking process in step S4, when FairMOT frame loss or temporary target loss occurs, target prediction processing is required to infer the target image position in the frame loss situation through historical image point cloud information.

Images