A method and system for dynamic target tracking based on depth cameras

Abstract

The application discloses a dynamic target tracking method and system based on a depth camera, relates to the fields of target detection and target tracking, and comprises the following steps: collecting original point cloud data covering 360-degree directions of a dynamic target by using a plurality of depth cameras and performing time registration; performing preprocessing on the original point cloud data obtained by each depth camera; judging whether the preprocessed point cloud data contain target information or not, and performing position registration and coordinate conversion on the preprocessed point cloud data containing target information to obtain fused real-time position information of the dynamic target; and tracking the dynamic target based on the fused real-time position information of the dynamic target. The application expands the applicable scene of the depth camera and improves the precision of dynamic target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection and target tracking, in particular to a dynamic target tracking method and system based on a depth camera. BACKGROUND

[0002] In the field of autonomous navigation, city modeling, robots and virtual reality, it is often necessary to identify and locate dynamic target objects. The data acquisition of traditional cameras is easily affected by weather, light and other factors, while the depth camera can obtain rich and clear three-dimensional point cloud data in the case of insufficient light, which makes the processing of depth camera three-dimensional point cloud data more important. The dynamic target detection and tracking method based on the depth camera can more accurately locate and represent the shape of the dynamic target, and is usually the preferred method of environmental perception.

[0003] At present, there are two difficulties in the application of depth cameras in target tracking: first, a single camera needs to move continuously to collect object surface data. Due to the limitations of camera position registration algorithm, the difference between each frame of data cannot be too large, which leads to slow movement of the camera and reduces the collection efficiency. If a flexible object is tracked, the shape of the object may change, which will cause the data collected before and after the camera moves around the object to be inconsistent, causing errors in target tracking. Second, due to the limitation of the field of view angle of the camera, the collected point cloud data is incomplete. In view of the above two difficulties, the present application provides a multi-camera target tracking method and system to obtain multi-angle information of dynamic targets. SUMMARY

[0004] The purpose of the present application is to provide a dynamic target tracking method and system based on a depth camera, which expands the adaptive scene of the depth camera system and improves the accuracy of dynamic target recognition and the accuracy of dynamic target tracking.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A dynamic target tracking method based on a depth camera, comprising the following steps:

[0007] Step 1: using several depth cameras to collect original point cloud data covering 360 degrees of the dynamic target, and performing time registration;

[0008] Step 2: preprocessing the original point cloud data obtained by each depth camera;

[0009] Step 3: determining whether the preprocessed point cloud data contains target information, and performing position registration and coordinate conversion on the preprocessed point cloud data containing target information to obtain fused dynamic target real-time position information;

[0010] Step 4: tracking the dynamic target based on the fused real-time position information of the dynamic target.

[0011] Further, the step 2 is specifically:

[0012] Step 2.1: filtering the original point cloud data collected by each camera through pass-through filtering, radius filtering and Gaussian filtering;

[0013] Step 2.2: segmenting the filtered point cloud corresponding to each camera to separate the target point cloud from the environment;

[0014] Step 2.3: smoothing the segmented target point cloud by using the least square method to obtain the three-dimensional reconstructed target point cloud.

[0015] Further, the step 3 is specifically:

[0016] Step 3.1: determining whether the preprocessed point cloud data contains target information, and eliminating the point cloud data that does not contain target information;

[0017] Step 3.2: using the ICP algorithm to register the point cloud data in the overlapping range of the remaining point cloud data;

[0018] Step 3.3: establishing a unified world coordinate system to convert the coordinates of the registered point cloud data to obtain the fused target point cloud data.

[0019] The application also provides a dynamic target tracking system based on a depth camera, which is used to realize the dynamic target tracking method based on the depth camera.

[0020] The depth camera group is used to be installed around the dynamic target and can cover the original point cloud data in 360-degree direction of the dynamic target.

[0021] The data acquisition module is used to acquire the original point cloud data collected by the depth camera.

[0022] The data fusion module is used to fuse the original point cloud data acquired by the data acquisition module to obtain the fused dynamic target information.

[0023] The tracking module is used to track the dynamic target according to the fused dynamic target information.

[0024] Further, the data fusion module comprises:

[0025] The preprocessing unit is used to preprocess the original point cloud data corresponding to each camera.

[0026] A judging unit is configured to judge whether the preprocessed point cloud data corresponding to each camera contains dynamic target information or not;

[0027] A point cloud registration unit is configured to register and coordinate the point cloud data containing dynamic target information, and output fused point cloud data.

[0028] The present application has the following advantages:

[0029] (1) The present application proposes a multi-camera based target tracking scheme, which improves the efficiency of data collection and can be used for target tracking of flexible objects, thereby widening the application range of depth cameras in the field of target tracking.

[0030] (2) The present application collects target data from multiple angles, solves the problem of data loss caused by self-occlusion of target objects and insufficient illumination, improves the efficiency of point cloud data processing, and perfects the effect of point cloud fusion.

[0031] (3) The present application proposes a series of pretreatments such as filtering, smoothing and surface reconstruction on the point cloud data collected by the depth camera, then uses ICP algorithm for point cloud registration, uses region growing algorithm for point cloud segmentation, and performs point cloud fusion, thereby improving the target recognition capability and the accuracy of target tracking. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 It is a schematic diagram of the installation positions of the four depth cameras of the present application; wherein 1 represents a dynamic target, and 2 represents a depth camera.

[0033] Figure 2 It is a top view of the installation positions of the four depth cameras;

[0034] Figure 3 It is a flowchart of the dynamic target tracking algorithm shown in the present application;

[0035] Figure 4 It is a flowchart of a dynamic target tracking method based on a depth camera shown in the present application;

[0036] Figure 5 It is a structural block diagram of a dynamic target tracking system based on a depth camera shown in the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0038] Given the background technology, current technology requires multi-sensor fusion to adapt to low-visual-range scenarios and obtain more accurate perceptual information. However, depth cameras only have a field of view of 40°–70°. Therefore, this invention provides a dynamic target tracking method and apparatus based on a depth camera, expanding the applicable scenarios of depth cameras and improving the accuracy of dynamic target tracking.

[0039] Example 1

[0040] This embodiment provides a dynamic target tracking method based on depth cameras, mainly addressing the impact of contact detection on the state of dynamic targets and the accuracy of real-time tracking. This embodiment constructs a sensor mounting platform and utilizes four depth cameras to resolve the issue of inconsistent horizontal viewing angles among different sensors. The detection information from multiple depth cameras is fused as input, and then the detected dynamic targets are tracked to obtain more accurate dynamic target information, thereby achieving stable and accurate detection and tracking of dynamic targets.

[0041] To achieve the above functions, the technical solution adopted in this embodiment is as follows: Figures 1-3 As shown:

[0042] Step 1: Install the depth camera.

[0043] Install the four depth cameras in fixed positions and set up network communication so that the external host can receive the depth camera data.

[0044] One example is: installing a bracket in each of the four directions outside the dynamic target, and mounting four depth cameras on the brackets, such as... Figure 1 As shown, the cameras are numbered 1, 2, 3, and 4 counterclockwise, and all four depth cameras exchange data with an external processing system via network cables. In this embodiment, a TOF camera is used as the depth camera.

[0045] Since each TOF camera has a close-range horizontal field of view of 40–70°, the superposition of the horizontal fields of view of four TOF cameras can cover dynamic target information in a 360° horizontal field of view, thereby expanding the overlap range of sensor detection, such as Figure 2 As shown.

[0046] Step 2: Preprocess the raw point cloud data acquired by the depth camera.

[0047] The point cloud data is subjected to background removal and filtering processing through Gaussian filtering, straight-through filtering and radius filtering. Specifically, first, the straight-through filtering method is adopted to realize first-stage filtering, the dynamic target is positioned by setting the range of the point cloud in the x, y and z directions, and the target and the environment point cloud are preliminarily segmented; then the radius filtering method is adopted to realize second-stage filtering, the number of points contained in a given circle with a certain point as the center is calculated, if the number is higher than a preset value, all the points in the circle are retained; otherwise, all the points in the circle are deleted, and the edge discrete points of the point cloud are filtered; finally, the Gaussian filtering method is adopted to realize third-stage filtering, the standard deviation is used for denoising, and the surface of the point cloud is preliminarily smoothed.

[0048] The point cloud collected by each camera is segmented, and the target point cloud is segmented from the environment. In the embodiment, the region growing algorithm is adopted to realize point cloud segmentation.

[0049] The least square method is used for smoothing and resampling processing of the point cloud, and the target point cloud after three-dimensional reconstruction is obtained, so as to perform dynamic target tracking. The least square method makes up for the uneven density of the point cloud, the surface obtained by the method is smoother, the generation of holes is reduced, and thus the final calculation result is more accurate.

[0050] Step three: realizing data fusion.

[0051] The four depth cameras are numbered as 1, 2, 3 and 4 respectively, and the point clouds of the dynamic targets obtained by the four depth cameras are respectively denoted as A, B, C and D.

[0052] It is judged whether the data collected by the depth camera contains dynamic target information;

[0053] If yes, the ICP algorithm is used to register the point clouds collected by the adjacent depth cameras in the data acquisition system two by two, and the position change relationship of the adjacent depth cameras is obtained. For example, the world coordinate system is established at the position of the camera 1, and the positions of the cameras 2, 3 and 4 relative to the camera 1 are calculated in sequence, so that the position and posture of each camera in the unified world coordinate system are obtained, and then the TSDF model is used for point cloud fusion of the point cloud data A and the point cloud data B, C and D; the B, C and D are the dynamic targets detected by the depth cameras 2, 3 and 4;

[0054] If no, the camera data information of the detected dynamic target point cloud is determined as the fused dynamic target information.

[0055] In a specific implementation of the application, let θ be the installation angle of the depth cameras 2, 3 and 4 relative to the camera 1, and x0, y0 and z0 be the x, y and z axis coordinates of the dynamic target in the world coordinate system; x TOF , y TOF , z TOFrespectively represent the x, y, z axis coordinates of the dynamic target in the depth camera coordinate system; T xyz is a translation vector, then the coordinate transformation formula is:

[0056]

[0057] Since the detection distance of the depth camera is not consistent, the detection range is also not constant, which also depends on the reflectivity of the surrounding environment problem. Therefore, what really needs to be fused is the area where the detection ranges overlap. In order to make the overlapping area completely cover the detection range, four depth cameras are used to obtain more accurate dynamic target information in the overlapping area.

[0058] Step four: based on the fused dynamic target information, tracking the dynamic target.

[0059] In this embodiment, based on the particle filtering algorithm and the fused dynamic target information, the dynamic target is tracked.

[0060] Embodiment two

[0061] As shown in the figure, the embodiment provides a dynamic target tracking method based on a depth camera, as shown in the figure, comprising: Figure 4

[0062] Step 100: the depth camera collects the original point cloud data and performs time registration.

[0063] Step 200: pre-processing the point cloud data;

[0064] Step 300: real-time detection of the existence of dynamic targets;

[0065] Step 400: registration and fusion of point cloud data obtained by four depth cameras;

[0066] Step 500: based on the fused dynamic target information, continuously tracking the dynamic target.

[0067] Among them, step 400 specifically includes:

[0068] The original point cloud data is pre-processed, and the pre-processing mainly includes denoising, point cloud segmentation and surface reconstruction, to obtain the position information of A, B, C and D in the camera coordinate system;

[0069] Determine whether the data collected by the depth camera contains dynamic target information;

[0070] ​If yes, the ICP algorithm is used to register the point clouds collected by the adjacent depth cameras in the data acquisition system two by two, and the positional change relationship of the adjacent depth cameras is obtained. Then, the world coordinate system is established at the position of camera 1, and the positions of cameras 2, 3 and 4 relative to the world coordinate system are calculated in sequence, so that the position and posture of each camera in the unified world coordinate system are obtained. Then, the TSDF model is used to fuse the point cloud data A and the point cloud data B, C and D detected by the depth cameras 2, 3 and 4.

[0071] If no, the camera data information of the point cloud of the detected dynamic target is determined as the fused dynamic target information.

[0072] Embodiment Three

[0073] As shown in Figure 5 , the embodiment provides a dynamic target tracking system based on a depth camera, which comprises a depth camera installed around a dynamic target and a point cloud data processing module.

[0074] The number of the depth cameras is 4, which are used to collect original point cloud data under different angles of view.

[0075] The point cloud data processing module comprises:

[0076] A data acquisition module is used to acquire the original point cloud data collected by the depth camera.

[0077] A data fusion module is used to fuse the original point cloud data to obtain fused dynamic target information.

[0078] A tracking module is used to track the dynamic target based on the fused dynamic target information.

[0079] In the embodiment, the data fusion module specifically comprises:

[0080] A preprocessing unit is used to preprocess the original point cloud data to obtain the detection results of the depth cameras 1, 2, 3 and 4.

[0081] A judging unit is used to judge whether the point cloud data collected by the depth camera contains dynamic target information.

[0082] A point cloud registration unit is used to register and convert the coordinates of the point cloud data containing dynamic target information, and the point cloud data not containing dynamic target information does not participate in the registration process. The fused point cloud data is outputted.

[0083] The above are only specific embodiments of the present application. Obviously, the present application is not limited to the above embodiments, but can have many variations. All variations that can be directly derived or inferred by those skilled in the art from the disclosure of the present application should be considered as falling within the scope of the present application.

Claims

1. A dynamic target tracking method based on a depth camera, characterized in that, Includes the following steps: Step 1: Use several depth cameras to acquire raw point cloud data covering the dynamic target in 360 degrees and perform time registration; Step 2: Preprocess the raw point cloud data acquired by each depth camera; Step 2 specifically refers to: Step 2.1: Filter the raw point cloud data acquired by each camera using pass-through filtering, radius filtering, and Gaussian filtering; In step 2.1, the first-level filtering is achieved using a direct-pass filtering method. By setting the range of the point cloud in the x, y, and z directions, the dynamic target is located, and the target and the environment point cloud are initially separated. Then, the second-level filtering is achieved using a radius filtering method. The number of points contained within a circle centered at a given point is calculated. If the number is higher than a preset value, all points within the circle are retained; otherwise, all points within the circle are deleted, and the discrete points at the edge of the point cloud are filtered. Finally, the third-level filtering is achieved using a Gaussian filtering method. The standard deviation is used for noise reduction, and the point cloud surface is initially smoothed. Step 2.2: Segment the filtered point cloud corresponding to each camera to separate the target point cloud from the environment; Step 2.3: The segmented target point cloud is smoothed using the least squares method to obtain the 3D reconstructed target point cloud; Step 3: Determine whether the preprocessed point cloud data contains target information, and perform position registration and coordinate transformation on the preprocessed point cloud data containing target information to obtain the fused dynamic real-time target position information; Step 3 specifically includes: Step 3.1: Determine whether the preprocessed point cloud data contains target information, and discard point cloud data that does not contain target information; Step 3.2: Use the ICP algorithm to register the point cloud data with overlapping regions in the remaining point cloud data; Step 3.3: Establish a unified world coordinate system, perform coordinate transformation on the registered point cloud data, and obtain the fused target point cloud data; Step 4: Track the dynamic target based on the fused real-time location information of the dynamic target.

2. The dynamic target tracking method based on a depth camera according to claim 1, characterized in that, The number of depth cameras is four, which are installed around the dynamic target.

3. The dynamic target tracking method based on a depth camera according to claim 1, characterized in that, In step 2.2, a region growing algorithm is used to segment the point cloud.

4. The dynamic target tracking method based on a depth camera according to claim 1, characterized in that, The coordinate transformation formula in step 3.3 is: Where, x TOF y TOF z TOF Let x, y, and z represent the x, y, and z coordinates of the dynamic target measured by the camera in the depth camera coordinate system, respectively; θ represents the camera's mounting angle relative to the unified world coordinate system, and T represents the depth camera coordinate system. xyz Let x0, y0, and z0 be the translation vectors, and let x0, y0, and z0 be the x, y, and z coordinates of the dynamic target in the world coordinate system, respectively.

5. A dynamic target tracking system based on a depth camera, used to implement the dynamic target tracking method based on a depth camera as described in claim 1; characterized in that, The dynamic target tracking system includes: A depth camera array is used to mount around a dynamic target and is capable of capturing raw point cloud data of the target in 360 degrees. The data acquisition module is used to acquire raw point cloud data collected by the depth camera; The data fusion module is used to fuse the raw point cloud data acquired by the data acquisition module to obtain fused dynamic target information. The tracking module is used to track dynamic targets based on the fused dynamic target information.

6. A dynamic target tracking system based on a depth camera according to claim 5, characterized in that, The data fusion module includes: The preprocessing unit is used to preprocess the raw point cloud data corresponding to each camera. The judgment unit is used to determine whether the preprocessed point cloud data corresponding to each camera contains dynamic target information; The point cloud registration unit is used to register and transform point cloud data containing dynamic target information, and output the fused point cloud data.

Images