Tracking method, tracking device, readable storage medium, electronic device and robot

Abstract

The present invention provides a tracking method, a tracking device, a readable storage medium, an electronic device, and a robot. The tracking method includes: determining local point cloud data of a target object according to point cloud information; performing a complement processing on the local point cloud data to obtain the overall point cloud data of the target object, wherein the local point cloud data is unordered point cloud data, and the overall point cloud data is ordered point cloud data; and tracking the target object according to the overall point cloud data. By complementing and processing the unordered local point cloud data generated by a lidar into ordered overall point cloud data, the present application improves the integrity of observation, and when processing consecutive multiple frames of point clouds, the matching accuracy for the same object is greatly improved, thereby realizing high-precision object registration and object tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of robot perception, and more particularly, to a tracking method, a tracking device, a readable storage medium, an electronic device, and a robot. Background Art

[0002] In the related art, for a movable robot, when the robot is required to perform an action, it needs to continuously perceive the surrounding objects and plan a travel route. Generally, when the robot perceives and tracks the surrounding objects, it needs to rely on the point cloud data obtained from image input and laser scanning. However, due to the discreteness and disorder of the observation data, it is impossible to achieve high-precision object registration, and the object tracking effect is poor. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related technologies.

[0004] To this end, a first aspect of the present invention provides a tracking method.

[0005] A second aspect of the present invention provides a tracking device.

[0006] A third aspect of the present invention provides a tracking device.

[0007] A fourth aspect of the present invention provides a readable storage medium.

[0008] A fifth aspect of the present invention provides an electronic device.

[0009] A sixth aspect of the present invention provides a robot.

[0010] In view of this, a first aspect of the present invention provides a tracking method, including: determining local point cloud data of a target object according to point cloud information; performing a completion process on the local point cloud data to obtain overall point cloud data of the target object, where the local point cloud data is unordered point cloud data and the overall point cloud data is ordered point cloud data; and tracking the target object according to the overall point cloud data.

[0011] In this technical solution, for a movable robot, such as a sweeping robot or a service robot, when it performs various behaviors, it needs to obtain sufficient information in the environment, continuously perceive the surrounding objects, track the objects, and perform dynamic path planning. And the continuous perception of the objects, that is, the accuracy of tracking the objects, will have a greater impact on the intelligence of the robot. Therefore, high-precision object tracking needs to be ensured.

[0012] In the embodiments of the present application, the above point cloud information may specifically be the point cloud information obtained by devices such as lidar. Due to the characteristics of lidar, the point cloud information it obtains belongs to unordered point cloud. And for a robot, since lidar can only obtain the point cloud information on the side of an object facing the radar emission station, in a frame of point cloud obtained by lidar, the point cloud of the target object included is local point cloud.

[0013] The lidar obtains the surrounding point cloud data in real time, and obtains multiple consecutive frames of point cloud arranged in chronological order. After obtaining a frame of point cloud, an object detection algorithm, such as the CenterPoint function (a function that can return the center coordinates of a rectangular area), can be used to detect the objects in the point cloud, such as detecting tables, chairs, people, etc. These detected objects are the above-mentioned target objects.

[0014] It can be understood that during the object detection process, at the position of the same object, multiple candidate boxes may be detected, and these candidate boxes may overlap. At this time, the non-maximum suppression algorithm can be used to eliminate redundant detection boxes.

[0015] According to the position of the final detection box, the local point cloud data of the target object can be obtained. This local point cloud data is the unordered point cloud data obtained by the lidar. To ensure the accuracy of object tracking, in the embodiments of the present application, through a preset completion algorithm, the local point cloud data of the target object is supplemented into complete global cloud data, and this global point cloud data is ordered point cloud.

[0016] Specifically, the global point cloud data is the complete point cloud cluster of the object obtained by an incomplete algorithm based on the double-sphere sampling model. Among them, the double-sphere sampling model is the process of forming ordered point cloud by double-sphere sampling of the original point cloud of a three-dimensional model. Specifically, first, based on the shape of the three-dimensional model, an enclosing sphere of the model is generated. On the enclosing sphere, multiple sampling points are uniformly sampled. Taking these sampling points as endpoints, rays are projected towards the three-dimensional model. According to the first intersection points of these rays and the three-dimensional model, the point cloud data of the three-dimensional model can be obtained.

[0017] Among them, the rays are emitted in a certain order, and the positions of the sampling points on the enclosing sphere are fixed. Therefore, the number of sampling rays is the same, and the sampling position points and order are the same. The obtained point cloud cluster of the three-dimensional model is the complete ordered point cloud.

[0018] By complementing and processing the unordered local point cloud data generated by the lidar into ordered global point cloud data, the integrity of the observation is improved. When processing multiple consecutive frames of point cloud, the matching accuracy of the same object is greatly improved, thereby realizing high-precision object registration and object tracking.

[0019] In addition, the tracking method in the above technical solution provided by the present invention may further have the following additional technical features:

[0020] In the above technical solution, the point cloud information includes first point cloud information and second point cloud information; tracking the target object according to the overall point cloud data includes: determining the first overall point cloud data of the first object in the first point cloud information, and determining the second overall point cloud data of the second object in the second point cloud information, and determining the difference degree between the first object and the second object; when the difference degree is less than the threshold, determining the first object and the second object as the target object.

[0021] In this technical solution, the lidar obtains the surrounding point cloud data in real time, and obtains a series of consecutive frames of point clouds arranged in chronological order. Among them, the first point cloud information and the second point cloud information are specifically two adjacent frames of point clouds, that is to say, the first point cloud information and the second point cloud information have temporal continuity. It can be understood that the first point cloud information is obtained before the second point cloud information.

[0022] First, in the first point cloud data, identify the locally disordered point cloud of the first object and perform a completion process on it to obtain the overall ordered point cloud of the first object. Then, in the second point cloud data, identify the locally disordered point cloud of the second object and perform a completion process on it to obtain the overall ordered point cloud of the second object.

[0023] After obtaining the first overall point cloud data and the second overall point cloud data, the difference degree between the first object and the second object is obtained by comparing the first overall point cloud data and the second overall point cloud data. It can be understood that when the difference degree between the first object and the second object is large, such as when the difference degree is greater than the preset threshold, it is determined that the first object and the second object are not the same object.

[0024] If the difference degree between the first object and the second object is small, specifically less than the preset threshold, it is determined that the first object and the second object are the same object, both of which are the above-mentioned target objects, thus realizing the accurate tracking of the same object in different time dimensions.

[0025] In any of the above technical solutions, the difference degree includes the target chamfer distance; determining the difference degree between the first object and the second object includes: projecting the second overall point cloud data into the coordinate system of the first point cloud information to obtain the third overall point cloud data, where the third overall point cloud data includes N first point clouds, the first overall point cloud data includes N second point clouds, and N is a positive integer; according to the N first point clouds and the N second point clouds, determining N groups of corresponding point cloud groups; and determining the target chamfer distance according to the average value of the chamfer distances of the N groups of point cloud groups.

[0026] In this technical solution, the difference degree between the first object and the second object specifically includes the Chamfer distance between each point cloud in the first overall point cloud data and the corresponding point cloud in the second point cloud data, that is, the above-mentioned target Chamfer distance.

[0027] Specifically, project the second overall point cloud data into the coordinate system corresponding to the first point cloud information. In an ideal situation, if the first object and the second object are the same object, then after the second overall point cloud data is projected into the first point cloud information, it will coincide with the second overall point cloud data. In the actual scenario, due to the existence of errors, noises, etc., the first overall point cloud data projected into the first point cloud information and the second overall point cloud data may not coincide.

[0028] Here, denote the first overall point cloud data as As, the first overall point cloud data after being projected into the first point cloud information as As’, and the second overall point cloud data as Bs. For the N groups of corresponding point clouds (a, b) in As’ and Bs, calculate their Chamfer distances.

[0029] In the calculation process, first, randomly select a point p in a’, and select the point p’ closest to the point p from b, and calculate the Chamfer distance d between p and p’.

[0030] Then, traverse all the points in a and all the corresponding points in b, calculate and accumulate the Chamfer distances between them respectively, and divide by the total number of points. The obtained average value is the target Chamfer distance.

[0031] It can be understood that when the Chamfer distance is larger, it indicates that the shape difference between the first object and the second object is larger; on the contrary, when the Chamfer distance is smaller, it means that the shapes of the first object and the second object are closer. Therefore, when the Chamfer distance is less than the threshold, it can be considered that the first object and the second object are similar. At this time, the first object and the second object are associated as the same target object, achieving the purpose of tracking the object.

[0032] In any of the above technical solutions, the local point cloud data is complemented, including: complementing the local point cloud data through a preset complementing algorithm model.

[0033] In this technical solution, the local point cloud data of the object is complemented through a pre-trained complementing algorithm model, so as to generate and obtain the overall point cloud data of the corresponding object. Among them, the complementing algorithm model is a neural network model based on machine learning training. Therefore, after the local point cloud data is input into the complementing algorithm model, the complementing algorithm model can output the predicted overall point cloud data to realize the complementing of the local point cloud data.

[0034] In any of the above technical solutions, before performing the completion process on the local point cloud data, the method further includes: cropping the point cloud data of the three-dimensional model to obtain the local point cloud corresponding to the three-dimensional model; sampling the three-dimensional model to obtain the overall ordered point cloud corresponding to the three-dimensional model; training a preset neural network model with the local point cloud and the overall ordered point cloud to obtain a completion algorithm model.

[0035] In this technical solution, when training to obtain a completion algorithm model, it is first necessary to process and obtain a training set. Among them, the training set includes two parts. The first part is the local point cloud of the object, and the second part is the overall ordered point cloud of the same object.

[0036] Specifically, first obtain a three-dimensional model, which is a model established from unordered point clouds. Cropping this model can obtain the local point cloud of the three-dimensional model. Then, by using the double-sphere sampling method, sample the three-dimensional model to establish the ordered point cloud corresponding to the three-dimensional model, and this ordered point cloud is the overall point cloud.

[0037] Using the collected overall ordered point cloud as the target and the cropped local point cloud as the material, train a preset neural network model. When the gap between the point cloud data output by the model and the overall ordered point cloud is less than the set value, it is determined that the model training is completed, and a completion algorithm model that meets the requirements is obtained.

[0038] In any of the above technical solutions, sampling the three-dimensional model includes: determining a first sphere according to the three-dimensional model, with the three-dimensional model located inside the first sphere; performing Fibonacci spherical sampling on the spherical surface of the first sphere to obtain first sampling points; generating rays with each first sampling point as an endpoint; and determining the overall ordered point cloud according to the generation order of the rays and the intersection points of the rays and the surface of the three-dimensional model.

[0039] In this technical solution, for double-sphere sampling of the three-dimensional model, specifically, based on the shape of the three-dimensional model, generate the bounding sphere of the model, that is, the above-mentioned first sphere. It can be understood that the first sphere is a virtual sphere. For this first sphere, a three-dimensional model can be formed, or a three-dimensional model may not be established, and only the coordinate data of each point position on the spherical surface of the first sphere is determined.

[0040] After obtaining the first sphere, that is, the bounding sphere of the three-dimensional model, perform Fibonacci spherical sampling on the spherical surface of the bounding sphere. Specifically, the Fibonacci spherical sampling method is an approximate algorithm that can generate uniformly sampled points on the unit sphere, where the unit sphere can be expressed as: Its main step is to map the two-dimensional Fibonacci grid onto the spherical surface. The definition of the Fibonacci grid is a unit square, which can be expressed as [0, 1) 2, in which there are any number of n points evenly distributed, and the two-dimensional point with index i can be expressed as:

[0041]

[0042] where p i is the two-dimensional point with index i, and (x i , y i ) are the two-dimensional coordinates of point p i . n is the number of points evenly distributed in the Fibonacci grid, and the %1 operator represents the fractional part of the parameter.

[0043] By using cylindrical equal-area projection to map the two-dimensional points on the Fibonacci grid to the unit sphere, evenly distributed three-dimensional points can be obtained. The above point p i can be expressed in spherical coordinates as:

[0044]

[0045] Through Fibonacci spherical sampling, N evenly distributed points sampled on the enclosing sphere can be regarded as virtual cameras for photographing the three-dimensional model. The M rays determined according to the N first sampling points can be regarded as the virtual rays corresponding to these virtual cameras, and the projection process of the virtual cameras is simulated through the virtual rays.

[0046] Specifically, each first sampling point can generate multiple rays. The endpoints of these rays are the first sampling points, the direction of the rays is the direction towards the three-dimensional model, and the direction of the rays projected each time remains unchanged.

[0047] Since these virtual cameras are spherically distributed, the observation directions of the virtual cameras simulated by the first sampling points are more diverse, which can effectively avoid the "self-occlusion" problem that the projection rays are blocked by the prominent structures of the model itself due to the single acquisition direction.

[0048] When the ray intersects with the surface of the three-dimensional model, the distance between the first intersection point of the ray and the three-dimensional model and the first sampling point that emits the ray is the point cloud data of the intersection point. Since the emission direction of the ray is known and fixed, when regenerating the point cloud of the three-dimensional model, the point cloud data of the three-dimensional model can be easily restored according to the distance information of each point. And since the arrangement of the first sampling point and the ray is fixed after the first sampling point is determined, the obtained point cloud data of the three-dimensional model is ordered point cloud data, so the overall ordered point cloud of the three-dimensional model is obtained.

[0049] In any of the above technical solutions, generating a ray with each first sampling point as an endpoint includes: taking the first sampling point as the center of a circle to generate a second sphere; performing Fibonacci spherical sampling on the spherical surface of the second sphere to obtain a second sampling point; and generating a ray passing through the second sampling point with the first sampling point as an endpoint.

[0050] In this technical solution, when generating a ray based on the first sampling point, the first sampling point is used as the endpoint of the ray. To determine the direction of the ray, in an embodiment of the present application, a second sphere is generated with the first sampling point as the center of a circle, and a plurality of second sampling points evenly distributed are sampled on the spherical surface of the second sphere by the Fibonacci sampling method.

[0051] Taking the example of sampling 32 points on the first sphere and 64 points on the second sphere, a total of 32×64 = 2048 rays can be generated. These rays respectively pass through each of the 64 second sampling points, and finally 2048 sampling values can be obtained. After screening out the rays that do not intersect with the three-dimensional model by using a clustering algorithm, the overall ordered point cloud of the three-dimensional model can be obtained.

[0052] In any of the above technical solutions, the degree of difference further includes a rotation matrix and a translation amount; determining the degree of difference between the first object and the second object further includes: respectively determining a first point set and a second point set according to the first generation order corresponding to the first overall point cloud data and the second generation order corresponding to the second overall point cloud data; performing singular value decomposition (SVD, Singular Value Decomposition) on the first point set and the second point set to obtain a rotation matrix and a translation amount.

[0053] In this technical solution, the degree of difference between the first object and the second object further includes the pose difference between the first object and the second object, specifically including a rotation matrix and a translation amount. The rotation matrix and the translation amount can be used to evaluate the pose difference between the first object and the second object. When the pose difference is smaller, it means they are closer; on the contrary, the difference is larger.

[0054] Here, the first overall point cloud data is denoted as As, the first overall point cloud data projected onto the first point cloud information is denoted as As', and the second overall point cloud data is denoted as Bs. For the N groups of corresponding point clouds (a, b) in As' and Bs, calculate their pose changes.

[0055] Among them, according to the sampling order when sampling the first object and the second object by the double-sphere sampling method, that is, the generation order of the rays, the corresponding sampling sequences S a and S b are respectively determined. According to the orderliness of the sampled sampling points, the point sets a' and b' corresponding to the intersection of the sampling numbers are respectively selected, that is, the above-mentioned first point set and second point set.

[0056] By performing singular value decomposition, the rotation matrix and translation amount of point set a' and point set b' are solved. By combining the Chamfer distance, rotation matrix and translation amount, it is determined whether the objects in two frames of point clouds, that is, the objects in the first point cloud information and the second point cloud information, are the same object, thus realizing the observation of the same object at different times, so that the objects at different times can be correlated with each other and recognized as the same target object (assigned the same recognition information), thereby achieving accurate object tracking.

[0057] In any of the above technical solutions, determining the first sphere according to the three-dimensional model includes: determining the maximum value of the point position coordinates of the three-dimensional model; determining the bounding box of the three-dimensional model according to the maximum value; determining the first sphere based on the bounding box.

[0058] In this technical solution, the first sphere is the circumsphere of the three-dimensional model. To ensure that the circumsphere can completely enclose the three-dimensional model, the bounding box of the three-dimensional model can be determined first. Specifically, the bounding box of the three-dimensional model can be found by calculating the difference between the extreme values of the point coordinates of the three-dimensional model in each dimension.

[0059] Among them, the bounding box is used to track the spatial position of the object in three-dimensional object detection. The bounding box can be a cube bounding box or a polyhedron bounding box. The bounding box can express the farthest boundaries of the object in all directions. Therefore, determining the first sphere based on the bounding box can ensure that the first sphere completely encloses the three-dimensional model and ensure the integrity of the point cloud data.

[0060] In any of the above technical solutions, the first sphere is the circumsphere of the bounding box.

[0061] In this technical solution, the first sphere is the circumsphere of the bounding box. Therefore, on the basis of ensuring that the first sphere completely encloses the three-dimensional model, it can be avoided that the first sphere is too large, resulting in a large amount of blank space in the first sphere, thereby reducing the number of invalid rays that do not intersect the three-dimensional object, reducing the amount of operations and calculations for determining the point cloud data, and improving the generation efficiency of the point cloud data.

[0062] The second aspect of the present invention provides a tracking device, including: a determination module for determining local point cloud data of a target object according to point cloud information; a processing module for performing a complementation process on the local point cloud data to obtain overall point cloud data of the target object, where the local point cloud data is unordered point cloud data and the overall point cloud data is ordered point cloud data; a tracking module for tracking the target object according to the overall point cloud data.

[0063] In this technical solution, for movable robots such as floor-sweeping robots and service robots, when they perform various behaviors, they need to obtain sufficient information in the environment, continuously perceive surrounding objects, track the objects, and perform dynamic path planning. And the continuous perception of objects, that is, the accuracy of tracking objects, will have a great impact on the intelligence of the robot. Therefore, it is necessary to ensure high-precision object tracking.

[0064] In the embodiment of the present application, the above-mentioned point cloud information can specifically be the point cloud information obtained by devices such as lidar. Due to the characteristics of lidar, the point cloud information it obtains belongs to unordered point cloud. And for the robot, since lidar can only obtain the point cloud information on the side of the object facing the radar transmitting station, in a frame of point cloud obtained by lidar, the point cloud of the target object included is local point cloud.

[0065] The lidar obtains the surrounding point cloud data in real time, and obtains a series of continuous frames of point cloud arranged in chronological order. After obtaining a frame of point cloud, the objects in the point cloud can be detected through an object detection algorithm, such as the CenterPoint function (a function that can return the center coordinates of a rectangular area), such as detecting a table, a chair, a person, etc. These detected objects are the above-mentioned target objects.

[0066] It can be understood that during the object detection process, at the position of the same object, multiple candidate boxes may be detected, and these candidate boxes may overlap. At this time, the non-maximum suppression algorithm can be used to eliminate redundant detection boxes.

[0067] According to the position of the final detection box, the local point cloud data of the target object can be obtained, and this local point cloud data is the unordered point cloud data obtained by the lidar. To ensure the accuracy of object tracking, in the embodiment of the present application, through a preset completion algorithm, the local point cloud data of the target object is supplemented into complete overall cloud data, and this overall point cloud data is ordered point cloud.

[0068] Specifically, the overall point cloud data is an incomplete algorithm based on the double-sphere sampling model to obtain a complete point cloud cluster of the object. Among them, the double-sphere sampling model is a process of forming ordered point cloud by double-sphere sampling of the original point cloud of the three-dimensional model. Specifically, first, based on the shape of the three-dimensional model, an enclosing sphere of the model is generated. On the enclosing sphere, a plurality of sampling points are uniformly sampled. Taking these sampling points as endpoints, rays are projected onto the three-dimensional model, and according to the first intersection points of these rays with the three-dimensional model, the point cloud data of the three-dimensional model can be obtained.

[0069] Among them, the rays are emitted in a certain order, and the positions of the sampling points on the surrounding sphere are fixed. Therefore, the number of sampling rays is the same, and the sampling position points and order are the same. The point cloud clusters of the obtained three-dimensional model are complete ordered point clouds.

[0070] By complementing and processing the disordered local point cloud data generated by the lidar into ordered overall point cloud data, the integrity of the observation is improved. When processing continuous multi-frame point clouds, the matching accuracy of the same object is greatly improved, thereby realizing high-precision object registration and object tracking.

[0071] The third aspect of the present invention provides a tracking device, including: a memory on which programs or instructions are stored; a processor for implementing the steps of the tracking method provided in any of the above technical solutions when executing the programs or instructions. Therefore, this tracking device also includes all the beneficial effects of the tracking method provided in any of the above technical solutions. To avoid repetition, it will not be elaborated here.

[0072] The fourth aspect of the present invention provides a readable storage medium on which programs or instructions are stored. When the programs or instructions are executed by a processor, they can implement the steps of the tracking method provided in any of the above technical solutions. Therefore, this readable storage medium also includes all the beneficial effects of the tracking method provided in any of the above technical solutions. To avoid repetition, it will not be elaborated here.

[0073] The fifth aspect of the present invention provides an electronic device, including: the tracking device provided in any of the above technical solutions; and / or the readable storage medium provided in any of the above technical solutions. Therefore, this electronic device also includes all the beneficial effects of the tracking device and / or the readable storage medium provided in any of the above technical solutions. To avoid repetition, it will not be elaborated here.

[0074] The sixth aspect of the present invention provides a robot, including: the tracking device provided in any of the above technical solutions; and / or the readable storage medium provided in any of the above technical solutions. Therefore, this robot also includes all the beneficial effects of the tracking device and / or the readable storage medium provided in any of the above technical solutions. To avoid repetition, it will not be elaborated here.

[0075] It can be understood that the above robots include floor-sweeping robots, service robots, engineering robots, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The above and / or additional aspects and advantages of the present invention will become apparent and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0077] Figure 1 The flowchart of the tracking method according to an embodiment of the present application is shown;

[0078] Figure 2 The schematic diagram of detecting an object according to an embodiment of the present application is shown;

[0079] Figure 3 The schematic diagram of double - sphere sampling according to an embodiment of the present invention is shown;

[0080] Figure 4 The schematic diagram of object tracking according to an embodiment of the present invention is shown;

[0081] Figure 5 The schematic diagram of calculating pose change according to an embodiment of the present invention is shown;

[0082] Figure 6 The block diagram of the structure of the tracking device according to an embodiment of the present invention is shown. Detailed implementation manners

[0083] In order to more clearly understand the above - mentioned objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific implementation manners. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0084] Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0085] The following refers to Figures 1 to 6 Describe the tracking method, tracking device, readable storage medium, electronic device and robot according to some embodiments of the present invention.

[0086] Embodiment 1

[0087] As Figure 1 shown, in the embodiment of the first aspect of the present invention, a tracking method is provided. Figure 1 The flowchart of the tracking method according to an embodiment of the present application is shown. As Figure 1 shown, the method includes:

[0088] Step 102, determine the local point cloud data of the target object according to the point cloud information;

[0089] Step 104, perform a complement processing on the local point cloud data to obtain the overall point cloud data of the target object, where the local point cloud data is unordered point cloud data and the overall point cloud data is ordered point cloud data;

[0090] Step 106: Track the target object based on the overall point cloud data.

[0091] In the embodiments of the present invention, for movable robots such as sweeping robots and service robots, when they perform various behaviors, they need to obtain sufficient information in the environment, continuously perceive the surrounding objects, track the objects, and perform dynamic path planning. And the continuous perception of objects, that is, the accuracy of tracking objects, will have a greater impact on the intelligence of the robot. Therefore, it is necessary to ensure high-precision object tracking.

[0092] In the embodiments of the present application, the above-mentioned point cloud information can specifically be the point cloud information obtained by devices such as lidar. Due to the characteristics of lidar, the point cloud information it obtains belongs to unordered point cloud. And for a robot, since lidar can only obtain the point cloud information on the side of the object facing the radar transmitting station, in a frame of point cloud obtained by lidar, the point cloud of the target object included is local point cloud.

[0093] The lidar obtains the surrounding point cloud data in real time and gets a series of consecutive frames of point cloud arranged in chronological order. After obtaining a frame of point cloud, the objects in the point cloud can be detected through an object detection algorithm, such as the CenterPoint function (a function that can return the center coordinates of a rectangular area). Figure 2 Shows a schematic diagram of detecting an object according to an embodiment of the present application, as Figure 2 shown, each cube box is an object such as a detected table, chair, person, etc. These detected objects are the above-mentioned target objects.

[0094] It can be understood that during the object detection process, at the position of the same object, multiple candidate boxes may be detected, and these candidate boxes may overlap. At this time, the non-maximum suppression algorithm can be used to eliminate redundant detection boxes.

[0095] According to the position of the final detection box, the local point cloud data of the target object can be obtained. This local point cloud data is the unordered point cloud data obtained by the lidar. To ensure the accuracy of object tracking, in the embodiments of the present application, through a preset completion algorithm, the local point cloud data of the target object is supplemented into complete overall cloud data, and this overall point cloud data is ordered point cloud.

[0096] Specifically, the overall point cloud data is a complete point cloud cluster of an object obtained by an incomplete algorithm based on a double-sphere sampling model. Among them, the double-sphere sampling model is a process of forming an ordered point cloud by double-sphere sampling of the original point cloud of a three-dimensional model. Specifically, first, based on the shape of the three-dimensional model, an enclosing sphere of the model is generated. On the enclosing sphere, a plurality of sampling points are uniformly sampled. Taking these sampling points as endpoints, rays are projected onto the three-dimensional model. According to the first intersection points of these rays and the three-dimensional model, the point cloud data of the three-dimensional model can be obtained.

[0097] Among them, the rays are emitted in a certain order, and the positions of the sampling points on the enclosing sphere are fixed. Therefore, the number of sampling rays is the same, and the sampling position points and order are the same. The obtained point cloud cluster of the three-dimensional model is a complete ordered point cloud.

[0098] By complementing and processing the unordered local point cloud data generated by the lidar into ordered overall point cloud data, the integrity of the observation is improved. When processing continuous multi-frame point clouds, the matching accuracy of the same object is greatly improved, thus realizing high-precision object registration and object tracking.

[0099] Based on any of the above embodiments, the point cloud information includes first point cloud information and second point cloud information; tracking the target object according to the overall point cloud data includes: determining the first overall point cloud data of the first object in the first point cloud information, and determining the second overall point cloud data of the second object in the second point cloud information, and determining the difference degree between the first object and the second object; in the case where the difference degree is less than the threshold, determining the first object and the second object as the target object.

[0100]

[0101] First, in the first point cloud data, identify the local unordered point cloud of the first object and perform a complementing process on it to obtain the overall ordered point cloud of the first object. Then, in the second point cloud data, identify the local unordered point cloud of the second object and perform a complementing process on it to obtain the overall ordered point cloud of the second object.

[0102] ​After obtaining the first overall point cloud data and the second overall point cloud data, the difference between the first object and the second object is obtained by comparing the first overall point cloud data and the second overall point cloud data. It can be understood that when the difference between the first object and the second object is large, such as when the difference is greater than a preset threshold, it is determined that the first object and the second object are not the same object.

[0103] If the difference between the first object and the second object is small, specifically less than a preset threshold, it is determined that the first object and the second object are the same object and are both the above-mentioned target objects, thereby achieving accurate tracking of the same object in different time dimensions.

[0104] Based on any of the above embodiments, the difference includes a target chamfer distance; determining the difference between the first object and the second object includes: projecting the second overall point cloud data into the coordinate system of the first point cloud information to obtain a third overall point cloud data, wherein the third overall point cloud data includes N first point clouds, and the first overall point cloud data includes N second point clouds, where N is a positive integer; determining N groups of point cloud groups that correspond one to one based on the N first point clouds and the N second point clouds; determining the target chamfer distance based on the average value of the chamfer distances of the N groups of point clouds.

[0105] In an embodiment of the present invention, the difference between the first object and the second object specifically includes the chamfer distance (Chamfer distance) between each point cloud in the first overall point cloud data and the corresponding point cloud in the second point cloud data, that is, the target chamfer distance.

[0106] Specifically, the second overall point cloud data is projected onto the coordinate system corresponding to the first point cloud information. Ideally, if the first object and the second object are the same object, the second overall point cloud data will overlap with the second overall point cloud data after being projected onto the first point cloud information. However, in real-world scenarios, due to errors, noise, and other factors, the first overall point cloud data projected onto the first point cloud information and the second overall point cloud data may not necessarily overlap.

[0107] Here, the first overall point cloud data is denoted as As, the first overall point cloud data after being projected onto the first point cloud information is denoted as As', and the second overall point cloud data is denoted as Bs. For the N sets of point clouds (a, b) that correspond one-to-one in As' and Bs, their Chamfer distance is calculated.

[0108] During the calculation process, first, select a point p in a', select the point p' closest to point p from b, and calculate the Chamfer distance d between p and p'.

[0109] Then, traverse all the points in a and all the corresponding points in b, calculate and accumulate the Chamfer distances between them respectively, and then divide by the total number of points to obtain the average value, which is the target chamfer distance.

[0110] It can be understood that when the Chamfer distance is larger, it indicates that the shape difference between the first object and the second object is larger; on the contrary, when the Chamfer distance is smaller, it means that the shapes of the first object and the second object are closer. Therefore, when the Chamfer distance is less than the threshold, it can be considered that the first object and the second object are similar, and at this time, the first object and the second object are associated as the same target object, achieving the purpose of tracking the object.

[0111] Based on any of the above embodiments, the method for complementing local point cloud data includes: complementing the local point cloud data through a preset complementing algorithm model.

[0112] In the embodiments of the present invention, the local point cloud data of an object is complemented through a pre-trained complementing algorithm model to generate and obtain the overall point cloud data corresponding to the object. Among them, the complementing algorithm model is a neural network model based on machine learning training. Therefore, after the local point cloud data is input into the complementing algorithm model, the complementing algorithm model can output the predicted overall point cloud data to achieve the complementing of the local point cloud data.

[0113] Based on any of the above embodiments, before complementing the local point cloud data, the method further includes: cropping the point cloud data of the 3D model to obtain the local point cloud corresponding to the 3D model; sampling the 3D model to obtain the overall ordered point cloud corresponding to the 3D model; training a preset neural network model through the local point cloud and the overall ordered point cloud to obtain the complementing algorithm model.

[0114] In the embodiments of the present invention, when training the complementing algorithm model, a training set needs to be processed first. Among them, the training set includes two parts. The first part is the local point cloud of the object, and the second part is the overall ordered point cloud of the same object.

[0115] Specifically, first obtain a 3D model, which is a model established by unordered point cloud. Cropping this model can obtain the local point cloud of the 3D model. Then, through the method of double-sphere sampling, sample the 3D model to establish the ordered point cloud corresponding to the 3D model, and this ordered point cloud is the overall point cloud.

[0116] Taking the overall ordered point cloud obtained by acquisition as the target and the partial point cloud obtained by cutting as the material, training a preset neural network model. After the gap between the point cloud data output by the model and the overall ordered point cloud is less than the set value, it is determined that the model training is completed, and a complementary algorithm model that meets the requirements is obtained.

[0117] Based on any of the above embodiments, sampling the 3D model includes: determining a first sphere according to the 3D model, with the 3D model located inside the first sphere; performing Fibonacci spherical sampling on the spherical surface of the first sphere to obtain first sampling points; generating rays with each first sampling point as an endpoint; and determining the overall ordered point cloud according to the generation order of the rays and the intersection points of the rays with the surface of the 3D model.

[0118] In the embodiments of the present invention, double-sphere sampling is performed on the 3D model. Specifically, based on the shape of the 3D model, an enclosing sphere of the model, that is, the above-mentioned first sphere, is generated. It can be understood that the first sphere is a virtual sphere. For this first sphere, a 3D model can be formed, or a 3D model can be not established, and only the coordinate data of each point on the spherical surface of the first sphere is determined.

[0119] After obtaining the first sphere, that is, the enclosing sphere of the 3D model, Fibonacci spherical sampling is performed on the spherical surface of the enclosing sphere. Specifically, the Fibonacci spherical sampling method is an approximate algorithm that can generate uniformly distributed sampling points on the unit sphere, where the unit sphere can be expressed as: Its main step is to map the two-dimensional Fibonacci grid onto the spherical surface. The definition of the Fibonacci grid is a unit square, which can be expressed as [0, 1) 2 , where any number of n points are uniformly distributed. Among them, the two-dimensional point with index i can be expressed as:

[0120]

[0121] Among them, p i is the two-dimensional point with index i, (x i , y i ) are the two-dimensional coordinates of the point p i , n is the number of points uniformly distributed in the Fibonacci grid, and the %1 operator represents taking the fractional part of the parameter.

[0122] Using cylindrical equal-area projection to map the two-dimensional points on the Fibonacci grid onto the unit sphere can obtain uniformly distributed three-dimensional points. The above point p i can be expressed in spherical coordinates as:

[0123]

[0124] Through Fibonacci spherical sampling, N uniformly distributed points sampled on the enclosing sphere can be regarded as virtual cameras for photographing a three-dimensional model. The M rays determined based on the N first sampling points can be regarded as the virtual rays corresponding to these virtual cameras, and the projection process of the virtual cameras is simulated through the virtual rays.

[0125] Specifically, each first sampling point can generate multiple rays. The endpoints of these rays are the first sampling points, and the directions of the rays are towards the three-dimensional model, and the direction of each projected ray remains unchanged.

[0126] Since these virtual cameras are spherically distributed, the viewing directions of the virtual cameras simulated by the first sampling points are more diverse, which can effectively avoid the "self-occlusion" problem that the prominent structure of the model itself blocks the projection rays due to a single acquisition direction.

[0127] Figure 3 Shows a schematic diagram of double-sphere sampling according to an embodiment of the present invention, as Figure 3 shown, when the ray intersects the surface of the three-dimensional model, the distance between the first intersection point of the ray and the three-dimensional model and the first sampling point that emits the ray is the point cloud data of the intersection point. Since the emission direction of the ray is known and fixed, when regenerating the point cloud of the three-dimensional model, according to the distance information of each point position, the point cloud data of the three-dimensional model can be easily restored. And since the arrangement of the first sampling point and the ray is fixed after determining the first sampling point, the obtained point cloud data of the three-dimensional model is ordered point cloud data, so the overall ordered point cloud of the three-dimensional model is obtained.

[0128] Based on any of the above embodiments, generating rays with each first sampling point as an endpoint includes: generating a second sphere with the first sampling point as the center; performing Fibonacci spherical sampling on the spherical surface of the second sphere to obtain second sampling points; generating rays passing through the second sampling points with the first sampling point as the endpoint.

[0129] In the embodiment of the present invention, when generating rays based on the first sampling point, the first sampling point is used as the endpoint of the ray. To determine the direction of the ray, in this application embodiment, a second sphere is generated with the first sampling point as the center, and a plurality of uniformly distributed second sampling points are sampled on the spherical surface of the second sphere through the Fibonacci sampling method.

[0130] Taking the example of sampling 32 points on the first sphere and 64 points on the second sphere, a total of 32×64 = 2048 rays can be generated. These rays pass through each of the 64 second sampling points respectively, and finally 2048 sampling values can be obtained. After screening out the rays that do not intersect the three-dimensional model by using the clustering algorithm, the overall ordered point cloud of the three-dimensional model can be obtained.

[0131] Based on any of the above embodiments, the degree of difference further includes a rotation matrix and a translation amount; determining the degree of difference between the first object and the second object further includes: respectively determining a first point set and a second point set according to the first generation order corresponding to the first overall point cloud data and the second generation order corresponding to the second overall point cloud data; performing singular value decomposition on the first point set and the second point set to obtain the rotation matrix and the translation amount.

[0132] In the embodiments of the present invention, the degree of difference between the first object and the second object further includes the pose difference between the first object and the second object, specifically including a rotation matrix and a translation amount. The rotation matrix and the translation amount can be used to evaluate the pose difference between the first object and the second object. When the pose difference is smaller, it means they are closer; on the contrary, the difference is larger.

[0133] Here, the first overall point cloud data is denoted as As, the first overall point cloud data projected onto the first point cloud information is denoted as As', and the second overall point cloud data is denoted as Bs. For the N groups of corresponding point clouds (a, b) in As' and Bs, calculate their pose changes.

[0134] Among them, according to the sampling order when sampling the first object and the second object by the double-sphere sampling method, that is, the generation order of the rays, the corresponding sampling sequences S a and S b are respectively determined. According to the orderliness of the sampled points obtained by sampling, the point sets a' and b' corresponding to the intersection of the sampling serial numbers are respectively selected, that is, the above-mentioned first point set and second point set.

[0135] By performing singular value decomposition to solve the rotation matrix and the translation amount of the point sets a' and b', and by combining the Chamfer distance, the rotation matrix and the translation amount, it is determined whether the objects in the two frames of point clouds, that is, the first point cloud information and the second point cloud information, are the same object, so as to realize the observation of the same object at different times, so that the objects at different times can be correlated with each other and identified as the same target object (assigned the same identification information), thus achieving accurate object tracking.

[0136] Based on any of the above embodiments, determining the first sphere according to the three-dimensional model includes: determining the maximum value of the point position coordinates of the three-dimensional model; determining the bounding box of the three-dimensional model according to the maximum value; determining the first sphere based on the bounding box.

[0137] In the embodiments of the present invention, the first sphere is the bounding sphere of the three-dimensional model. In order to ensure that the bounding sphere can completely enclose the three-dimensional model, the bounding box of the three-dimensional model can be determined first. Specifically, the bounding box of the three-dimensional model can be found by calculating the difference between the extreme values of the point coordinates of the three-dimensional model in each dimension.

[0138] Among them, the bounding box is used to track the spatial position of an object in 3D object detection. The bounding box can be a cubic bounding box or a polyhedral bounding box. This bounding box can express the outermost boundaries of the object in all directions. Therefore, determining the first sphere based on this bounding box can ensure that the first sphere completely encloses the 3D model and guarantee the integrity of the point cloud data.

[0139] Based on any of the above embodiments, the first sphere is the circumscribed sphere of the bounding box.

[0140] In the embodiments of the present invention, since the first sphere is the circumscribed sphere of the bounding box, on the basis of ensuring that the first sphere completely encloses the 3D model, it can avoid a situation where the first sphere is too large and there is a large amount of blank space inside the first sphere. Thus, the number of invalid rays that do not intersect with the 3D object is reduced, the amount of operations and calculations for determining the point cloud data is decreased, and the generation efficiency of the point cloud data is improved.

[0141] Embodiment Two

[0142] In some embodiments of the present invention, based on the sampling method of the double-sphere model, an ordered set of sampled points on the object surface is used to represent the object. As Figure 3 shown, this sampling model consists of two types of spheres. Among them, the large sphere is the circumscribed sphere of the detected object. The center of the small sphere is on the surface of the large sphere, and the center of the small sphere and the emission point of the sampling ray are obtained by the Fibonacci sampling method. The sampling ray is emitted from the center of each small sphere towards the inside of the large sphere, and the ratio of the length from the center of the small sphere to the object surface to the diameter of the large sphere is taken as the sampling value of this point. Therefore, when the settings of the sampling model are the same, for the same object, the number of sampling rays is the same, and the sampling position points and order are the same.

[0143] Registration and correlation method based on reprojection: According to the observation of the previous frame, the complete point cloud cluster of the object can be obtained according to the completion algorithm based on the double-sphere sampling model. Then, combined with the pose of the observer, this point cloud cluster can be transformed into the coordinate system of the current frame. Assuming that the displacement of the object between two frames is very small, the transformed point cloud cluster is very close to the point cloud cluster completed according to the observation of the current frame. Calculate the Chamfer distance and pose transformation between them respectively, and evaluate the similarity of their shapes and the change of poses. It can be considered that the more similar their shapes are and the closer their poses are, the greater the possibility that they are the same object. Thus, the same object at different times is correlated with each other, and the continuous trajectory of the object is further obtained, that is, tracking the object is realized.

[0144] According to the sampling characteristics of the double-sphere sampling model, for the same object, the sampling points always remain unchanged, and the corresponding completed object point cloud also remains consistent. When detecting and completing objects in different frames, when using the registration algorithm to compare object pairs, only need to compare each pair of points in the point clouds of the two in sequence. If it is the same object, the point clouds of the two are theoretically exactly the same. Compared with registration algorithms such as ICP, using the orderliness of the completed point cloud can greatly improve the speed and efficiency of registration.

[0145] During the reprojection process, since the completion algorithm improves the integrity of the observation, the registration accuracy between the object projected from the previous frame in the current frame and the object in the current frame will be due to the registration between some point clouds.

[0146] Figure 4 Shows a schematic diagram of object tracking according to an embodiment of the present invention, as Figure 4 shown. Generally, a frame of lidar point cloud usually contains a large number of points, which are discretely distributed in space. Reading in each frame of point cloud in chronological order, configuring point cloud-based object detection methods such as CenterPoint, detecting the point cloud clusters of objects such as cars and pedestrians, and giving the corresponding detection boxes. Generally speaking, the detection box can provide the central point position, size specification, and orientation of the point cloud, etc.

[0147] According to the detection box information and point cloud cluster of the object, a completion algorithm based on the double-sphere sampling model can be used to generate a complete and ordered object point cloud. When the large-sphere sampling points are set to 32 and the sampling points on the small sphere are set to 64, there are a total of 32×64 = 2048 sampling rays, and finally 2048 sampling values can be obtained. Use the clustering method to filter out the sampling points that do not hit the object surface, and then the completed point cloud at the same scale as the object can be restored according to the diameter of the sampling sphere.

[0148] As Figure 4 shown, A and B are the point cloud clusters of the object W in the (k - 1)-th frame and the k-th frame respectively. The pose change between the (k - 1)-th frame and the k-th frame is T k . During the object detection process, a large number of candidate boxes will be generated at the position of the same object, and there may be overlaps between them.

[0149] The non-maximum suppression algorithm can be used to eliminate redundant detection boxes. Taking A as an example, when there are multiple detection boxes corresponding to it, multiple completed point cloud clusters can be generated. For the convenience of expression, we call the completed point clouds based on A and B the overall point clouds As and Bs.

[0150] Project As onto the k-th frame, denoted as As'. Ideally, As' should coincide with the completed point cloud Bs of the object B detected in the current frame. However, in the presence of errors and noise, As' and the completed point cloud Bs may not fully coincide. To select the most accurate pose of the object in the two frames, calculate the Chamfer distance and pose transformation between each point cloud in As' and each point cloud in the completed point cloud Bs.

[0151] For each pair of point clouds (a, b) in As' and Bs, calculate the Chamfer distance:

[0152] 1. Select a point p in a, and select the point p' closest to p from b, and calculate the distance d between p and p';

[0153] 2. Traverse all points in a and accumulate the distance d;

[0154] 3. Calculate the average distance between all points p and p', which is the Chamfer distance.

[0155] Figure 5 Shows a schematic diagram of calculating the pose change according to an embodiment of the present invention, as Figure 5 shown, calculate the pose change:

[0156] 1. As Figure 5 shown, (a, b) are respectively restored and generated from the filtered sampling sequences Sa and Sb. According to the orderliness of the sampling points, select the point sets a' and b' corresponding to the intersection of the sampling serial numbers respectively;

[0157] 2. Use SVD decomposition to solve the rotation matrix and translation amount of a' and b'.

[0158] Generally, the Chamfer distance is used to characterize the difference in shape between the two. When the Chamfer distance is larger, it indicates that the shape difference between the two is larger. The rotation matrix and translation amount can be used to evaluate the pose difference between the two. When the difference is smaller, it indicates that they are closer. Combining the above two criteria, we can consider that two objects with similar shapes and close positions are observations of the same object at different times. In other words, objects at different times can be correlated with each other and given the same identifier, thus achieving the purpose of tracking.

[0159] Embodiment III

[0160] In some embodiments of the present invention, a tracking device is provided, Figure 6 Shows a structural block diagram of the tracking device according to an embodiment of the present invention, as Figure 6 shown, the tracking device 600 includes:

[0161] A determination module 602, configured to determine local point cloud data of a target object according to point cloud information;

[0162] A processing module 604 is configured to perform a completion process on the local point cloud data to obtain the overall point cloud data of the target object, where the local point cloud data is unordered point cloud data, and the overall point cloud data is ordered point cloud data;

[0163] A tracking module 606 is configured to track the target object according to the overall point cloud data.

[0164] In an embodiment of the present invention, for a movable robot, such as a sweeping robot, a service robot, etc., when it performs various behaviors, it needs to obtain sufficient information in the environment, continuously sense the surrounding objects and track the objects, and perform dynamic path planning. And continuously sensing the object, that is, the accuracy of tracking the object, will have a greater impact on the intelligence of the robot. Therefore, high-precision object tracking needs to be ensured.

[0165] In an embodiment of the present application, the above-mentioned point cloud information can specifically be the point cloud information obtained by devices such as lidar. Due to the characteristics of lidar, the point cloud information it obtains belongs to unordered point cloud. And for a robot, since lidar can only obtain the point cloud information on the side of the object facing the radar transmitting station, in a frame of point cloud obtained by lidar, the point cloud of the target object included is local point cloud.

[0166] The lidar continuously obtains the surrounding point cloud data in real time to obtain a series of consecutive frames of point cloud arranged in chronological order. After obtaining a frame of point cloud, object detection algorithms can be used, such as the center point function (CenterPoint, a function that can return the center coordinates of a rectangular area), to detect the objects in the point cloud, such as detecting a table, a chair, a person, etc. These detected objects are the above-mentioned target objects.

[0167] It can be understood that during the object detection process, at the position of the same object, multiple candidate boxes may be detected, and these candidate boxes may overlap. At this time, the non-maximum suppression algorithm can be used to eliminate redundant detection boxes.

[0168] According to the position of the final detection box, the local point cloud data of the target object can be obtained. This local point cloud data is the unordered point cloud data obtained by the lidar. To ensure the accuracy of object tracking, in an embodiment of the present application, through a preset completion algorithm, the local point cloud data of the target object is supplemented into complete overall cloud data, and this overall point cloud data is ordered point cloud.

[0169] Specifically, the overall point cloud data is a complete algorithm based on the double-sphere sampling model, obtaining a complete point cloud cluster of the object. Among them, the double-sphere sampling model is a process of forming an ordered point cloud by double-sphere sampling of the original point cloud of the three-dimensional model. Specifically, first, based on the shape of the three-dimensional model, an enclosing sphere of the model is generated. On the enclosing sphere, multiple sampling points are evenly sampled. Taking these sampling points as endpoints, rays are projected towards the three-dimensional model. According to the first intersection points of these rays with the three-dimensional model, the point cloud data of the three-dimensional model can be obtained.

[0170] Among them, the rays are emitted in a certain order, and the positions of the sampling points on the enclosing sphere are fixed. Therefore, the number of sampling rays is consistent, and the sampling position points and order are consistent. The obtained point cloud cluster of the three-dimensional model is a complete ordered point cloud.

[0171] By complementing and processing the disordered local point cloud data generated by the lidar into ordered overall point cloud data, the integrity of the observation is improved. When processing continuous multiple frames of point clouds, the matching accuracy of the same object is greatly improved, thus realizing high-precision object registration and object tracking.

[0172] Based on any of the above embodiments, the point cloud information includes first point cloud information and second point cloud information; the root determination module is further configured to: determine the first overall point cloud data of the first object in the first point cloud information, and determine the second overall point cloud data of the second object in the second point cloud information, and determine the difference degree between the first object and the second object; in the case where the difference degree is less than the threshold, determine the first object and the second object as the target objects.

[0173] In the embodiments of the present invention, the lidar obtains the surrounding point cloud data in real time, obtaining continuous multiple frames of point clouds arranged in chronological order. Among them, the first point cloud information and the second point cloud information are specifically two adjacent frames of point clouds, that is to say, the first point cloud information and the second point cloud information have temporal continuity. It can be understood that the first point cloud information is obtained prior to the second point cloud information.

[0174] First, in the first point cloud data, identify the local disordered point cloud of the first object and perform a complementation process on it to obtain the overall ordered point cloud of the first object. Then, in the second point cloud data, identify the local disordered point cloud of the second object and perform a complementation process on it to obtain the overall ordered point cloud of the second object.

[0175] After obtaining the first overall point cloud data and the second overall point cloud data, by comparing the first overall point cloud data and the second overall point cloud data, the difference degree between the first object and the second object is obtained. It can be understood that when the difference degree between the first object and the second object is large, such as when the difference degree is greater than the preset threshold, it is determined that the first object and the second object are not the same object.

[0176] If the difference degree between the first object and the second object is small, specifically less than a preset threshold, it is determined that the first object and the second object are the same object, both being the above-mentioned target object, thereby achieving accurate tracking of the same object in different time dimensions.

[0177] Based on any of the above embodiments, the difference degree includes the target chamfer distance; the tracking device further includes: a projection module for projecting the second overall point cloud data into the coordinate system of the first point cloud information to obtain third overall point cloud data, where the third overall point cloud data includes N first point clouds, the first overall point cloud data includes N second point clouds, and N is a positive integer;

[0178] The determination module is further configured to determine N groups of corresponding point cloud groups according to the N first point clouds and the N second point clouds; and determine the target chamfer distance according to the average value of the chamfer distances of the N groups of point cloud groups.

[0179] In the embodiments of the present invention, the difference degree between the first object and the second object specifically includes the chamfer distance (Chamfer distance) between each point cloud in the first overall point cloud data and the corresponding point cloud in the second point cloud data, that is, the above-mentioned target chamfer distance.

[0180] Specifically, the second overall point cloud data is projected into the coordinate system corresponding to the first point cloud information. In an ideal situation, if the first object and the second object are the same object, then after the second overall point cloud data is projected into the first point cloud information, it will coincide with the second overall point cloud data. In an actual scenario, due to the existence of errors, noises, etc., the first overall point cloud data projected into the first point cloud information and the second overall point cloud data may not coincide.

[0181] Here, the first overall point cloud data is denoted as As, the first overall point cloud data after being projected into the first point cloud information is denoted as As’, and the second overall point cloud data is denoted as Bs. For the N groups of corresponding point clouds (a, b) in As’ and Bs, calculate their Chamfer distances.

[0182] During the calculation process, first, in a’, randomly select a point p, and select the point p’ closest to the point p from b, and calculate the Chamfer distance d between p and p’.

[0183] Then, traverse all the points in a and all the corresponding points in b, calculate and accumulate the Chamfer distances between them respectively, and divide by the total number of points. The obtained average value is the target chamfer distance.

[0184] It can be understood that when the Chamfer distance is larger, it indicates that the shape difference between the first object and the second object is greater. Conversely, when the Chamfer distance is smaller, it means that the shapes of the first object and the second object are closer. Therefore, when the Chamfer distance is less than the threshold, it can be considered that the first object and the second object are similar. At this time, the first object and the second object are associated as the same target object, achieving the purpose of tracking the object.

[0185] Based on any of the above embodiments, the processing module is further configured to perform a completion process on the local point cloud data through a preset completion algorithm model.

[0186] In the embodiments of the present invention, the local point cloud data of an object is completed through a pre-trained completion algorithm model, thereby generating and obtaining the overall point cloud data corresponding to the object. Among them, the completion algorithm model is a neural network model trained based on machine learning. Therefore, after the local point cloud data is input into the completion algorithm model, the completion algorithm model can output the predicted overall point cloud data to complete the local point cloud data.

[0187] Based on any of the above embodiments, the tracking device further includes: a cutting module for cutting the point cloud data of the three-dimensional model to obtain the local point cloud corresponding to the three-dimensional model; a sampling module for sampling the three-dimensional model to obtain the overall ordered point cloud corresponding to the three-dimensional model; a training module for training a preset neural network model through the local point cloud and the overall ordered point cloud to obtain a completion algorithm model.

[0188] In the embodiments of the present invention, when training the completion algorithm model, it is first necessary to process and obtain a training set. Among them, the training set includes two parts. The first part is the local point cloud of the object, and the second part is the overall ordered point cloud of the same object.

[0189] Specifically, first obtain a three-dimensional model, which is a model established through unordered point clouds. Cutting this model can obtain the local point cloud of the three-dimensional model. Then, through the method of dual-sphere sampling, sample the three-dimensional model to establish the ordered point cloud corresponding to the three-dimensional model, and this ordered point cloud is the overall point cloud.

[0190] Taking the collected overall ordered point cloud as the target and the cut local point cloud as the material, train a preset neural network model. When the gap between the point cloud data output by the model and the overall ordered point cloud is less than the set value, it is determined that the model training is completed, and a completion algorithm model that meets the requirements is obtained.

[0191] Based on any of the above embodiments, the determination module is further configured to determine a first sphere according to the three-dimensional model, where the three-dimensional model is located inside the first sphere; the sampling module is further configured to perform Fibonacci spherical sampling on the spherical surface of the first sphere to obtain first sampling points; the tracking device further includes: a generation module, configured to generate rays with each first sampling point as an endpoint; the determination module is further configured to determine an overall ordered point cloud according to the generation order of the rays and the intersection points of the rays with the surface of the three-dimensional model.

[0192] In the embodiments of the present invention, double-sphere sampling is performed on the three-dimensional model. Specifically, based on the shape of the three-dimensional model, an enclosing sphere of the model is generated, that is, the above-mentioned first sphere. It can be understood that the first sphere is a virtual sphere. For this first sphere, a three-dimensional model can be formed, or a three-dimensional model may not be established, and only the coordinate data of each point position on the spherical surface of the first sphere is determined.

[0193] After obtaining the first sphere, that is, the enclosing sphere of the three-dimensional model, Fibonacci spherical sampling is performed on the spherical surface of the enclosing sphere. Specifically, the Fibonacci spherical sampling method is an approximate algorithm that can generate uniformly distributed sampling points on the unit sphere, where the unit sphere can be expressed as: Its main step is to map the two-dimensional Fibonacci grid onto the spherical surface. The definition of the Fibonacci grid is a unit square, which can be expressed as [0, 1) 2 , where any number n of points are uniformly distributed. Among them, the two-dimensional point with index i can be expressed as:

[0194]

[0195] Among them, p i is the two-dimensional point with index i, (x i , y i is the two-dimensional coordinate of the point p i , n is the number of points uniformly distributed in the Fibonacci grid, and the %1 operator represents taking the fractional part of the parameter.

[0196] Using cylindrical equal-area projection to map the two-dimensional points on the Fibonacci grid onto the unit sphere can obtain uniformly distributed three-dimensional points. The above point p i can be expressed in the spherical coordinate system as:

[0197]

[0198] Through Fibonacci spherical sampling, the N uniformly distributed points sampled on the enclosing sphere can be regarded as virtual cameras for photographing the three-dimensional model. The M rays determined according to the N first sampling points can be regarded as virtual light rays corresponding to these virtual cameras. The projection process of the virtual cameras is simulated through the virtual light rays.

[0199] Specifically, each first sampling point can generate multiple rays. The endpoints of these rays are the first sampling points, and the directions of the rays are towards the three-dimensional model, and the direction of each projected ray remains unchanged.

[0200] Since these virtual cameras are spherically distributed, the viewing directions of the virtual cameras simulated by the first sampling points towards the three-dimensional model are more diverse, which can effectively avoid the "self-occlusion" problem that the prominent structures of the model itself block the projection rays due to a single acquisition direction.

[0201] When the ray intersects with the surface of the three-dimensional model, the distance between the first intersection point of the ray and the three-dimensional model and the first sampling point that emits the ray is the point cloud data of that intersection point. Since the emission direction of the ray is known and fixed, when regenerating the point cloud of the three-dimensional model, the point cloud data of the three-dimensional model can be easily restored according to the distance information of each point. And since after determining the first sampling point, the arrangement of the first sampling point and the rays is a fixed arrangement, the obtained point cloud data of the three-dimensional model is ordered point cloud data, so the overall ordered point cloud of the three-dimensional model is obtained.

[0202] Based on any of the above embodiments, the generation module is further configured to generate a second sphere with the first sampling point as the center; the sampling module is further configured to perform Fibonacci spherical sampling on the spherical surface of the second sphere to obtain second sampling points; the generation module is further configured to generate a ray passing through the second sampling point with the first sampling point as the endpoint.

[0203] In the embodiments of the present invention, when generating a ray based on the first sampling point, the first sampling point is used as the endpoint of the ray. To determine the direction of the ray, in the embodiments of the present application, a second sphere is generated with the first sampling point as the center, and a plurality of second sampling points evenly distributed are sampled on the spherical surface of the second sphere by the Fibonacci sampling method.

[0204] Taking the example of sampling 32 points on the first sphere and 64 points on the second sphere, a total of 32×64 = 2048 rays can be generated. These rays pass through each of the 64 second sampling points respectively, and finally 2048 sampling values can be obtained. After screening out the rays that do not intersect with the three-dimensional model by using a clustering algorithm, the overall ordered point cloud of the three-dimensional model can be obtained.

[0205] Based on any of the above embodiments, the difference degree further includes a rotation matrix and a translation amount; the determination module is further configured to: respectively determine a first point set and a second point set according to the first generation order corresponding to the first overall point cloud data and the second generation order corresponding to the second overall point cloud data; perform singular value decomposition on the first point set and the second point set to obtain the rotation matrix and the translation amount.

[0206] In the embodiments of the present invention, the degree of difference between the first object and the second object further includes the pose difference between the first object and the second object, specifically including the rotation matrix and the translation amount. The rotation matrix and the translation amount can be used to evaluate the pose difference between the first object and the second object. When the pose difference is smaller, it means they are closer; on the contrary, the difference is larger.

[0207] Here, the first overall point cloud data is denoted as As, the first overall point cloud data after being projected onto the first point cloud information is denoted as As’, and the second overall point cloud data is denoted as Bs. For the N groups of corresponding point clouds (a, b) in As’ and Bs, calculate their pose changes.

[0208] Among them, according to the sampling order when sampling the first object and the second object by the double-sphere sampling method, that is, the generation order of the rays, the corresponding sampling sequences S a and S b are respectively determined. According to the orderliness of the sampled points obtained by sampling, the point sets a’ and b’ corresponding to the intersection of the sampling numbers are respectively selected, that is, the above-mentioned first point set and second point set.

[0209] By performing singular value decomposition to solve the rotation matrix and the translation amount of the point sets a’ and b’, and by combining the Chamfer distance, the rotation matrix and the translation amount, it is determined whether the objects in the two frames of point clouds, that is, the first point cloud information and the second point cloud information, are the same object, so as to realize the observation of the same object at different times, so that the objects at different times can be correlated with each other and identified as the same target object (given the same identification information), thus achieving accurate object tracking.

[0210] Based on any of the above embodiments, the determination module is further configured to: determine the maximum value of the point position coordinates of the three-dimensional model; determine the bounding box of the three-dimensional model according to the maximum value; and determine the first sphere based on the bounding box.

[0211] In the embodiments of the present invention, the first sphere is the circumsphere of the three-dimensional model. In order to ensure that the circumsphere can completely enclose the three-dimensional model, the bounding box of the three-dimensional model can be determined first. Specifically, the bounding box of the three-dimensional model can be found by calculating the difference between the extreme values of the point coordinates of the three-dimensional model in each dimension.

[0212] Among them, the bounding box is used to track the spatial position of the object in three-dimensional object detection. The bounding box can be a cube bounding box or a polyhedron bounding box. The bounding box can express the farthest boundaries of the object in all directions. Therefore, determining the first sphere based on the bounding box can ensure that the first sphere completely encloses the three-dimensional model and ensure the integrity of the point cloud data.

[0213] Based on any of the above embodiments, the first sphere is the circumsphere of the bounding box.

[0214] In an embodiment of the present invention, the first sphere is the circumscribed sphere of the bounding box. Therefore, on the basis of ensuring that the first sphere completely encloses the three-dimensional model, it is possible to avoid the first sphere being too large and resulting in a large amount of blank space inside the first sphere, thereby reducing the number of invalid rays that do not intersect the three-dimensional object, reducing the amount of operations and calculations for determining the point cloud data, and improving the generation efficiency of the point cloud data.

[0215] Embodiment Four

[0216] In some embodiments of the present invention, a tracking device is provided, including: a memory on which a program or instruction is stored; a processor for implementing the steps of the tracking method provided in any of the above embodiments when executing the program or instruction. Therefore, this tracking device also includes all the beneficial effects of the tracking method provided in any of the above embodiments. To avoid repetition, it will not be elaborated here.

[0217] Embodiment Five

[0218] In some embodiments of the present invention, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, it can implement the steps of the tracking method provided in any of the above embodiments. Therefore, this readable storage medium also includes all the beneficial effects of the tracking method provided in any of the above embodiments. To avoid repetition, it will not be elaborated here.

[0219] Embodiment Six

[0220] In some embodiments of the present invention, an electronic device is provided, including: the tracking device provided in any of the above embodiments; and / or the readable storage medium provided in any of the above embodiments. Therefore, this electronic device also includes all the beneficial effects of the tracking device and / or the readable storage medium provided in any of the above embodiments. To avoid repetition, it will not be elaborated here.

[0221] Embodiment Seven

[0222] The sixth aspect of the present invention provides a robot, including: the tracking device provided in any of the above embodiments; and / or the readable storage medium provided in any of the above embodiments. Therefore, this robot also includes all the beneficial effects of the tracking device and / or the readable storage medium provided in any of the above embodiments. To avoid repetition, it will not be elaborated here.

[0223] It can be understood that the above-mentioned robots include floor-sweeping robots, service robots, engineering robots, etc.

[0224] In the description of the present invention, the term "a plurality of" means two or more, unless otherwise clearly defined. The orientation or positional relationship indicated by terms such as "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention; terms such as "connection", "installation", "fixation", etc. should all be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0225] In the description of the present invention, the descriptions of terms such as "one embodiment", "some embodiments", "specific embodiments", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In the present invention, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0226] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A tracking method, characterized in that, It includes: Determine the local point cloud data of the target object according to the point cloud information; Perform a completion process on the local point cloud data to obtain the overall point cloud data of the target object, where the local point cloud data is unordered point cloud data and the overall point cloud data is ordered point cloud data; Track the target object according to the overall point cloud data; Before performing the completion process on the local point cloud data, the method further includes: Crop the point cloud data of the 3D model to obtain the local point cloud corresponding to the 3D model; Sample the 3D model to obtain the overall ordered point cloud corresponding to the 3D model; Train a preset neural network model through the local point cloud and the overall ordered point cloud to obtain a completion algorithm model.

2. The tracking method according to claim 1, wherein The point cloud information includes first point cloud information and second point cloud information; The tracking of the target object according to the overall point cloud data includes: Determine the first overall point cloud data of the first object in the first point cloud information and the second overall point cloud data of the second object in the second point cloud information, and determine the difference degree between the first object and the second object; In the case where the difference degree is less than the threshold, determine the first object and the second object as the target object.

3. The tracking method according to claim 2, characterized in that The difference degree includes a target chamfer distance; The determination of the difference degree between the first object and the second object includes: Project the second overall point cloud data into the coordinate system of the first point cloud information to obtain third overall point cloud data, where the third overall point cloud data includes N first point clouds, the first overall point cloud data includes N second point clouds, and N is a positive integer; Determine N groups of corresponding point cloud groups according to the N first point clouds and the N second point clouds; Determine the target chamfer distance according to the average value of the chamfer distances of the N groups of point cloud groups.

4. The tracking method according to claim 2, wherein The completion process on the local point cloud data includes: Perform a completion process on the local point cloud data through the preset completion algorithm model.

5. The tracking method according to claim 2, characterized in that The sampling of the 3D model includes: Determine a first sphere according to the 3D model, and the 3D model is located inside the first sphere; Perform Fibonacci spherical sampling on the spherical surface of the first sphere to obtain first sampling points; Generate rays with each first sampling point as an endpoint; Determine the overall ordered point cloud according to the generation order of the rays and the intersection points of the rays with the surface of the 3D model.

6. The tracking method according to claim 5, characterized in that The generation of rays with each first sampling point as an endpoint includes: Generate a second sphere with the first sampling point as the center; Perform Fibonacci spherical sampling on the spherical surface of the second sphere to obtain second sampling points; Generate the ray passing through the second sampling point with the first sampling point as the endpoint.

7. The tracking method according to claim 5, wherein The difference degree further includes a rotation matrix and a translation amount; The determination of the difference degree between the first object and the second object further includes: Respectively determine a first point set and a second point set according to the first generation order corresponding to the first overall point cloud data and the second generation order corresponding to the second overall point cloud data; Perform singular value decomposition on the first point set and the second point set to obtain the rotation matrix and the translation amount.

8. The tracking method according to claim 5, wherein The determining the first sphere according to the three-dimensional model includes: Determine the maximum value of the point position coordinates of the three-dimensional model; Determine the bounding box of the three-dimensional model according to the maximum value; Determine the first sphere based on the bounding box.

9. The tracking method according to claim 8, characterized in that The first sphere is the circumscribed sphere of the bounding box.

10. A tracking device, characterized in that, Includes: A determining module, configured to determine local point cloud data of a target object according to point cloud information; A processing module, configured to perform a completion process on the local point cloud data to obtain the overall point cloud data of the target object, where the local point cloud data is unordered point cloud data and the overall point cloud data is ordered point cloud data; A tracking module, configured to track the target object according to the overall point cloud data; A cutting module, configured to cut the point cloud data of the three-dimensional model to obtain the local point cloud corresponding to the three-dimensional model; A sampling module, configured to sample the three-dimensional model to obtain the overall ordered point cloud corresponding to the three-dimensional model; A training module, configured to train a preset neural network model through the local point cloud and the overall ordered point cloud to obtain a completion algorithm model.

11. A tracking device, characterized in that, Includes: A memory, on which programs or instructions are stored; A processor, configured to implement the tracking method according to any one of claims 1 to 9 when executing the programs or instructions.

12. A readable storage medium, on which a program or instructions are stored, characterized in that, The programs or instructions, when executed by the processor, implement the tracking method according to any one of claims 1 to 9.

13. An electronic device, characterized in that, Includes: The tracking device according to claim 10 or 11; And / or The readable storage medium according to claim 12.

14. A robot, characterized in that, Includes: The tracking device according to claim 10 or 11; And / or The readable storage medium according to claim 12.

Images