Method for simplifying point cloud depicting environment

Abstract

The invention relates to a method for simplifying a point cloud depicting an environment, comprising: providing a data set which represents a point cloud on the basis of a coordinate system (400), said point cloud being captured, in particular by means of at least one sensor, from the environment, wherein a grid plane (404) with grid cells (410) is defined by the coordinate system (400); one or more 3D shapes are determined based on the data set, where each 3D shape (430, 432) has a bottom surface (418) and a height (416), where the bottom surface corresponds to one or more consecutive grid cells of a grid plane, respectively, and where the height (416) corresponds to one or more consecutive grid cells of the grid plane. And wherein the height is determined based on points (420) in the point cloud corresponding to the one or more consecutive grid cells by projection; a simplified data set representing the one or more 3D shapes is provided for further use.

Description

Technical Field

[0001] The present invention relates to a method for simplifying the depiction of point clouds of an environment, a computing unit and computer program for performing the method, and a robot. Background of the Invention In various fields, it may be necessary to be able to capture and identify objects in the environment. This can be done based on what is known as a point cloud, which depicts the environment, including the object to be identified. Summary of the Invention

[0002] According to the present invention, a method for simplifying the depiction of point clouds of an environment, a computing unit and a computer program for performing the method, and a robot, having the features of the independent claims, are proposed. Advantageous design embodiments are those described in the dependent claims and hereinafter.

[0003] This invention generally relates to acquiring point clouds from an environment using sensors. Typical sensors that can be used to acquire point clouds depicting an environment are so-called LiDAR sensors, laser scanners, or stripe projection systems. Such point clouds are typically a collection of points in 3D, where these points typically correspond to reflective points at surfaces (i.e., especially objects) in the environment. However, in principle, environmental information acquired using other sensors (e.g., other depth sensors, or, if necessary, a 3D camera with depth information at individual image points (pixels)) can also be considered as point clouds. Similarly, point clouds created and computed, for example, using software algorithms from individual camera images and / or data acquired using camera sensors can also be considered.

[0004] The invention will be described in more detail below, primarily in conjunction with robots that can be used in, for example, production facilities, logistics, or other facilities. Types of such robots (often also referred to as so-called motion mechanisms) include, for example, Cartesian robots, Scara robots, and articulated arm robots. Such robots can be used to grasp and / or move objects (i.e., objects or parts). A typical application is retrieving objects from a box containing, for example, a large number (identical or different) of objects and placing them, for example, in a location or, if necessary, installing them. For this purpose, such robots have, for example, grippers or, generally, end effectors. Such robots can also be called manipulators or manipulation devices, and grasping (possibly accompanied by movement) can also be called manipulation.

[0005] In order for a robot to grasp and / or move objects in a working environment (such as in a box, or on or in other carriers such as containers or pallets) with the help of an end effector, the robot (especially its end effector) must perform a movement process to move the end effector to the location of the object, and then, after grasping the object, move the end effector away from the location of the object again, for example, to the desired placement location.

[0006] For successful grasping, the precise posture of the gripper—hereinafter referred to as the grasping posture—can be predetermined to enable successful grasping. Similarly, the movement process should be determined so that the end effector or other parts of the robot do not come into contact with other objects in the environment during the movement process. For this purpose, environmental information, such as point clouds of the environment (scene, which may also be referred to as the working environment in the robot's context), especially point clouds of the carriers (e.g., boxes) present there, can be acquired and provided. Sensors such as the aforementioned LiDAR sensors (laser scanners) or other depth sensors can be used for this purpose. The environmental information thus also includes, in particular, one or, if necessary, multiple objects located in the working environment.

[0007] Then, in principle, the grasping posture can be determined based on environmental information (i.e., point clouds). Based on this, the robot's movement sequence can be determined, for example, to determine control information for moving the robot, which can then be implemented by a control unit or other computing unit. However, in principle, the movement sequence can also be determined without a grasping posture.

[0008] Sensor data or environmental information is provided in this case as point clouds or datasets; this environmental information typically requires further processing to identify environmental surfaces and disturbance profiles. This information can be used to update data models for complex algorithms, such as calculating avoidance maneuvers, part localization, or obstacle avoidance. However, because point clouds are typically large in volume, processing them is computationally intensive and time-consuming.

[0009] However, as mentioned earlier, there are other application scenarios where the environment and its objects are collected and represented as point clouds, and these problems also arise. For example, in the field of 3D games, this includes terrain simplification and collision detection. In navigation, path planning, and autonomous driving, it may involve, for example, simplifying terrain or its representation. In particular, consider applications where the terrain, landforms, topography, topography, surface (or combinations thereof) is simplified after point cloud representation. This is also of interest for collision detection, as it can reduce the amount of data and simplify computation.

[0010] Against this backdrop, a simplification of point clouds depicting an environment is proposed within the scope of this invention. Simplification is understood herein, in particular, as reducing the amount of data, while retaining as much application-relevant content as possible.

[0011] It should be pointed out again that this method of simplifying point clouds can be applied not only to the specific field of robotics, but also to other fields that process such point clouds, such as autonomous (or automated or semi-automated) driving or other application areas mentioned above.

[0012] In this case, a dataset representing a point cloud based on a coordinate system is provided, which is specifically acquired from the environment using at least one sensor. Coordinates can be assigned to each point using this coordinate system. The Cartesian coordinate system is used exemplarily below; a point is thus defined by its x, y, and z coordinates (i.e., translation vectors relative to the origin). However, it should be understood that other coordinate systems can also be used.

[0013] This coordinate system defines a grid plane with grid cells. In the case of a Cartesian coordinate system, this grid plane can be, for example, an xy plane; the grid plane with grid cells can also be referred to here as a grid. The grid cell itself can then be, for example, a square with a side length of 1 in the x-direction and a side length of 1 in the y-direction (with corresponding scales if necessary). It should be understood that this is merely an example. Even in the case of a Cartesian coordinate system, the shape of the grid cell is not necessarily a square; for example, it can be a rectangle. In other coordinate systems, such a grid cell can also be, for example, a parallelogram or other shapes.

[0014] Then, one or more 3D shapes are determined based on this dataset, each 3D shape having a base and a height. The base here corresponds to one or more coherent mesh cells in the mesh plane. In the example above, the base is therefore, for example, a square or a rectangle. In the case of multiple coherent mesh cells, the base may not be a square or a rectangle, although a regular base is preferred, as will be explained in detail later.

[0015] Height is determined based on points in the point cloud, which correspond to one or more coherent grid cells through projection, i.e., the projection onto the grid plane lies within the relevant one or more grid cells. Each point is assigned a height relative to the grid plane using a coordinate system, which, in the Cartesian coordinate system described above, is, for example, the value of the z-coordinate. The specific value of the height can then be, for example, the highest or average value (e.g., the arithmetic mean) of the relevant points.

[0016] In a coordinate system, the base and height allow for the definition of a distinct 3D shape. Thus, when using a Cartesian coordinate system, these 3D shapes can be, in particular, cuboids. However, in the case of non-Cartesian coordinate axes, they can also be, for example, parallelepipeds.

[0017] Unlike the large number of individual points (from a point cloud), the amount of data used to represent these 3D shapes is significantly smaller because a certain number of points are combined into a single 3D shape, such as a cuboid. This simplified representation is sufficient, especially for recognizing objects in an environment, such as representing the outline of an object, while requiring significantly less computational power and time during processing.

[0018] But this approach can greatly simplify collision calculations; thus, for example, it can determine movement flows for a device (such as the robot mentioned above), using a simplified dataset to avoid collisions between the device and objects. This is because, in this case, a particularly detailed representation of the environment is not important or of little importance.

[0019] The proposed method is considered here, particularly for simplifying surface topology, without considering undercut volume. As previously mentioned, the point cloud is preferably simplified using cuboids to represent the surface. This cuboid representation of objects is advantageous, especially for calculating collisions between objects, as the computation time is significantly shorter than that required for more complex objects (e.g., 3D meshes). Therefore, this simplification can significantly reduce the processing time for collision detection.

[0020] The simplified dataset representing one or more 3D shapes in this way is then provided for further use. This can be used, for example, to determine the movement process for a robot in order to grasp and / or move objects in the environment using the robot's end effector.

[0021] In one embodiment, one or more 3D shapes are determined by determining multiple individual 3D shapes based on the dataset. For each individual 3D shape, a base and a height are determined, where the base corresponds to a grid cell in a grid plane, and the height is determined based on a point in the point cloud that corresponds to that grid cell through projection. In other words, the corresponding 3D shape (also called an individual 3D shape), such as a cuboid, is first determined solely based on each individual grid cell.

[0022] Then, one or more of the multiple 3D shapes are determined based on at least two coherent 3D shapes within the individual 3D shapes. This further reduces the amount of data. However, at least one 3D shape within an individual 3D shape can also be used as one or more of the multiple 3D shapes. This is the case, for example, when a particular individual 3D shape cannot be merged with at least one other 3D shape.

[0023] One or more of the multiple 3D shapes are determined based on at least two coherent individual 3D shapes, which can be done in particular as follows: For at least a portion (i.e., some) of the individual 3D shapes, it is checked whether two or more individual 3D shapes with consistent heights within a preset tolerance are coherent. The tolerance can be selected as needed, for example, as an absolute or relative value. If so, one or more of the multiple 3D shapes are determined based on the coherent individual 3D shapes. The height of the 3D shape generated from two or more coherent individual 3D shapes can be, for example, the average height of the individual 3D shapes, or the highest height of the individual 3D shapes.

[0024] In this way, it is possible to check, for example, whether each individual 3D shape can be merged with one or more other coherent individual 3D shapes. However, this is only appropriate if the (new) 3D shape corresponds to a 3D shape of a preset type, especially the same type as the individual 3D shapes, before merging multiple coherent individual 3D shapes into one 3D shape. If the individual 3D shapes are cuboids, then it can be proposed that the (new or merged) 3D shape must also be a cuboid. This is because such 3D shapes can be represented with very little data (e.g., two points on the base and the height), while more complex shapes (e.g., L-shapes, etc.) require more data to represent.

[0025] The computing unit (i.e., a system for data processing in general) according to the invention, such as the control unit of a control device or robot, or a central server or other computing system, is configured, in particular, by programming techniques, to perform the method according to the invention.

[0026] The present invention also relates to a robot configured to obtain control information as determined as described above. Alternatively or additionally, the robot has a computing unit according to the invention. Furthermore, the robot particularly has a control or regulation unit and a drive unit for moving the robot. Additionally, the robot may have at least one sensor for acquiring environmental information, such as a camera and / or a lidar sensor.

[0027] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product containing program code for performing all method steps, as this results in particularly low cost, especially when the control device used for execution is also used for other tasks and therefore exists anyway. Finally, a machine-readable storage medium is provided on which the computer program as described above is stored. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memory, EEPROM, DVDs, etc. Downloading the program via a computer network (Internet, intranet, etc.) is also possible. Such downloading can be performed wired or wirelessly (e.g., via WLAN networks, 3G connections, 4G connections, 5G connections, or 6G connections, etc.). Attached Figure Description

[0028] Other advantages and design solutions of the present invention can be derived from the specification and drawings.

[0029] The present invention is schematically illustrated in the accompanying drawings with reference to embodiments, and will be described below with reference to the drawings.

[0030] Figure 1 A robot used to illustrate the invention is shown schematically.

[0031] Figure 2 A container used to illustrate the invention is shown schematically.

[0032] Figure 3 The flowchart of one implementation of the method is illustrated schematically.

[0033] Figure 4a and 4b The details used to illustrate the invention are shown schematically. Detailed Implementation

[0034] exist Figure 1 The diagram schematically and exemplaryly illustrates a robot 100 used to illustrate the present invention. The robot 100 exemplarily has a base and a plurality of arm assemblies 102, 104, 106, i.e., axes, which are movably and removably connected by means of joints 112, 114, respectively.

[0035] In addition, the robot 100 has an end effector 108, such as a gripper. The end effector 108 is movably and movablely connected to the arm assembly 106 via a joint 116.

[0036] Furthermore, the robot 100 has a drive system 120, shown only schematically herein, and a computing unit 122 designed as, for example, a control or adjustment unit. Thus, the drive system 120 can be manipulated, for example, according to control information, to make the robot move according to a desired movement sequence. This can include, for example, moving these axes relative to each other by means of joints, but also includes rotation of these axes themselves (if a corresponding drive mechanism is present).

[0037] It should be noted that robot 100 is used here for illustrative purposes only. Robots used for grasping and / or moving objects in containers may also be constructed differently, for example, by means of end effectors that can only move linearly along multiple different guide rails.

[0038] Furthermore, a box 132 is shown in the work environment 130, in which an object 140 is exemplarily present. The robot 100 can now be manipulated, for example, to grasp and / or move the object 140 by means of an end effector 108, especially to remove it from the box and place it, for example, elsewhere.

[0039] Furthermore, sensor 124 is shown as an example, which can be, for example, a camera or a lidar sensor. Both types of sensors can also be used. Similarly, multiple sensors of the same type can be used. As previously mentioned, using a lidar sensor, other time-of-flight (TOF) sensors, or a 3D stripe projection scanner to acquire point clouds as environmental information is particularly suitable. However, it is also conceivable to acquire 3D and / or 2D images using a camera. It is also conceivable to acquire only 2D images using a camera. Sensor 124 can be arranged, for example, in a suitable manner in the working environment, such as on the ceiling, thus being separate from robot 100. However, sensor 124 can also be, for example, part of the robot and arranged, for example, on arm assembly 106 or end effector 108.

[0040] With the help of sensor 124, the working environment 130, and especially the box 132 and its interior, including object 140, can now be collected. Based on the environmental information or environmental sensor data thus obtained, a movement process can be created for the robot to grasp and / or move object 140 with the help of end effector 108.

[0041] Therefore, in Figure 2 The text shows the relationship with... Figure 1Box 132 is similar to box 232, and different objects 240, 241, 242, 243, and 244 are exemplarily contained within the boxes. A typical task of the robot is to remove as many objects as possible from the boxes. Here, it is appropriate to remove the easiest or safest objects to grasp first, and so on, but certain rules must be followed, such as the end effector not obstructing prohibited areas. It can also be seen that these objects can be different from each other, for example, they can be cuboid or cylindrical, and they can also be unordered.

[0042] An exemplary movement flow for grasping object 240 and removing it from a box is shown. For this purpose, an approach path or first part of the movement flow 251 is shown, by which the end effector is moved to object 240 so that it can grasp the object. Furthermore, a return path or second part of the movement flow 252 is shown, along which the end effector carrying object 240 can be removed from the box.

[0043] The first part of the movement process 251 and the second part of the movement process 252 together form a complete movement process. It can be seen here that this movement process puts the end effector in a specific posture—a grasping posture—in order to properly grasp the object 240.

[0044] To find, for example, the optimal grasping pose and collision-free movement, it is appropriate to acquire environmental data (i.e., boxes containing objects) using sensors. The point cloud acquired in this process, such as using a 270-degree label, can then be simplified to find the grasping pose as quickly and computationally as possible. This simplification of the point cloud will be explained in more detail below.

[0045] exist Figure 3 The diagram schematically illustrates the flow of a method in one implementation. Reference is also made here. Figure 4a and Figure 4b The diagram shows the container or box and the object, and will be used to explain the different steps in more detail.

[0046] In step 300, a dataset 302 is provided, which represents a point cloud acquired from the environment using at least one sensor, based on a coordinate system. A grid plane with grid cells is defined through this coordinate system. Such a point cloud... Figure 2 The image 270 is shown for a box 232 containing an object located therein.

[0047] To further elaborate, Figure 4aThe diagram now shows a coordinate system 400 with an origin 402, or more precisely, an exemplary Cartesian coordinate system with x, y, and z axes. The grid plane defined by coordinate system 400 is denoted by 404 and corresponds here to the xy plane. Grid plane 404 has grid cells, one of which is denoted by 410. These grid cells exemplarily have a length 412 in the x-direction and a length (or width) 414 in the y-direction. These can, in this case, be, for example, unit lengths.

[0048] In addition, Figure 4a The diagram shows a set of points 420, which may be part of a point cloud acquired using a lidar sensor. This coordinate system assigns x, y, and z coordinates to each point; the corresponding vector for each point is exemplarily represented by 422.

[0049] Then, in step 310, one or more 3D shapes, such as cuboids, are determined based on the dataset. This specifically includes, first and foremost, determining multiple individual 3D shapes, also cuboids, based on the dataset in step 320. For each individual cuboid, the base and height are determined, wherein the base corresponds to a grid cell in the grid plane, and the height is determined based on a point in the point cloud that corresponds to that grid cell through projection.

[0050] This is Figure 4a The cuboid 430 is illustrated exemplarily. The base 418 of the cuboid corresponds here to a grid cell in the grid plane, for example, the bottom left corner grid cell. This grid cell, or a corner point of the base, is denoted by 432. The length and width of the base correspond to the length 412 and width 414 of the grid cell, respectively. The height of the cuboid 430 (denoted here by 416) is derived from the points 420 shown, which are the portions of the point cloud (not shown here) that correspond to the grid cells used as the base by projection (to the grid plane or xy plane). A specific value for the height 416 may, for example, correspond to the arithmetic mean of the z-values ​​of these points or the highest z-value of these points.

[0051] exist Figure 4a The diagram illustrates other cuboids that can be determined in the same manner. In this case, a cuboid can be determined, for example, for each grid cell containing at least one point in the point cloud (at projection). However, it is also conceivable that this is done only for grid cells containing at least two or at least five points (at projection), or generally a predetermined minimum number of points.

[0052] In step 330, then for at least a portion of an individual 3D shape or cuboid, it is checked whether two or more individual 3D shapes or cuboids with consistent heights within a predetermined tolerance are continuous.

[0053] This is Figure 4b This was explained in the text, which again shows... Figure 4a The coordinate system 400 in the diagram, along with other cuboids besides the individual cuboid 430, also shows some other cuboids (these other cuboids do not correspond to the coordinate system 400 in the diagram). Figure 4a (a cuboid in the middle).

[0054] Therefore, for example, based on the individual cuboid 430 or its corresponding mesh cell, it can be checked whether the height of adjacent individual cuboids is within the tolerance range. For the individual cuboid 430, the height is again shown as 416, and as a tolerance, a minimum height of 416.1 and a maximum height of 416.2 are shown. Depending on the scale and application scenario, different values ​​or ranges can be considered here as tolerances, for example. In the field of autonomous driving, this might be in the range of (possibly double-digit) centimeters, while for handling technologies (such as the described object grasping), this might be in the range of (possibly sub) millimeters.

[0055] Individual cuboid 430 is also exemplarily represented here as 1,1. Based on this, other individual cuboids are represented along the y-axis as 1,2,1,3,1,4, and along the x-axis as 2,1,3,1. Other individual cuboids shown are represented as 2,2,3,2,3,2,3,3. The adjacent individual cuboids of individual cuboid 430 or 1,1 are first individual cuboids 1,3,2,2, and 2,1. The height is exemplarily within a preset tolerance. Therefore, individual cuboids 1,1,1,2,2,1,2,2 can be merged into a (new) cuboid 432 (illustrated only here).

[0056] The heights of individual cuboids 1 and 3 are still within the tolerance range, while the heights of individual cuboids 1, 4, 2, 3, 3, 1, 3, 2, and 3, 3 are not. Therefore, cuboid 432 will not be further expanded, because simply adding individual cuboids 1 and 3 will no longer form a cuboid. However, it is conceivable, for example, to merge individual cuboids 2, 3, and 3, 3 into a (new) cuboid.

[0057] Alternatively, one could imagine merging individual cuboids 1,1,1,2 and 1,3 into a (new) cuboid, and similarly merging individual cuboids 2,1 and 2,2 into a (new) cuboid.

[0058] To determine which individual cuboids are merged, one can examine, for example, every possibility of forming a (new) cuboid based on an individual cuboid (here, cuboid 430 or 1,1). A variant that merges the majority of individual cuboids can then be selected. Alternatively, one could envision selecting the variant that forms the largest coherent base (because different results could occur if the mesh cells are not all the same size).

[0059] This can continue until each individual cuboid has been checked to see if it can be merged with at least one other individual cuboid.

[0060] In step 332, one or more of the plurality of 3D shapes or cuboids are then determined based on the coherent individual 3D shapes or cuboids.

[0061] Then, an individual 3D shape or cuboid that is not merged with at least one other 3D shape or cuboid can be directly used as a 3D shape or cuboid, i.e., step 322.

[0062] In step 340, a simplified dataset 342 representing one or more 3D shapes is then provided for further use. In step 350, a movement sequence can then be determined, for example, for the robot, as referenced. Figure 2 As described above, this enables the robot to be moved accordingly based on the control information determined in step 360.

Claims

1. A method for simplifying point cloud representations of an environment, comprising: Provide (300) a dataset (302) representing a point cloud based on a coordinate system (400), the point cloud being acquired from the environment (130) in particular by means of at least one sensor (124), wherein a grid plane (404) with grid cells (410) is defined by the coordinate system (400). Based on the dataset, one or more 3D shapes (430, 432) are determined (310), wherein each 3D shape has a base (418) and a height (416), wherein the base corresponds to one or more coherent grid cells of a grid plane, and wherein the height is determined based on points in the point cloud that correspond to the one or more coherent grid cells by projection. Provide (340) a simplified dataset representing the one or more 3D shapes for further use.

2. The method according to claim 1, wherein, Based on the dataset, one or more 3D shapes are determined, including: Based on the dataset, multiple individual 3D shapes (430) are determined (320), wherein a base (418) and a height (416) are determined for each individual 3D shape, wherein the base corresponds to a grid cell of the grid plane, and wherein the height is determined based on points in the point cloud that correspond to the grid cells by projection; and One or at least one of the plurality of 3D shapes is determined based on at least two coherent individual 3D shapes, and / or, at least one of the individual 3D shapes is used as one or at least one of the plurality of 3D shapes.

3. The method according to claim 2, wherein, Determining one or more of the plurality of 3D shapes based on at least two coherent individual 3D shapes includes: For at least a portion of the individual 3D shapes, check whether two or more individual 3D shapes with heights consistent within a predetermined tolerance are continuous, and If coherent, then one or at least one of the plurality of 3D shapes is determined based on the coherent individual 3D shapes.

4. The method according to claim 3, wherein, The multiple 3D shapes are of the same type.

5. The method according to any one of the preceding claims, wherein, One or each of the plurality of 3D shapes corresponds to a parallelepiped, particularly a cuboid.

6. The method according to any one of the preceding claims, for determining a movement flow of a device, wherein the simplified dataset is used to avoid collisions between the device and objects.

7. The method of claim 6, for determining a movement sequence of a robot (100) as a device to move the robot's end effector to an object in the environment and to grasp and / or move the object by means of the end effector (108), comprising: The movement process is determined (350) based on the simplified dataset; as well as Provide the aforementioned mobile process.

8. The method according to claim 7, further comprising: Based on the aforementioned movement process, control information (360) for moving the robot is determined, and Provide the control information and / or move the robot based on the control information.

9. A computing unit (122) comprising means for performing the method according to any one of the preceding claims.

10. A robot (100) configured to acquire control information determined by means of the method according to claim 8, and / or having a computing unit according to claim 9, and It has a drive system and a control or adjustment unit for operating the drive system. It has an end effector with at least two gripping devices for gripping and / or moving objects, and Preferably, it has at least one sensor for collecting environmental information about the environment, especially a camera.

11. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon according to claim 11.

Images