Method for determining point group visible or invisible from predefined observation point

Abstract

The invention relates to a method for determining, from a set of point groups, a target set of point groups in an environment (150) which are visible or invisible from a predefined observation point (214) in the environment (150), the point groups comprising one or more points (P, A, B), comprising: providing the set of point groups in the environment having coordinates in an original coordinate system (200), transforming coordinates of the set of point groups into spherical coordinates in a target coordinate system, wherein an observation point (214) in the environment is located at an origin of the target coordinate system; determining a spherically curved grid in spherical coordinates in the target coordinate system, wherein the grid comprises grid points and / or cells; assigning each point group of at least a portion of the set to a cell or a grid point, for each of at least a portion of the cells of the grid or for each of at least a portion of the grid points of the grid assigned with more than one point group: determining a minimum spacing and / or a maximum spacing, a) a target set of point groups visible from a predefined observation point (214) in the environment is determined, and / or b) a target set of point groups invisible from a predefined observation point in the environment is determined, and the target set or the target set of point groups is provided.

Description

Technical Field

[0001] The invention relates to a method for determining a target set of a point group visible or invisible from a predetermined observation point, a system for data processing and a computer program for executing the method, and a device for using such a target set of a point group. Background Art

[0002] Devices such as at least partially autonomous vehicles or robots move in an environment such as a home, a garden, a factory hall or a street, in the air or in water, for example along a movement path. For this purpose, it is necessary that the current position and / or orientation (also called pose) of the mobile device is known and can also be repeatedly determined. For this purpose, for example, laser distance measurements (for example using so-called lidar sensors) can be used. Likewise, the acquisition of environmental information such as images of the environment (for example with the aid of a camera) can be used. Summary of the invention

[0003] According to the invention, a method for determining a target set for a point group, a system for data processing and a computer program for executing the method, and a device for using such a target set for a point group are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The present invention discusses a collection of points in an environment or a so-called point cloud. Such points or point clouds are acquired, for example, by means of a lidar sensor or other types of distance measurements, and then respectively describe the reflection points of, for example, a laser beam in the environment emitted by the lidar sensor. If these points are observed in a coordinate system with the lidar sensor at its origin, then, for example, the spacing or distance to objects in the environment can be described based on these points or their coordinates. For example, in the case of a device, in particular a mobile device such as an opening volume, an orientation in the environment and, based on this, (automatic) navigation and, in particular, movement of the device in the environment can be achieved.

[0005] Another possibility in the case of such a device is to detect environmental information, such as images, for example, by means of a camera. Such images can also be used for orientation in the environment and, based on the (automatic) movement of the device in the environment, the images can also be semantically analyzed if necessary in order to recognize objects based on this.

[0006] If two types of environmental detection are now present, namely the mentioned points or point clouds on the one hand and environmental information such as images on the other hand, the points can be associated with the environmental information, i.e., for example, the distances of the points can be mapped onto the image. For example, it can be assumed that for an application, a text description predicted by an image classification algorithm in such an image should be mapped onto a LiDAR point cloud of the same scene in the environment in order to extend the semantic information with 3D information.

[0007] In practice, however, for example in the case of a device with a lidar sensor and a camera, the point cloud on the one hand and the environmental information on the other hand are detected or recorded from different perspectives or from different observation points. Due to the different perspectives of the lidar and the camera, visibility problems arise. In particular, at the edges of objects in the environment, it can happen that the lidar sensor sees or detects, for example, two separate points in the environment that are farther away than the object, but the camera only detects one of these points. The other point is hidden by the object.

[0008] Therefore, if the information of a camera is to be mapped onto a LiDAR point cloud, or vice versa, without analyzing the visibility of the points from the camera's point of view (camera foothold), points not detected by the camera will be incorrectly marked, for example with a color or a text description on the side of the object. In general, therefore, this actually invisible point - and therefore its distance greater than the distance of the object - will be mapped onto the object or its surface.

[0009] The question of which points of a point cloud are visible from a particular observation point can be solved, for example, by S. Katz et al., "Hidden Point Removal", "ACM Transactions on Graphics", Volume 26, Issue 3, July 2007, pp 24. This approach was also successfully applied to the task of camera lidar mapping by Vechersky et al., "Colourising Point Clouds Using Independent Cameras," in IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3575-3582, Oct. 2018.

[0010] The scheme of Katz et al. describes an operator for "inverting" all points of a given point cloud. This is also called "spherical flipping" because all points are reflected on a sphere containing all points (but the transformation function can also be different). Subsequently, the scheme constructs a convex envelope from the inverted point cloud and the coordinate center of the camera. There it is shown that every point lying on this convex envelope is visible to the camera, all other points are invisible and can be removed.

[0011] Other approaches for solving the mentioned problems are based, for example, on surface reconstruction and then using so-called ray casting methods or the like.

[0012] In the context of the invention, an action is now proposed for determining a target set of point groups that are visible or invisible from a predetermined observation point in the environment, that is, from a (existing) set of point groups. In this case, for example, a point group can include only one (single) point or also a plurality of points. As will be explained in more detail later, the point groups of a set can differ from one another. In other words, the points of a set can be grouped.

[0013] Although the proposed actions are described below in particular with regard to the mentioned examples using lidar sensors and cameras which detect (or see) the environment from different observation points or observation angles, it is thus possible to determine from an arbitrary set of points or observation points the set of targets that are visible (or even invisible) from a predetermined observation point in the environment.

[0014] For this purpose, a set of point groups in the environment is first provided, that is, a set of point groups with coordinates in an original coordinate system. In this case, the points of the point group or the point group can exist, for example, in Cartesian or spherical coordinates. For example, the set of point groups in the environment can be or have been determined by means of distance measurement, in particular by means of a lidar sensor. The starting point of the distance measurement, in particular the position of the lidar sensor, is then located, in particular, at the origin of the original coordinate system.

[0015] The coordinates of the set of point groups are then transformed into spherical coordinates in a target coordinate system, where the observation point in the environment (from which the point to be determined should be visible, ie for example the location of a camera) is located at the origin of the target coordinate system.

[0016] In addition, a spherical curved grid is determined in the spherical coordinates in the target coordinate system, wherein the grid includes grid points and / or cells. The grid points and / or cells are arranged uniformly here. A simple example for this would be a grid on a spherical surface with the center of the sphere belonging to the origin of the target coordinate system, wherein the grid lines are evenly spaced from each other in the polar direction and the azimuth direction. In this way, a cell is formed by four adjacent grid points (where two grid lines intersect), with these four grid points as angles. The basic idea here is how a typical laser radar sensor detects points, i.e., in the polar direction and in the azimuth direction with uniform spacing. If such a grid is created for the laser radar sensor in the original coordinate system, then - according to the spacing of the grid lines - it will be possible to assign exactly one point or point group to each cell. Such a uniform arrangement can be used universally, i.e., for different environments. However, an uneven arrangement of grid points and / or cells is also conceivable. For example, cells or grid points in the area of ​​the floor and / or ceiling (in an environment, such as in a hall) may be arranged finer or more densely than in the rest, where typically no shading should be expected.

[0017] In the case of a grid in the target coordinate system, each point group of at least part of the set (which point groups may also each include only one point, for example) is now assigned to a cell or grid point. Since the origin of the target coordinate system (the desired observation point and therefore the center of the spherical curved grid) does not coincide with the origin of the original coordinate system (the lidar sensor with which the points of the set are detected may be located, for example), it may happen that multiple point groups may be assigned to one cell. Here, the points are assigned to the cells according to spherical coordinates.

[0018] In a graphical manner, a connecting line between the origin of the original coordinate system and a specific point will intersect exactly one of the cells; these points can then be assigned to this cell. In the case of a plurality of points for each point group, an average value of the coordinates of the points can also be used. If a plurality of points for each point group are detected simultaneously or very close in time by the lidar sensor, these are taken into account, for example. Since the cells are defined or at least can be defined by the grid points, point groups can also be assigned to the grid points instead of the cells, for example those point groups which are closest to the intersection of the imaginary connecting line on the surface of the sphere with the grid.

[0019] In this context, it is then possible to determine a cell or a grid point to which more than one point group is assigned. For each of at least a portion of these cells of the grid or for each of at least a portion of these point groups, a minimum or maximum spacing from the origin of the target coordinate system is then determined (for which there is then also a corresponding point group). For this purpose, for example, a spacing from the origin of the target coordinate system can be determined for each point group for each cell or grid point. The spacing of a point from the origin corresponds here to a radial coordinate in spherical coordinates. If the point group comprises a plurality of points, the average spacing of the plurality of points or the minimum or maximum spacing occurring at the plurality of points of the point group can be used.

[0020] Based on this, a target set of point groups visible from a predetermined observation point in the environment can then be determined, i.e., the target set includes those point groups whose spacing from the origin of the target coordinate system for each cell or grid point corresponds to a minimum spacing or exceeds the minimum spacing by at most a predetermined first threshold value and / or whose spacing from the origin of the target coordinate system for each cell or grid point is below a maximum distance minus a predetermined second threshold value. In addition, point groups assigned to cells or grid points to which only one point group is assigned can also be included.

[0021] To achieve this, one or more point groups can be identified, for example, for each cell or each grid point, whose spacing from the origin of the target coordinate system is greater than a minimum spacing or exceeds the minimum spacing by more than a predetermined first threshold, and / or whose spacing from the origin of the target coordinate system exceeds a maximum spacing minus a predetermined second threshold. These point groups can then be removed from the set, and the remaining point groups of the set can then be provided as a target set.

[0022] The idea in this case is that from the desired observation point - if there would be a lidar sensor there - only one point or one point group would be located in a cell. If two or more points or point groups are assigned to a cell with very different spacings, then not all of these points or point groups would be detected by the lidar sensor. The point with the shortest spacing would be generated by a nearby object, and the point with a significantly larger spacing would be behind the object and not be detected at all.

[0023] If now all points or point groups whose distance from the origin of the target coordinate system is greater than a minimum distance are removed from the set, only points or point groups that are also visible or will be visible from the desired observation point remain in the set. It can be provided that only points or point groups whose distance from the origin of the target coordinate system exceeds the minimum distance by more than a first predetermined threshold value are removed. This first threshold value can depend on the minimum distance, in particular corresponds to the corresponding minimum distance multiplied by a corresponding factor. Thus, the following fact can be taken into account: points that have only a slightly larger distance than the minimum distance will not actually be located behind the object, but will still be located on the surface of the object, but the surface is, for example, slightly tilted in the environment, so that the distance there is slightly larger. By selecting the first threshold value to be zero, it is also possible in this way to adapt to (only) the minimum distance.

[0024] Such points or point groups can also be identified by searching for points or point groups whose distance from the origin of the target coordinate system exceeds the maximum distance minus a predefined second threshold value. In the case of a suitable choice of the second threshold value, points or point groups are thus also identified that should not actually be visible from the observation point. The second threshold value can depend on the maximum distance, in particular, corresponds to the corresponding maximum distance multiplied by a corresponding factor.

[0025] However, the target set can also be obtained in the following way, that is, those point groups whose distance from the origin of the target coordinate system for each cell or grid point exceeds the minimum distance at most a predefined first threshold value (or wherein the threshold value 0 corresponds to the minimum distance) and / or whose distance from the origin of the target coordinate system for each cell or grid point is lower than the maximum distance minus the predefined second threshold value are directly output as the target set. Point groups assigned to cells or grid points that are assigned only one point group can also be included in the target set, because there the unique distance corresponds to the minimum distance at the same time.

[0026] However, these identified points or point groups therefore also represent those points or point groups of the set that are not visible from a predefined observation point in the environment. If, for example, these points or point groups are exactly desired for a specific application, they can be determined as a target set.

[0027] The proposed action is much faster than, for example, the possibilities mentioned at the beginning, since it requires neither surface reconstruction nor convex envelopes. Both tasks can be calculated in a time-consuming manner, especially in the case of larger point clouds. The above-mentioned approach for estimating the visibility of a point cloud has a time complexity of the order of O(n*log n), and typical approaches for surface reconstruction have a time complexity of the order of O(n^2). The action proposed within the scope of the present invention, however, has only a time complexity of the order of O(n) and still provides suitable results, for example, for applications with a lidar sensor and a camera. In practice, this results in a time saving of an order of magnitude.

[0028] Furthermore, the proposed action can be implemented simply and quickly and is therefore suitable as a quick solution to the described problem. It has also been shown that the solution of Katz et al. filters out many points that are actually visible from the observation point at a distance and on the ground. The proposed action retains these points. Even if slightly more false alarms should occur here, this is negligible in practice.

[0029] Preferably, determining the grid in spherical coordinates in the target coordinate system includes determining in spherical coordinates or, if necessary, using the coordinates of a set of point groups in the original coordinate system in spherical coordinates (if the points or point groups should already exist). The frequency distribution of the point groups in the polar direction can then be determined. Based on the number of local maxima of the frequency distribution, the size of the cell (as an angle) or the spacing of the grid points (as an angle) in the polar direction can then be determined. This number itself results in the size of the cell corresponding to the basic lidar sensor, but it is also possible to select, for example, double or other suitable values. The size of the cell (as an angle) or the spacing of the grid points (as an angle) in the azimuthal direction can be determined for each local maximum based on the number of point groups. Here, the number itself also results in the size of the cell corresponding to the basic lidar sensor, but it is also possible to select, for example, double or other suitable values. In this way, the grid itself can therefore also be automatically determined.

[0030] Advantageously, the size of the cells or the spacing of the grid points in the polar direction and / or the size of the cells or the spacing of the grid points in the azimuthal direction are determined based on the distance of a predetermined observation point from the origin of the original coordinate system. For example, the further the sensors (lidar sensor and camera) are from one another, the larger the cells can be selected.

[0031] Preferably, environmental information, such as image information and / or semantic information, is also provided with coordinates in the target coordinate system. At least part of the point group in the target set of point groups visible from a pre-given observation point in the environment can then be associated with the environmental information. In other words, points with distance information, for example, can thus be mapped onto the image.

[0032] This then also allows, for example, control information for the mobile device to be determined based on the environment information and the point group of the target set associated therewith. The control information can then be provided (for the device) so that the device can be controlled, ie, for example, moved in the environment, in particular based on the control information.

[0033] In particular, the proposed action can be used whenever the visibility of points in a point cloud must be derived from a specific viewing angle. This applies both to artificially generated point clouds (e.g. from CAD models) and to recorded real data (e.g. point clouds from LiDAR sensors, which should be transformed to different viewing angles, as already explained in detail above).

[0034] The proposed actions can be applied in combination with (if necessary different) distance sensors (e.g. lidar or radar) and (if necessary different) image-based sensors (e.g. cameras or thermal imaging devices). The proposed actions can be used in such systems to improve the data quality by coordinating the two data sources with one another. Many other algorithms can thus benefit from the improved data quality and quantity (since the data now contain not only image / semantic or 3D information, but both).

[0035] The proposed actions can be used in general for transforming the footing of a point cloud source "correctly", i.e. with visibility constraints and not just with distortion of the reference frame. The proposed actions can also be used to fuse multiple point cloud sources (LiDAR, Radar, Stereo Vision, Structured Light) into a point cloud from arbitrary (virtual) viewpoints.

[0036] For example, the proposed actions can also be used for simulating virtual (lidar) sensors, for example in a synthetic environment or on real data. A specific application of the proposed actions is the recognition of boxes and other objects in a warehouse environment, for example, when using lidar and camera sensors. For this purpose, the assignment of the camera's RGB data and the semantic labels of these camera images to the lidar point cloud can be improved using the proposed actions. The point cloud can then be accumulated and objects therein recognized. The proposed actions can also be used in the scope of autonomous driving, for example, when the information of a lidar sensor or a radar sensor and one or more cameras is or should be combined.

[0037] The system for data processing according to the invention comprises means for carrying out the method according to the invention or its method steps. The system can be, for example, a computer or a server in a so-called cloud or cloud environment. Here, then - in the case of an application for a (mobile) device - a set of point groups can be transmitted (for example via a wireless data connection) and then the target set can be transmitted back to the device again. It is also conceivable that such a system for data processing is a computer or a control device in such a (mobile) device.

[0038] The invention further relates to a device which is designed to provide a set of point groups in an environment with coordinates in an original coordinate system and preferably to provide environmental information with coordinates in a target coordinate system. The device is further designed to obtain a target set of point groups which are visible or invisible from a predetermined observation point in the environment and which have been determined from the set of points according to the method according to the invention. Preferably, the device has a distance measurement or lidar sensor for detecting point groups in the environment, more preferably also a sensor for detecting environmental information, in particular an image sensor, and further preferably also a control and / or regulation unit and a drive unit for moving the device. As mentioned, the system for performing data processing may also be included by the device.

[0039] The device is preferably configured as an at least partially automatically moving vehicle, in particular a passenger vehicle or freight vehicle, and / or as a robot, in particular a household robot, such as a suction and / or wiping robot, a floor or street cleaning device or a lawnmowing robot, and / or as a drone.

[0040] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with instructions or program codes for executing the method or method steps, because this causes particularly low costs, especially if the execution control device is still used for other tasks and therefore originally exists. Finally, a machine-readable storage medium is provided, with a computer program as described above stored thereon. Suitable storage media or data carriers for providing computer programs are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download programs via computer networks (Internet, intranets, etc.). Such downloading can be done wired or cabled or wirelessly (e.g., via WLAN networks, 3G, 4G, 5G or 6G connections, etc.).

[0041] Further advantages and embodiments of the invention are apparent from the description and the accompanying drawings.

[0042] The invention is schematically illustrated in the drawings on the basis of exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The device is schematically shown in an environment for explaining the invention.

[0044] Figure 2 Shown in other details Figure 1 environment.

[0045] Figure 3 A grid is shown for explaining the present invention.

[0046] Figure 4 Different spacings of the points are shown.

[0047] Figure 5 In a preferred embodiment, the sequence of the method according to the invention is shown.

[0048] Figure 6 In a preferred embodiment, a diagram is shown before and after the use of the method according to the invention. DETAILED DESCRIPTION

[0049] exist Figure 1 Schematically shows a device 100 in an environment 150, based on which the present invention is explained. The device 100 is, for example, a robot or an automatically (or autonomously) driven vehicle, having a drive unit 102, a control and / or regulating unit 106 and a communication and computing unit 106, via which a wireless data connection to a system 130 for data processing, such as a server, is possible. The device 100 also has a lidar sensor 110 for laser distance measurement and a camera 120 for capturing images of the environment 150. In a preferred embodiment, the device 100 can be, in particular, a device according to the present invention.

[0050] Environment 150 is exemplarily the interior of a building, for example a corridor with side walls. Objects 152 or obstacles are exemplarily shown at the side walls. For example, it can be provided that information (distance and image information) of environment 150 should be detected by means of lidar sensor 110 and camera 120, and then based on this information, device 100 should be moved in environment 150. For example, it can be provided that device 100 should be driven along a corridor, but here should be driven around object 152.

[0051] For this purpose, the environment 150 can be scanned by means of the lidar sensor 110. In this case, the laser scanner in the lidar sensor is moved and a large number of points P or a collection of points P are obtained by means of the emitted laser beam and the received reflected radiation; each point represents a point or a small area in the environment, for example at an object, from which the reflected radiation is received.

[0052] exist Figure 2 Re-shown in Figure 1 Only the lidar sensor 110 and the camera 120 of the device are shown. In particular, it can be seen here that the lidar sensor 110 and the camera 120 are separated from each other in space and therefore also have different viewpoints or view angles for the environment 150. Although the situation is shown here only in two dimensions, the lidar sensor 110 and the camera 120 can also be spaced apart from each other in three dimensions, but this does not necessarily have to be the case. In addition, from Figure 1Two points A and B are exemplarily selected and shown from the set of points P.

[0053] according to Figure 2 As can be seen, the lidar sensor 110 can see or detect point A as well as point B. The camera 120, however, can only see or detect point A due to its different viewing angle or point of view 214. Therefore, if the camera's information is mapped to the lidar point cloud (a collection of points P, which also includes A and B) or vice versa, without analyzing the visibility of the points from the viewpoint of the camera 120 (camera foothold), then point B will likely be incorrectly labeled with, for example, a color or text description of the upper portion of the object 152 or generally associated with the upper side or surface of the object 152.

[0054] Point P (also A and B) can exist or be shown with coordinates in the coordinate system 200 of the lidar sensor 110 at the origin 202 of the coordinate system. Exemplarily, the coordinate system 200 is shown as a Cartesian coordinate system with axes x and y. The axis z would be perpendicular to the drawing plane here and cannot be seen. In this way, everything can be precisely defined. If the coordinate system 200 would be a coordinate system with spherical coordinates, then, for example, the distance of a point from the origin 202 corresponds to the value of the radial coordinate.

[0055] A coordinate system 210 may also be defined for the camera 120, wherein the camera 120 is located at an origin 212 of the coordinate system 210. By way of example, the coordinate system 210 is shown as a Cartesian coordinate system with axes x' and y'. The axis z' would be perpendicular to the drawing plane here and cannot be seen.

[0056] Now, the actions described below as an example can be used to determine a target set of points visible from the set of points P (including points A and B) from the camera 120, that is, from a predetermined observation point in the environment, the origin of the coordinate system 210. This target set will then continue to include point A, but not point B.

[0057] For the proposed actions, the typical shadow behavior of a LiDAR sensor can be considered. A 3D LiDAR sensor scans its environment in a discretized angular pattern not only in the horizontal (azimuth) and vertical (polar) directions. For each discretized point (such as Figure 1 ), the distance to the nearest reflecting surface is always measured. Due to the discretization, no point is identified within a certain azimuth and vertical or polar spacing from one point. This means graphically that the identified point casts a pyramidal shadow on all other points behind it, as Figure 3 As can be seen in the figure and will be explained later.

[0058] This behavior of the LiDAR sensor should now be simulated for different views or observation points. For the sake of clarification, the algorithm is explained using the example of the conversion of LiDAR data into a camera image, as mentioned earlier. However, it should be mentioned again that this can generally be from a coordinate system such as Figure 2 200 in the original coordinate system - to any other coordinate system - such as Figure 1 The target coordinate system - the transformation of the coordinate system 210 in the coordinate system 210 is not limited to the application case of a lidar camera. This is for explanation only.

[0059] For this purpose, we can first convert the coordinate system (CS) L A collection or point cloud L of points of a lidar sensor (e.g. corresponding to coordinate system 200) P Transform to the camera coordinate system (CS) c (e.g. corresponding to coordinate system 210), that is, by the following transformation

[0060] C P=T CL L P among them

[0061]

[0062] can be given by an external calibration. After this transformation, then c P All points P = (x', y', z') in can be converted or transformed into spherical coordinates by the following equation At this point it should be mentioned that a typical robot coordinate orientation is assumed here, in which x stands for forward, y stands for left and z stands for upward.

[0063] For example, this transformation from Cartesian to spherical coordinates is performed according to:

[0064]

[0065] If the points in the original coordinate system should already be in spherical coordinates, then only a transformation to the target coordinate system is required.

[0066] In addition, the polar or azimuthal directions are determined to have a specific dimension (size) in the target coordinate system. Cell G ij A spherical curved or spherical grid G. Such a grid 310 having cells 308 is Figure 3 302 and in spherical coordinates. Here, each cell is defined, for example, by two grid lines 304 extending in the polar direction and two grid lines 306 extending in the azimuthal direction. For example, the following should apply:

[0067]

[0068] Then each point is assigned to one of these cells by determining its coordinates, for example, according to:

[0069]

[0070] It should be noted here that a cell describes an entire frustum in 3D space, such as Figure 3 Indicated in.

[0071] As mentioned, there may then be cells to which no points are assigned, cells to which exactly one point is assigned, and cells to which more than one point is assigned. Figure 3 In the example of , three points P1, P2 and P3 are assigned to the cell 308. For each cell having a plurality of points, the point having the least or minimum spacing from the coordinate origin (here the origin 302) can be determined, for example, by comparing the corresponding values ​​of the radial coordinate (r coordinate component). For example, the minimum r or the smallest r or the smallest spacing for a cell can be referred to as r0.

[0072] Furthermore, all points of the cell whose minimum distance r0 is exceeded by more than a first predetermined threshold value Δr can then be identified, wherein for these points

[0073] r>r0+Δr

[0074] applies, and is removed from the entire original set of points.

[0075] Δr=α r ·r0

[0076] can be applied. The reason for such a (first) threshold Δr is, for example, a practical property. Due to the different viewing angles between the camera and the lidar, a cell may also contain multiple points that are adjacent on nearby faces. These points, for example, should not be removed, but only those points that are clearly far apart should be removed.

[0077] For example, it follows from practical considerations that by multiplying by the factor α r to determine Δr; this is Figure 4 For points located in a plane orthogonal to the image plane, such as plane 400, the differences in the r coordinates between these points (here points P4, P5, P6) increase with a fixed Δθ offset, depending on how far apart these points are. These differences relative to the minimum spacing r0 are denoted by r1 or r2, respectively. For this reason, Δr should be determined in a multiplicative manner so as not to erroneously remove points located on the same plane.

[0078] Depending on the application, however, the (first) threshold value Δr can also be determined using other functions, for example as an exponential function or as a constant.

[0079] exist Figure 5 The method according to the invention is shown in a preferred embodiment as a flowchart in FIG. 5 . After the start 500 , in step 502 , a set 503 of points (or generally a point cloud) is provided. In addition to the set of points or point cloud L detected or recorded, for example, by a lidar (or other) sensor, P In addition, the input of the described algorithm also includes an external transformation T to another coordinate system with another origin (e.g. a camera) CL The algorithm then aims to alter the point cloud so that if the point cloud were to be viewed from another foothold or observation point, points that would be behind objects in the original scene are removed.

[0080] To this end, in step 504, by means of the transformation T CL The point set or point cloud L P Transform from the original or LiDAR sensor coordinate system to the target coordinate system of interest, resulting in a transformed point cloud c P Generate spherical coordinates, that is, transform into spherical coordinates, that is, for each point there is The transformation typically does not directly result in spherical coordinates, but must be converted to spherical coordinates separately if necessary. This set of points in spherical coordinates can also be called

[0081] In step 506, a spherical curved grid is determined in spherical coordinates, as shown in FIG. Figure 3 Here, in particular, a so-called "r buffer" can be initialized, in which (later) the points with the minimum spacing can be stored for each cell. For example, for each cell, this value of the "r buffer" can first be initialized to infinity.

[0082] Each point is assigned to a cell in step 508. At the same time, the point with the minimum distance from the origin of the target coordinate system can also be determined for these cells.

[0083] This can be done in a loop so that, for example, in step 510 it is always checked whether the point just assigned to the cell is a member of the set If not, then in step 512, the grid index (i, j) as mentioned above is determined, that is, the cell to which the point can be assigned. Therefore, for each point, the corresponding grid index can be calculated according to its spherical coordinates. The grid index corresponding to the point (e.g., represented by the point index) is stored for later use.

[0084] In step 514, it can then be checked whether the value of the "r buffer" of the cell is greater than the value of the r coordinate of the current point. If so, this means that the current point has a smaller spacing than another point assigned to the cell. Then, in step 516, the value of the "r buffer" of the cell is replaced with the value of the r coordinate of the current point.

[0085] In this way, all points are therefore assigned to cells and for each cell the point with the smallest spacing is determined at the same time or at least the value of this smallest spacing is determined.

[0086] If it is determined during the loop in step 510 that the point just assigned to the cell is a set Then, for all those cells that have more than one point assigned to them, identify the specific points that can be removed from the set.

[0087] This can also be done in a loop, so that, for example, in step 518 it is always checked whether the point for which the check has just been performed is a member of the set If not, then in step 520, for example, the grid index of the current point is checked, i.e., to which cell the current point is assigned. For example, in step 522, the value of the "r buffer" of the cell can be loaded (the value has been determined in the previous cycle). This value corresponds to the minimum spacing r0.

[0088] In step 524, it is then checked whether the distance, ie the value of the r coordinate of the current point, is less than or equal to the minimum distance plus the mentioned (first) threshold value, ie whether the following applies:

[0089]

[0090] If so, then in step 526 the point is assigned to the target set If this is not the case, nothing is done and the point in question is therefore not considered further.

[0091] If it is determined during the loop in step 518 that the point just tested is a set The final point can then be output or provided in step 528 as the target set formed here. or 529 , the method may end at step 530 .

[0092] It should be mentioned that, instead of checking in step 524 whether the distance, i.e. the value of the r coordinate of the current point, is less than or equal to the minimum distance plus the mentioned (first) threshold, it is also possible to check whether the value of the r coordinate of the current point is greater than the minimum distance plus the mentioned (first) threshold, i.e.

[0093] That is, whether the following apply:

[0094]

[0095] In this case, if this is the case, then in alternative step 526 the point will be removed from the set If this is not the case, nothing is done, i.e. the point in question remains in the set The remaining set can then be output or provided in alternative step 528 As the target set

[0096] Furthermore, it is preferred if environment information 533, such as image information or an image, with coordinates in the target coordinate system is provided in step 532. In step 534, points of the target set visible from a pre-given observation point in the environment can then be associated with the environment information.

[0097] Furthermore, within the scope of the method according to the invention, in a preferred embodiment, the parameters required for the grid and Δθ (distance) can also be obtained from a pre-given set of points or a point cloud L P Derived or determined automatically. In this case, it can be assumed, for example, that the point cloud comes from a typical laser scanner (lidar sensor) which has a characteristic ring around a vertical axis.

[0098] To this end, in step 540, if these points in the original coordinate system, namely L P If the coordinates of the set of points in do not already exist, the coordinates are determined in spherical coordinates.

[0099] In step 542, the frequency distribution of the points in the polar direction is then determined, i.e., a θ histogram is created. To this end, the θ coordinates of all points can first be binned with a certain accuracy, such as 2 16 The histogram (frequency distribution) then contains peaks of approximately the same height for each laser ring. These peaks can be counted by first assigning a threshold to the histogram, for example, in order to remove noise, and then finding local maxima and counting them.

[0100] In step 544, the size (angle) of the cell in the polar direction is then determined, for example based on the number of local maxima of the frequency distribution. The value of the spacing Δθ can be determined according to

[0101]

[0102] is determined, where N specifies the number of local maxima of the frequency distribution. For example, in step 546, the size (angle) of the cell in the azimuthal direction is determined for each local maximum based on the number of point clusters. The value of the spacing is can be determined directly from the one peak, since this height corresponds to the number of points with a particular value of θ. In practice, not all measurements lead to a point in the point cloud, and therefore Preferably, it is determined according to the maximum number of points in a single laser ring, for example according to

[0103]

[0104] Where M specifies the number of points for each local maximum of the frequency distribution. In practice, multiplying the two parameters by a factor of 2 (or a factor greater than 2) leads to a more robust result. It is thus basically ensured that visible and hidden points arrive in similar cells, rather than each point getting its own cell and not being filtered. It goes without saying that other values ​​or factors can also be used for the multiplication. These spacings can also be adapted or pre-given based on the distance of the predetermined observation point from the origin of the original coordinate system.

[0105] exist Figure 6 , a diagram is shown by way of example before and after the application of the proposed method. Figure 6 A) Draw the object outline 652, that is, from the perspective of the device's camera, for example along the Figure 2 The axis x' is drawn. Figure 2 In contrast there, the lidar sensor should be arranged slightly higher than the camera, ie it can, so to speak, see better over objects.

[0106] In this case, points located on the object are represented by spheres (small dots), while points located behind the object are represented by crosses. Both types of points are detected by the LiDAR sensor; the different illustrations should only be used for explanation purposes here. It can be seen in this case that when the points of the LiDAR sensor are transformed into the coordinate system of the camera (e.g. Figure 6 A), points that are actually behind the object are projected onto the object.

[0107] Object outline 652 is also shown on the right (view B or Figure 6 B), but there, in a preferred embodiment, certain points are removed according to the method according to the invention. It can be seen here that almost all points that are located behind the object and have previously been mapped onto the object are no longer present. It should be mentioned that the object contour 652 is drawn here only for the purpose of illustration; this object contour is not known to the algorithm used for removing the points.

Claims

1. A method for determining a target set of points in an environment (150) from a set of point groups, the point group comprising one or more points (P, A, B), which is visible or invisible from a predetermined observation point (214) in the environment (150), in particular for navigation of a vehicle moving at least partially automatically, comprising: providing (502) a set (503) of point groups in the environment having coordinates in the original coordinate system (200), transforming (504) the coordinates of the set (503) of point groups into spherical coordinates in a target coordinate system (300) wherein the observation point (214) in the environment is located at the origin (302) of said target coordinate system (300), determining (506) a spherically curved grid (310) in spherical coordinates in the target coordinate system (300), wherein the grid (310) includes grid points and / or cells (308), assigning (508) each point group of at least a portion of the set to a cell (308) or grid point, For each of at least a portion of the cells (308) of the grid or for each of at least a portion of the grid points of the grid to which more than one point group is assigned: determining a minimum spacing (r0) and / or a maximum spacing based on the spacing of the point group from the origin (302) of the target coordinate system (300), a) determining a target set (529) of point groups visible from a predetermined observation point (214) in the environment, including those point groups whose distance from the origin of the target coordinate system for each cell or grid point exceeds a minimum distance (r0) by at most a predetermined first threshold value (Δr) and / or whose distance from the origin of the target coordinate system for each cell or grid point is below a maximum distance minus a predetermined second threshold value, and in particular also including those point groups to which cells or grid points are assigned only one point group, and / or b) determining a target set of point groups that are not visible from a predetermined observation point (302) in the environment, including those point groups whose spacing from the origin of the target coordinate system (300) for each cell or grid point exceeds a minimum spacing (r0) by more than a predetermined first threshold (Δr) and / or whose spacing from the origin of the target coordinate system for each cell or grid point exceeds a maximum spacing minus a predetermined second threshold, and A target set (529) or a target set of point groups is provided (528), and is particularly useful for navigating and / or moving a vehicle that moves at least partially autonomously. 2 . The method according to claim 1 , wherein the respective predetermined first threshold value (Δr) and / or second threshold value depends on the respective minimum spacing (r0) or maximum spacing, in particular corresponds to the respective minimum or maximum spacing multiplied by a respective factor.

3. The method of claim 1 or 2, wherein determining (506) a grid (310) in spherical coordinates in the target coordinate system comprises: Determine (540) the coordinates of the set of point groups in spherical coordinates or using the original coordinate system, Determine the frequency distribution of the (542) point group in the polar direction, determining (544) the size of the cell (Δθ) or the spacing of the grid points in the polar direction based on the number of local maxima of the frequency distribution, and For each local maximum, the size of the cell in the azimuthal direction is determined (546) based on the number of point clusters. or the spacing of the grid points.

4. The method according to any of the preceding claims, wherein the size (Δθ) of the cell (308) in the polar direction or the spacing of the grid points and / or the size of the cell (308) in the azimuthal direction Or the spacing of the grid points is determined based on the distance of a predetermined observation point (302) from the origin (202) of the original coordinate system.

5. The method according to any of the preceding claims, wherein the number of point groups in the environment is or has been determined by means of distance measurement, in particular by means of a lidar sensor (110).

6. The method according to claim 5, wherein the starting point of the distance measurement, in particular the position of the lidar sensor, is located at the origin (202) of the original coordinate system (200).

7. The method according to any one of the preceding claims, further comprising: providing (532) environmental information (533) having coordinates in a target coordinate system, and At least a portion of the point clusters in the target set (529) of point clusters visible from a pre-given observation point in the environment are associated (534) with environmental information (533).

8. The method according to claim 7, wherein the environmental information (533) comprises image information and / or semantic information and is or has been determined, in particular, by means of an image sensor (120).

9. The method according to claim 8, wherein the environmental information (533) is shown from the perspective of the predetermined observation point (214), and wherein in particular the position of the image sensor is located at the origin (302) of the target coordinate system.

10. The method according to any one of claims 7 to 9, further comprising: determining control information for the mobile device (100) based on the environmental information (533) and the point group associated therewith of the target set (529), and Control information is provided and the device (100) is actuated, in particular based on the control information.

11. A system (106, 130) for data processing, comprising means for performing the method according to any of the preceding claims.

12. A device (100) configured to provide a set (503) of point groups in an environment with coordinates in an original coordinate system, and preferably to provide environment information (533) with coordinates in a target coordinate system, wherein the device is further configured to obtain a target set (529) of point groups that are visible or invisible from a predetermined observation point in the environment, which has been determined from a set of points according to a method according to any one of claims 1 to 10, and in particular for navigation, Preferably, it has a distance measurement or lidar sensor (110) for detecting point groups in the environment, more preferably, it has a sensor for detecting environmental information, in particular an image sensor (120), further preferably, it has a control and / or regulation unit (104) and a drive unit (102) for the mobile device (100), and further preferably, it has a system (106) according to claim 11.

13. The device (100) according to claim 12 is constructed as an at least partially automatically moving vehicle, in particular a passenger vehicle or a freight vehicle, and / or is constructed as a robot, in particular a household robot, such as a suction and / or wiping robot, a floor or street cleaning device or a lawn mowing robot, and / or is constructed as a drone.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method steps of the method according to any one of claims 1 to 10 when the program is executed on a computing unit.

15. A computer-readable storage medium having stored thereon the computer program according to claim 14.

Images