METHOD FOR DETERMINING POINT GROUPS THAT ARE VISIBLE OR NOT VISIBLE FROM A PREDECIDENT VIEWING POINT

Description

[0001] The present invention relates to a method for determining a target set of point groups that are visible or not visible from a given viewing point, a data processing system and a computer program for carrying it out, and a device for using such a target set of point groups. Background of the invention

[0002] Devices such as vehicles or robots that move at least semi-automatically move along a path in an environment such as an apartment, a garden, a factory hall, or on the street, in the air, or in water. For this to work, the current position and / or orientation (also called pose) of the mobile device needs to be known and continuously updated. This can be achieved, for example, using laser distance measurement (e.g., with lidar sensors). Alternatively, environmental information such as images of the surroundings (e.g., using a camera) can be used. US 2019 / 197711 A1 deals generally with determining points in a point cloud that are visible from another point and applies this in image processing. US 2020 / 018852 A1 deals with the navigation of automated vehicles using lidar. Disclosure of the invention

[0003] According to the invention, a method for determining a target set of point groups, a data processing system, a computer program for its execution, and a device for using such a target set of point groups, comprising the features of the independent claims, are proposed. Advantageous embodiments are the subject of the dependent claims and the following description.

[0004] The invention relates to sets of points or a so-called point cloud in an environment. Such points or point clouds are acquired, for example, by means of a lidar sensor or another type of distance measurement and each represents a reflection point, for example, of a laser beam emitted by the lidar sensor. If these points are considered in a coordinate system with the lidar sensor at its origin, then, for example, distances to objects in the environment can be determined based on the points or their coordinates. For example, in devices, especially mobile devices as mentioned above, this enables orientation in the environment and, based on this, (automated) navigation and, in particular, movement of the device within the environment.

[0005] Another possibility with such devices is to capture environmental information, such as images, for example using a camera. These images, which can also be semantically analyzed to recognize specific objects, can be used for orientation within the environment and, based on this, for (automated) movement of the device.

[0006] If both types of environmental sensing are available—the aforementioned points or point clouds on the one hand, and environmental information such as images on the other—the points can be associated with the environmental information. For example, the distances between the points can be mapped onto an image. It can be assumed, for instance, that for an application, labels predicted by an image classification algorithm in such an image should be mapped onto lidar point clouds of the same scene in the surrounding environment, in order to augment the semantic information with 3D information.

[0007] In practice, the point cloud and the environmental information are captured from different perspectives or viewpoints, for example, by a device with a lidar sensor and a camera. This difference in perspective leads to a visibility problem. Particularly at the edges of objects in the environment, the lidar sensor might detect two separate points in the environment that are farther away than the object itself, while the camera only detects one of these points. The other point is obscured by the object.

[0008] If information from the camera were mapped onto the lidar point cloud, or vice versa, without analyzing the visibility of the points from the camera's viewing point, the point not captured by the camera would be incorrectly labeled, for example, with a color or text from the top of the object. In general, this point, which is actually invisible—and thus its distance, which is greater than that of the object—would be mapped onto the object or its surface.

[0009] The problem of determining which points in a point cloud are visible from a given viewpoint can be solved, for example, using the "Hidden Point Removal" method described by S. Katz et al., "ACM Transactions on Graphics", Volume 26, Issue 3, July 2007, pp. 24-25. This approach has also been successfully applied to camera-LiDAR mapping by Vechersky et al., "Colourising Point Clouds Using Independent Cameras", in IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3575-3582, Oct. 2018.

[0010] The approach by Katz et al. describes an operator that "inverts" all points in the given point cloud. This is also called "spherical flipping" because all points are reflected across a sphere containing all the points (although the transformation function could be different). The approach then constructs a convex hull from the inverted point cloud and the camera's coordinate center. It is shown that every point lying on this convex hull is visible to the camera, while all other points are not and can be removed.

[0011] Other approaches to solving the aforementioned problem are based, for example, on surface reconstructions and then use methods of so-called raycasting or similar techniques.

[0012] Within the scope of the present invention, a method is proposed for determining a target set of point groups that are visible or invisible from a given observation point in the environment, from an (existing) set of point groups. A point group can, for example, comprise only one (single) point or several points. As will be explained in more detail later, point groups within the set can be distinct from one another. In other words, points within the set can be grouped.

[0013] Although the proposed procedure is described below in particular with regard to the aforementioned example with a lidar sensor and camera that capture (or see) the environment from different viewpoints or viewing angles, it is possible to determine a target set from any set of points or point groups that are visible (or not visible) from a given viewpoint in the environment.

[0014] First, a set of point groups in the environment is provided, with coordinates in an origin coordinate system. The points within the point groups, or the point groups themselves, can be represented in Cartesian or spherical coordinates, for example. The set of point groups in the environment can be determined, or may have been determined, by means of distance measurement, particularly using a lidar sensor. A starting point for the distance measurement, specifically a position of the lidar sensor, is then located at an origin of the origin coordinate system.

[0015] The coordinates of the set of point groups are then transformed into spherical coordinates in a target coordinate system, where the point of observation (from which the points to be determined should be visible, e.g. the location of the camera) lies in the environment at the origin of the target coordinate system.

[0016] Furthermore, a spherically curved grid in spherical coordinates is defined in the target coordinate system, where the grid comprises grid points and / or cells. The grid points and / or cells are, in particular, uniformly spaced. A simple example would be a grid on a spherical surface with the center of the sphere at the origin of the target coordinate system, where the grid lines are uniformly spaced in the polar and azimuthal directions. In this way, four adjacent grid points (where two grid lines intersect) each form a cell, with the four grid points as their vertices. The underlying principle here is the way in which points are detected by a typical lidar sensor, namely with uniform spacing in the polar and azimuthal directions.If such a grid were created for a lidar sensor in the original coordinate system, each cell could be assigned exactly one point or group of points, depending on the spacing of the grid lines. Such a uniform arrangement is particularly universally applicable, i.e., for various environments. However, a non-uniform arrangement of grid points and / or cells is also conceivable. For example, cells or grid points in the area of ​​the floor and / or ceiling (in the environment, e.g., a hall) can be arranged more finely or densely than in the rest of the environment, since occlusions are typically not expected in the rest.

[0017] In the target coordinate system, each group of points (which may, for example, each contain only one point) from at least a portion of the set is assigned to a cell or grid point. Since the origin of the target coordinate system (the desired viewing point and thus the center of the spherically curved grid) does not coincide with the origin of the original coordinate system (where, for example, the lidar sensor used to detect the points in the set might be located), it is possible that a cell can be assigned not just one point or point group, but several point groups. The assignment of a point to a cell is based on spherical coordinates.

[0018] Visually, the line connecting the origin of the coordinate system to a specific point would intersect exactly one of the cells; the point can then be assigned to this cell. For multiple points within a point group, an average of the points' coordinates can also be used. Multiple points within a point group are relevant, for example, when these points were detected simultaneously or very close together in time by a lidar sensor. Since the cells are defined, or at least can be defined, by the grid points, a point group can also be assigned to a grid point instead of a cell—for example, the one closest to the intersection of the imaginary line connecting the points on the sphere's surface with the grid.

[0019] Against this background, cells or grid points can be determined to which more than one point group is assigned. For each of at least some of these grid cells or for each of at least some of these grid points, a minimum or maximum distance from the origin of the target coordinate system is then determined (a corresponding point group exists for this). For example, the distance from the origin of the target coordinate system can be determined for each cell or grid point within each point group. The distance of a point from the origin corresponds to the radial coordinate in spherical coordinates. If a point group comprises several points, an average distance between the several points can be used, or the minimum or maximum distance that occurs among the several points in the point group.

[0020] Based on this, the target set of point groups visible from a given viewpoint in the environment can be determined such that it includes those point groups whose distance from the origin of the target coordinate system, per cell or grid point, corresponds to the minimum distance or exceeds the minimum distance by no more than a given first threshold, and / or whose distance from the origin of the target coordinate system, per cell or grid point, is less than the maximum distance minus a given second threshold. Furthermore, it can also include those point groups assigned to cells or grid points to which only one point group is assigned.

[0021] To achieve this, for example, for each cell or grid point, one or more point groups can be identified whose distance from the origin of the target coordinate system is greater than the minimum distance, or exceeds the minimum distance by more than a predefined first threshold, and / or whose distance from the origin of the target coordinate system exceeds the maximum distance minus a predefined second threshold. These can then be removed from the set, and the remaining point groups can then be provided as the target set.

[0022] The idea here is that, from the desired viewing point – if a lidar sensor were located there – only one point or group of points would be present in a cell. However, if two or more points or groups of points are assigned to a cell, and these points or groups are at very different distances, it's possible that not all of them could be detected by a single lidar sensor. The point with the shortest distance would be caused by a nearby object, while the point with a significantly greater distance would be behind the object and might not be detected at all.

[0023] If all points or point groups whose distance from the origin of the target coordinate system is greater than the minimum distance are removed from the set, only points or point groups that are visible from the desired viewing point remain. It can be specified that only points or point groups whose distance from the origin of the target coordinate system exceeds the minimum distance by more than a predefined first threshold are removed. This first threshold can depend on the minimum distance, in particular, it can be the respective minimum distance multiplied by a specific factor. This takes into account the fact that points with a distance only slightly greater than the minimum will, in practice, not lie behind an object, but will still be on the object's surface, which, for example,The location is slightly angled in the surroundings, so the distance is slightly greater there. By choosing the first threshold as zero, it is also possible to focus on (only) the minimum distance.

[0024] Such points or point groups can also be identified by searching for points or point groups whose distance from the origin of the target coordinate system exceeds the maximum distance minus a predefined second threshold. This method, too, identifies points or point groups that should not actually be visible from the observation point, provided the second threshold is chosen appropriately. This second threshold can depend on the maximum distance, in particular, it can be the respective maximum distance multiplied by a specific factor.

[0025] The target set can also be obtained by directly defining as the target set those point groups whose distance from the origin of the target coordinate system, per cell or grid point, exceeds the minimum distance by no more than a predefined first threshold (or, with a threshold of zero, equals the minimum distance), and / or whose distance from the origin of the target coordinate system, per cell or grid point, is less than the maximum distance minus a predefined second threshold. Point groups assigned to cells or grid points to which only one point group is assigned can also be included in the target set, because in these cases the single distance also corresponds to the minimum distance.

[0026] These identified points or point groups also represent those points or point groups in the set that are not visible from the given observation point in the surrounding environment. Therefore, if precisely these points or point groups are desired for a specific application, they can be defined as the target set.

[0027] The proposed approach is significantly faster than the methods mentioned earlier, as it requires neither surface reconstruction nor a convex hull. Both of these tasks are computationally intensive, especially for larger point clouds. The approach mentioned above for estimating the visibility of point clouds has a time complexity of order O(n*log n), and a typical approach for surface reconstruction has a time complexity of order O(n^2). In contrast, the approach proposed within the scope of the invention has a time complexity of only order O(n) and still delivers suitable results, for example, for the application with a lidar sensor and camera. In practice, this leads to a time saving of an order of magnitude.

[0028] Furthermore, the proposed approach is simple and quick to implement, making it a suitable rapid solution to the described problem. It has also been shown that the approach by Katz et al. filters out many distant and ground-level points that are actually visible from the viewpoint. The proposed approach retains these points. Even if this results in slightly more false positives, the impact is negligible in practice.

[0029] Preferably, determining the grid in spherical coordinates in the target coordinate system involves determining the coordinates of the set of point groups in the original coordinate system in spherical coordinates, or using them if the points or point groups are already in that configuration. A frequency distribution of the point groups in the polar direction can then be determined. Based on a number of local maxima of the frequency distribution, a dimension (as an angle) of the cells or of the distances (as an angle) between the grid points in the polar direction can then be determined. The number of points itself yields a cell dimension corresponding to the underlying lidar sensor; however, a value twice this or another suitable value can also be chosen.The dimensions (as angles) of the cells or the spacing (as angles) of the grid points in the azimuth direction can be determined based on a number of point groups per local maximum. The number of points itself also determines the cell dimension according to the underlying lidar sensor, but it can also be, for example, doubled or another suitable value. In this way, the grid itself can also be determined automatically.

[0030] Advantageously, the dimensions of the cells or the spacing of the grid points in the polar direction and / or the dimensions of the cells or the spacing of the grid points in the azimuth direction are determined based on the distance of the specified viewing point from an origin of the origin coordinate system. For example, the further apart the sensors (LiDAR sensor and camera) are, the larger the cells can be.

[0031] Preferably, environmental information, including coordinates in the target coordinate system, such as image information and / or semantic information, is also provided. This allows at least some of the point groups in the target set, visible from a given viewpoint in the environment, to be associated with this environmental information. In other words, points with distance information, for example, can be mapped onto an image.

[0032] This then allows, for example, the determination of control information for moving a device, based on environmental information and the associated point groups of the target set. This control information can then be provided (to the device) so that the device can be controlled based on this information, i.e., moved within the environment.

[0033] The proposed approach can be used in particular whenever the visibility of points in a point cloud needs to be derived from a specific viewing angle. This applies both to artificially generated point clouds (e.g., from a CAD model) and to recorded real-world data (e.g., point clouds from a lidar sensor that need to be transformed to a different viewing angle, as explained in detail above).

[0034] One application for the proposed approach is the combination of a (possibly different) distance sensor (e.g., lidar or radar) and an (possibly different) image-based sensor (e.g., camera or thermal imaging device). The proposed approach can be used in such systems to improve data quality by aligning the two data sources. This allows many other algorithms to benefit from the improved data quality and quantity (since the data now contains not only image / semantic or 3D information, but both).

[0035] The proposed approach can be used generally to transform the viewpoint of a point cloud source "correctly," i.e., with visibility limitations and not just with a distortion of the reference frame. The proposed approach can also be used to fuse multiple point cloud sources (Lidar, radar, stereovision, structured light) into a single point cloud from any (virtual) viewpoint.

[0036] The proposed approach can also be used, for example, to simulate a virtual (Lidar) sensor, either in synthetic environments or on real-world data. A specific application of the proposed approach is the detection of boxes and other objects, for example, in warehouse environments, using a Lidar sensor and a camera sensor. For this purpose, the proposed approach can improve the mapping of RGB data from a camera and semantic labels from these camera images to a Lidar point cloud. The point cloud could then be accumulated, and objects within it could be detected. The proposed approach can also be used, for example, in the context of autonomous driving when information from a Lidar or radar sensor and one or more cameras is or is to be combined.

[0037] A data processing system according to the invention comprises means for executing the method according to the invention or its method steps. The system can be a computer or server, e.g., in a so-called cloud or cloud environment. In the case of an application for a (mobile) device, the set of point groups can then be transmitted there (e.g., via a wireless data connection), and the target set can then be transmitted back to the device. It is also conceivable that such a data processing system could be a computer or a control unit in such a (mobile) device.

[0038] The invention also relates to a device configured to provide a set of point groups in an environment, with coordinates in an origin coordinate system, and preferably also to provide environmental information with coordinates in a target coordinate system. The device is further configured to obtain a target set of point groups, which are visible or invisible from a predetermined viewing point in the environment, and which has been determined from the set of points according to a method according to the invention. Preferably, the device includes a distance measurement or lidar sensor for detecting the point groups in the environment, more preferably also a sensor, in particular an image sensor, for detecting environmental information, and more preferably also a control and / or regulation unit and a drive unit for moving the device. The data processing system can also be included in the device, as mentioned.

[0039] The device is preferably designed as a vehicle that moves at least partially automatically, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. vacuuming and / or mopping robot, floor or street cleaning device or lawn mowing robot, and / or as a drone.

[0040] Implementing a method according to the invention in the form of a computer program or computer program product with commands or program code for carrying out the method or the method steps is also advantageous, as this incurs particularly low costs, especially if an executing control unit is already available for other tasks. Finally, a machine-readable storage medium is provided with a computer program stored on it as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical storage media, such as hard drives, flash memory, EEPROMs, DVDs, etc. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0041] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

[0042] The invention is schematically illustrated in the drawing using exemplary embodiments and is described below with reference to the drawing. Brief description of the drawings

[0043] Figure 1 schematically shows a device in an environment to illustrate the invention. Figure 2 shows the environment. Figure 1 with further details. Figure 3 shows a grid to illustrate the invention. Figure 4 shows different distances between points. Figure 5 shows a sequence of a method according to the invention in a preferred embodiment. Figure 6 shows before and after representations of a method according to the invention in a preferred embodiment. embodiment(s) of the invention

[0044] In Figure 1A schematic representation of a device 100 in an environment 150 is shown, with reference to which the invention will be explained. The device 100 is, for example, a robot or an automated (or autonomous) vehicle, with a drive unit 102, a control and / or regulation unit 106, and a communication and computing unit 106, through which a wireless data connection with a data processing system 130, e.g., a server, is possible. Furthermore, the device 100 has a lidar sensor 110 for laser distance measurement and a camera 120 for capturing images of the environment 150. The device 100 can, in particular, be a device according to the invention in a preferred embodiment.

[0045] Environment 150 is, for example, the interior of a building, such as a corridor with side walls. An object 152 or obstacle is shown on one of the side walls. It may be intended, for example, that the lidar sensor 110 and the camera 120 are to acquire information (distance and image information) of environment 150, based on which the device 100 is then to move within environment 150. For example, it may be intended that the device 100 is to move along the corridor, but should move around object 152.

[0046] For this purpose, the environment 150 can be scanned using the lidar sensor 110. The laser scanner in the lidar sensor is moved, while a multitude or a quantity of points P are obtained by means of emitted laser beams and received, reflected radiation; each point represents a point or a small area in the environment, e.g., on an object from which reflected radiation is received.

[0047] In Figure 2 The surroundings are 150% out Figure 1The diagram is shown again. Only the lidar sensor 110 and the camera 120 of the device are shown. It is particularly evident that the lidar sensor 110 and the camera 120 are spatially separated and therefore have different viewpoints or viewing angles for the environment 150. Although the situation is only depicted two-dimensionally here, the lidar sensor 110 and the camera 120 can also be separated in the third dimension, but this is not a requirement. Furthermore, the set of points P is shown. Figure 1 Two points, A and B, were selected and illustrated as examples.

[0048] Based on the Figure 2It is evident that the lidar sensor 110 can see or detect both point A and point B. Camera 120, however, due to its different viewing angle or point of view 214, can only see or detect point A. Therefore, if information from the camera were to be projected onto the lidar point cloud (the set of points)... P, (which also include A and B) or vice versa, without analyzing the visibility of the points from the viewing point of camera 120 (camera position), point B would be incorrectly labelled, for example, with a color or inscription from the top of object 152 or generally associated with a top or surface of object 152.

[0049] Points P (including A and B) can be represented by coordinates in a coordinate system 200 of the lidar sensor 110, with the lidar sensor 110 at the origin 202 of this coordinate system. The coordinate system 200 is shown as an example Cartesian coordinate system with axes x and y. An axis z would be perpendicular to the drawing plane and is not shown. In this way, each point can be precisely defined. If the coordinate system 200 were one with spherical coordinates, the distances of the points from the origin 202 would, for example, correspond to the magnitude of the radial coordinate.

[0050] A coordinate system 210 can also be defined for camera 120, with camera 120 at the origin 212 of this coordinate system 210. The coordinate system 210 is shown as an example Cartesian coordinate system with axes x' and y'. An axis z' would be perpendicular to the drawing plane and is not shown.

[0051] To determine a target set of points visible from camera 120, i.e., from a given viewing point in the environment – ​​the origin of the coordinate system 210 – from the set of points P (comprising points A and B), the procedure described below as an example can be used. This target set will then still include point A, but not point B.

[0052] The proposed approach can be based on the typical shadowing behavior of a lidar sensor. A 3D lidar sensor scans its surroundings in a discretized angular pattern, both horizontally (azimuthally) and vertically (polarly). For each discretized point—such as the points P in Figure 1The distance to the nearest reflecting surface is always measured. Due to discretization, no other point is detected within a certain azimuthal and vertical (or polar) distance of a given point. Figuratively speaking, this means that the detected point casts a pyramid-shaped shadow on all other points behind it, as shown in... Figure 3 to be seen and will be explained later.

[0053] This behavior of a lidar sensor will now be replicated for a different viewing angle or point of view. To illustrate this, the algorithm will be explained using the example of converting lidar data into a camera image, as mentioned above. However, it should be noted again that this is a transformation from a coordinate system - an origin coordinate system such as the 200 coordinate system in Figure 2 - into any other coordinate system - a target coordinate system such as the 210 coordinate system in Figure 1- can be general and is not limited to the use case of lidar cameras. This is for illustrative purposes only.

[0054] For this purpose, a set of points or a point cloud can first be used. LP are considered, which are viewed by a lidar sensor in the coordinate system ( CS ) L (corresponds, for example, to the 200 coordinate system) originates in the camera coordinate system ( CS ) C (corresponds, for example, to the 210 coordinate system) can be transformed, namely by a transformation P <mprescripts / > C <none / > = T CL P <mprescripts / > L <none / > where T CL = R CL t CL 0 T 1 This can be achieved through an extrinsic calibration. After this transformation, all points P = ( x', y', z') in CP by the following equations in spherical coordinates P̂ = (r, θ, φ ) are converted or transformed. It should be noted that typical robot coordinate orientations are assumed, with x for forward, y for left, and z for up.

[0055] This transformation from Cartesian to spherical coordinates is carried out, for example, according to r = x ′ 2 + y ′ 2 + z ′ 2 φ = arctan z ′ r θ = arctan y ′ x ′ If the points in the original coordinate system are already in spherical coordinates, only the transformation to the target coordinate system is necessary.

[0056] Furthermore, a spherically curved or spherical grid is used in the target coordinate system. G with cells G ij of a specific dimension (measurement) Δ θ × Δ φ determined in the polar direction or azimuth direction. Such a grid 310 with cells 308 is in Figure 3 in a coordinate system with origin 302 and represented in spherical coordinates. Each cell is defined here, for example, by two grid lines 304 running in the polar direction and two grid lines 306 running in the azimuth direction. The following should apply, for example: 0 ≤ i ≤ 360 ° Δ φ , i ∈ ℕ 0 0 ≤ j ≤ 180 ° Δ θ , j ∈ ℕ 0 .

[0057] Then each point P̂one of these cells is assigned, for example by determining its coordinates according to: i P ^ = φ P ^ Δ φ , j P ^ = θ P ^ Δ θ .

[0058] It should be noted that one cell describes an entire truncated cone in 3D space, as in Figure 3 hinted at.

[0059] As mentioned, there can then be cells that have no point assigned to them, cells that have exactly one point assigned to them, and cells that have more than one point assigned to them. In the example in Figure 3 For example, cell 308 is assigned three points, P1, P2, and P3. For any cell with multiple points, the point with the smallest or minimum distance to the origin (here, origin 302) can be determined by comparing, for example, the respective values ​​for the radial coordinate (r-coordinate component). The minimum or smallest r, or the minimum distance, for a cell can be expressed, for example, as... r will be designated as 0.

[0060] Furthermore, all points of the cell can then be separated by a distance that is the minimum distance. r 0 by more than a given first threshold Δ r exceeds, for which therefore r > r 0 + Δ r It is valid, identified, and removed from the entire original set of points. This can be done in a way that... Δ r = α r ⋅ r 0 apply. One reason for such a (first) threshold Δ r This is, for example, of a practical nature. Due to the different viewing angles between the camera and the lidar, a cell can contain several points that are located next to each other on a nearby surface. These should not be removed, but only those that are significantly farther away.

[0061] A method for determining Δ r by multiplication by a factor α r This also arises, for example, from practical considerations; this is in Figure 4This is illustrated. For points lying in planes such as plane 400, which are orthogonal to the image plane, the difference between the r-coordinates of the points, here points P4, Ps, P6, with fixed Δ θ The offset increases the further apart these points are. These differences are relative to the minimum distance. r 0 are with r 1 or r 2 is designated. For this reason, Δ r They must be determined multiplicatively to avoid mistakenly removing points that lie on the same levels.

[0062] Depending on the application, the (first) threshold Δ could be r but can also be determined using other functions, e.g. as an exponential function or as a constant.

[0063] In Figure 5An exemplary method according to the invention in a preferred embodiment is illustrated as a flowchart. After a start 500, a set 503 of points (or, more generally, groups of points) is provided in step 502. The input for the described algorithm includes, in addition to the set of points or a point cloud, LP , which was detected or recorded by a lidar (or other) sensor, also constitutes an extrinsic transformation T CL into a different coordinate system with a different origin (e.g., that of a camera). The algorithm then aims to modify the point cloud so that, if the point cloud were viewed from a different standpoint or viewing point, points that would lie behind objects in the original scene would be removed.

[0064] In step 504, the set of points or the point cloud is then calculated. LP with the help of the transformation T CLfrom the original or lidar sensor coordinate system to the target coordinate system of interest, thereby transforming the point cloud CP This results in spherical coordinates, i.e., for every point there are coordinates of the form P̂ = ( r, θ, φ ) . The transformation typically does not directly yield spherical coordinates; the conversion to spherical coordinates may need to be performed separately. This set of points in spherical coordinates can also be referred to as... C P̂ be designated.

[0065] In step 506, a spherically curved grid is determined in spherical coordinates, as in relation to Figure 3 explained. In particular, a so-called " r The "r buffer" can be initialized in which (later) the point with the minimum distance can be stored for each cell. For each cell, this value of the "r buffer" can initially be initialized to, for example, infinity.

[0066] In step 508, each point is assigned to a cell. At the same time, the point with the minimum distance from the origin of the target coordinate system for that cell can also be determined.

[0067] This can in a kind of loop, so that, for example, in step 510 it is always checked whether the point for which an assignment to a cell has just been made is the last one in the set C P̂ was. If not, a grid index is calculated in step 512. (i,j), As mentioned above, the cell to which the point belongs is determined. Therefore, for each point, the corresponding grid index can be calculated based on its spherical coordinates. The grid index corresponding to the point (e.g., denoted by a point index) is stored for later use.

[0068] In step 514, it can then be checked whether the value of the "r-buffer" of this cell is greater than the value of the r-coordinate of the current point. If so, this means that the current point has a smaller distance than another point assigned to this cell. Then, in step 516, the value of the "r-buffer" of this cell is replaced with the value of the r-coordinate of the current point.

[0069] In this way, all points are assigned to a cell, and for each cell, the point with the minimum distance, or at least the value of this minimum distance, is determined.

[0070] If, during step 510 of the loop, it is determined that the point for which an assignment to a cell has just been made is the last one in the set C P̂ In this process, for all cells to which more than one point is assigned, specific points are identified that can be removed from the set.

[0071] This too can be done in a kind of loop, so that, for example, in step 518 it is always checked whether the point for which a check has just been carried out is the last one in the set. C P̂ was. If not, in step 520, for example, the grid index of the current point is checked, i.e., it is checked which cell it is assigned to. In step 522, for example, the value of the "r" buffer of this cell can then be loaded (this was determined in the previous loop). This value corresponds to the minimum distance. r 0 .

[0072] In step 524, it is then checked whether the distance, i.e., the value of the r-coordinate of the current point, is less than or equal to the minimum distance plus the mentioned (first) threshold, i.e., whether the following holds: r P ^ ≤ r 0 + Δ r

[0073] If the Thus, in step 526, the point becomes a target set C P̃assigned. If this is not the case, nothing will be done; the point in question will not be considered further.

[0074] If, during the loop in step 518, it is determined that the point for which a check has just been performed is the last one in the set C P̂ was, the resulting target set C P̃ or 529 can be output or provided in step 528, the procedure can end in step 530.

[0075] It should be mentioned that instead of checking in step 524 whether the distance, i.e., the value of the r-coordinate of the current point, is less than or equal to the minimum distance plus the mentioned (first) threshold, it can also be checked whether the value of the r-coordinate of the current point is greater than the minimum distance plus the mentioned (first) threshold, i.e., whether: r P ^ > r 0 + Δ r

[0076] In this case, if that is the case, the point would, in an alternative step 526, be taken from the set C P̂ removed. If this is not the case, nothing is done; the item in question remains in the quantity. C P̂. The remaining amount C P̂ In an alternative step, 528 can then be used as the target set. C P̃ be issued or made available.

[0077] Furthermore, it is preferred if, in step 532, environmental information 533, such as image information or images, with coordinates in the target coordinate system, is provided. In step 534, the points of the target set that are visible from a given viewpoint in the environment can then be associated with the environmental information.

[0078] Furthermore, in a preferred embodiment of a method according to the invention, the parameters Δ required for the grid can also be determined. φ and Δ θ(Distances) automatically from the given set of points or the point cloud LP can be derived or determined. For example, it can be assumed that the point cloud originates from a typical laser scanner (Lidar sensor) that exhibits characteristic rings around the vertical axis.

[0079] For this purpose, in step 540, coordinates of the set of points, i.e. points in LP , determined in spherical coordinates in the original coordinate system if these are not already available.

[0080] In step 542, a frequency distribution of the points in the polar direction is then determined, i.e., a θ -Histogram created. θThe coordinates of all points are first binned with a certain accuracy, e.g., 216 bins. The histogram (frequency distribution) then contains peaks of approximately the same height for each laser ring. These peaks can be counted by first applying a threshold to the histogram, for example, to remove noise, and then finding and counting local maxima.

[0081] In step 544, for example, a dimension (angle) of the cells in the polar direction is determined based on the number of local maxima of the frequency distribution. The value for the distance, Δ θ , can according to Δ θ = 180 ° N to be determined, where N denotes the number of local maxima of the frequency distribution. In step 546, for example, a dimension (angle) of the cells in the azimuth direction is determined based on a number of point groups per local maximum. The value for the distance, Δ φ,can be determined directly from the height of a peak, since this height corresponds to the number of points with a specific value of θ This corresponds to the following. In practice, not all measurements result in a point in the point cloud, and therefore Δ φ preferably determined based on the maximum number of points in a single laser ring, e.g. according to Δ φ = 360 ° M where M represents the number of points per local maximum of the frequency distribution. In practice, multiplying both parameters by a factor of 2 (or a factor greater than 2) leads to more robust results. This essentially ensures that visible and hidden points end up in similar cells, rather than each receiving its own cell and not being filtered. It goes without saying that other values ​​or factors can also be used for multiplication. Furthermore, these distances can be adjusted or specified based on the distance of the given observation point from an origin of the origin coordinate system.

[0082] In Figure 6 The following are exemplary representations of before and after the application of a proposed method. This is shown on the left (View A). Figure 6A ) an object contour 652 is drawn, namely from the viewpoint of the device's camera, e.g. along the x' axis according to Figure 2 Unlike in Figure 2 The lidar sensor should be positioned slightly higher than the camera, so that it can see over the object more effectively.

[0083] Here, dots (spheres) represent points lying on the object, while crosses represent points lying behind the object. Both types of points are detected by the lidar sensor; the different representations are used here for illustrative purposes only. It can be seen that when the lidar sensor points are transformed into the camera's coordinate system (as in...) Figure 6A ) Points that are actually behind the object are projected onto the object.

[0084] Right (View B or Figure 6BThe object contour 652 is also shown, but certain points have been removed according to a preferred embodiment of a method according to the invention. It can be seen that almost all points that lie behind the object and were previously mapped onto the object are now no longer present. It should be noted that the object contour 652 is shown here only for illustrative purposes; the algorithm used to remove the points does not have access to it.

Claims

1. Method for determining a target set of point groups in an environment (150), which are visible or not visible from a predefined viewing point (214) in the environment (150), from a set of point groups, wherein a point group comprises one or more points (P, A, B), for navigation for a vehicle moving in an at least partially automated manner, comprising: providing (502) the set (503) of point groups in the environment, with coordinates in an origin coordinate system (200), transforming (504) the coordinates of the set (503) of point groups into spherical coordinates in a target coordinate system (300), wherein the viewing point (214) in the environment is at the origin (302) of the target coordinate system (300), determining (506) a spherically curved grid (310) in spherical coordinates in the target coordinate system (300), wherein the grid (310) comprises grid points and / or cells (308), assigning (508) each point group of at least one portion of the set to a cell (308) or a grid point using the spherical coordinates, for each of at least one portion of cells (308) of the grid or for each of at least one portion of grid points of the grid that have been assigned more than one point group: determining a minimum (r0) and / or maximum spacing based on spacings of the point groups from the origin (302) of the target coordinate system (300), a) determining a target set (529) of point groups, which are visible from a predefined viewing point (214) in the environment, comprising those point groups whose spacing from the origin of the target coordinate system per cell or grid point exceeds the minimum (r0) spacing assigned to the cell or grid point at most by a predefined first threshold value (Δr) and / or whose spacing from the origin of the target coordinate system per cell or grid point falls below the maximum spacing minus a predefined second threshold value, and in particular also comprising those point groups which are assigned cells or grid points which are assigned only one point group, and / or b) determining a target set of point groups, which are not visible from a predefined viewing point (302) in the environment, comprising those point groups whose spacing from the origin of the target coordinate system (300) per cell or grid point exceeds the minimum (r0) spacing by more than the predefined first threshold value (Δr), and / or whose spacing from the origin of the target coordinate system per cell or grid point falls below the maximum spacing minus the predefined second threshold value, and providing (528) the target set (529) or the target sets of point groups, and in particular use for navigating and / or moving the vehicle moving in an at least partially automated manner.

2. Method according to Claim 1, wherein the respective predefined first (Δr) and / or second threshold value depends on the respective minimum (r0) or maximum spacing, in particular corresponds to the respective minimum or maximum spacing multiplied by a respective factor.

3. Method according to Claim 1 or 2, wherein the determination (506) of the grid (310) in spherical coordinates in the target coordinate system comprises: determining (540) or using the coordinates of the set of point groups in the origin coordinate system in spherical coordinates, determining (542) a frequency distribution of the point groups in the polar direction, determining (544) a dimension (Δθ) of the cells or spacings of grid points in the polar direction based on a number of local maxima of the frequency distribution, and determining (546) a dimension (Δφ) of cells or the spacings of the grid points in the azimuth direction based on a number of point groups per local maximum.

4. Method according to one of the preceding claims, wherein a dimension (Δθ) of the cells (308) or spacings of the grid points in the polar direction and / or a dimension (Δφ) of the cells (308) or spacings of the grid points in the azimuth direction is / are determined based on a distance of the predefined viewing point (302) from an origin (202) of the origin coordinate system.

5. Method according to one of the preceding claims, wherein the set of point groups in the environment is determined or has been determined by means of a distance measurement, in particular by means of a lidar sensor (110).

6. Method according to Claim 5, wherein a starting point of the distance measurement, in particular a position of the lidar sensor, is at an origin (202) of the origin coordinate system (200).

7. Method according to one of the preceding claims, furthermore comprising: providing (532) environmental information (533), with coordinates in the target coordinate system, and associating (534) at least one portion of the point groups in the target set (529) of point groups, which are visible from a predefined viewing point in the environment, with the environmental information (533).

8. Method according to Claim 7, wherein the environmental information (533) comprises image information and / or semantic information, and in particular is or has been determined by means of an image sensor (120).

9. Method according to Claim 8, wherein the environmental information (533) is presented from the point of view of the predefined viewing point (214), and wherein in particular a position of the image sensor is at the origin (302) of the target coordinate system.

10. Method according to one of Claims 7 to 9, furthermore comprising: determining control information for moving a device (100) based on the environmental information (533) and the associated point groups in the target set (529), and providing the control information and, in particular, controlling the device (100) based on the control information.

11. Data processing system (106, 130) comprising means for carrying out the method according to one of the preceding claims.

12. Device (100), which is configured to provide a set (503) of point groups in an environment, with coordinates in an origin coordinate system, and preferably to provide environmental information (533) with coordinates in a target coordinate system, wherein the device is further configured to obtain a target set (529) of point groups, which are visible or not visible from a predefined viewing point in the environment, which has been determined from the set of points according to a method according to one of Claims 1 to 10, and to use it in particular for navigation, preferably having a distance measurement or lidar sensor (110) for capturing the point groups in the environment, further preferably having a sensor, in particular an image sensor (120), for capturing environmental information, further preferably having a control and / or regulating unit (104) and a drive unit (102) for moving the device (100), and further preferably having a system (106) according to Claim 11.

13. Device (100) according to Claim 12, which is designed as a vehicle moving in an at least partially automated manner, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. a robotic vacuum cleaner and / or a robotic mop, floor or road cleaning device or robotic lawnmower, and / or as a drone.

14. Computer program comprising instructions which, when the program is executed by a computer, cause the latter to carry out the method steps of a method according to one of Claims 1 to 10 when it is executed on the computing unit.

15. Computer-readable storage medium on which the computer program according to Claim 14 is stored.