Method for object detection and corresponding vehicle

Abstract

Method for detecting objects (12), characterized by the following features: a three-dimensional point cloud (10) is captured using a lidar sensor (11), For each object (12) among the objects (12), a corner point (13) of the object (12) that is closest to the sensor (11) is located in the point cloud (10) and The objects (12) are recognized based on their respective corner points (13).

Description

[0001] The present invention relates to a method for detecting objects. The present invention also relates to a corresponding vehicle, a corresponding computer program, and a corresponding machine-readable storage medium.

[0002] The detection of three-dimensional (3D) objects, particularly by advanced driver assistance systems (ADAS) and autonomous vehicles, is the subject of intensive research. One of the primary techniques used in this area is based on lidar, a remote sensing method that uses light in the form of pulsed laser beams to measure distances. A lidar sensor captures 3D point clouds, with each point characterized by its Cartesian spatial coordinates and the strength of the backscattered signal. Based on these points, which are thus located and described by their reflectivity, the task is to identify objects—whether moving or stationary—that are within the sensor's range, and to do this in real time.

[0003] Recently, artificial intelligence (AI), particularly in the form of deep neural networks, has emerged as a promising approach to solving this problem.

[0004] The VoxelNet architecture for lidar-based object detection is well known to those skilled in the art. It is based on the decomposition of three-dimensional point clouds into a grid of isotropic volume elements (technically known as "voxels"), each of which is assigned a discrete value – typically the reflectance. This so-called voxelization enables effective data reduction and abstraction for subsequent processing. The VoxelNet framework provides for two essential process steps: feature extraction and the generation of object proposals. For feature extraction, three-dimensional convolutional neural networks (CNNs) are used, which are responsible for extracting significant features from the voxels. The latter uses two-dimensional convolutional neural networks to identify possible objects in the observed space and to suggest regions where these objects could be located.VoxelNet is trained on a large amount of lidar data in which the positions and classes of objects have been manually marked.

[0005] Since point clouds captured using VoxelNet, for example, have a high spatial resolution and it can therefore be difficult to determine the precise boundaries of objects or their bounding boxes, center-based approaches to object detection have been proposed. Such methods focus on locating the center of an object rather than outlining its entire perimeter. The voxel to which the located center is assigned is typically evaluated to infer the class of the object—for example, a road user.

[0006] A representative of this method genre is "CenterPoint" and is described in YIN, Tianwei; ZHOU, Xingyi; KRAHENBUHL, Philipp. Center-based 3D Object Detection and Tracking. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021. pp. 11784–11793.

[0007] KR2023072584A discloses a method for predicting the movement of a so-called measurement point on a vehicle using the Kalman filter algorithm. This method involves selecting the vertex closest to the measurement point and inputting the position values ​​of both the selected vertex and the measurement point into the Kalman filter. The resulting prediction includes the velocity, position, and acceleration of the measurement point. The measurement point selection process may also include selecting the vertex closest to a specific sensor.

[0008] EP3644276A1 discloses a method for fitting a three-dimensional bounding box around each point cluster representing an object. The process begins by estimating the center of the point cloud cluster as the center of the 3D bounding box. Subsequently, the extreme positions of the bounding box along the x-, y-, and z-axes are determined. In the final step, the positions of eight 3D vertices are estimated, which are configured such that the bounding rectangle occupies the smallest possible volume and area, resulting in a tight 3D box around the object.

[0009] EP4174611A1 discloses a mobile robot configured to use an object detection neural network (ODNN) on camera and lidar data to detect objects. The ODNN outputs the bounding box position of a detected object in several ways. In one method, the position is specified as a combination of a central point and offsets in height, longitude, and latitude from the central point within the lidar sensor's coordinate system. Alternatively, the position can also be output as a combination of the coordinates of the bounding box's corners within the same coordinate system.

[0010] US20190385457A1 discloses a system for identifying and analyzing obstacles. It extracts feature points from images from a camera or a three-dimensional image generated by a radar sensor. For example, if the obstacle is a truck, the system could identify and extract prominent features such as corners or vertices. Furthermore, the processor can create a bounding box for the detected obstacle based on the three-dimensional image. This process involves identifying the obstacle by compressing the point cloud using MPEG-I standard technology, followed by generating a bounding box enclosing the obstacle.

[0011] WO2019060125A1 discloses a CNN used to determine parameters from a 3D point cloud. To do this, an offset is calculated for each corner of a three-dimensional bounding box with respect to each point in the cloud. The algorithm also determines a confidence value for the offset of each point. The point and associated offset with the highest confidence value are then selected to define the smallest possible cuboid-shaped bounding box.

[0012] WO2021094398A1 discloses another CNN used for image processing tasks, particularly for the construction of bounding boxes. The model projects the corners of the predicted 3D frame onto an image plane and determines its minimum bounding rectangle (MUR). The accuracy of the prediction is assessed by minimizing the distance between this MUR and the original. The CNN considers consistency and geometry losses, as well as losses related to the position, orientation, and size of the object center.

[0013] One problem is that the lidar sensor—unlike a camera, for example—often does not capture information from the center of an object. This is because the reflection of the lidar signal occurs primarily on the surface of the object facing the sensor. Nevertheless, the detection of an object according to the CenterPoint method is based precisely on its center point (which is inadequately detectable by lidar). Thus, an anchor point is ultimately used for the prediction, which may contain little information about the object as a whole.

[0014] Furthermore, it's important to note that the second stage of the CenterPoint model, while faster than other two-stage approaches, still requires time and computational power to detect an object's surfaces. This can be problematic in scenarios where real-time detection and processing are required, such as autonomous driving.

[0015] Against this background, the object of the invention is to create a method for lidar-based object detection which uses the available point cloud information with the greatest possible accuracy and efficiency, taking into account the specific properties of the lidar sensor used to capture it.

[0016] The described problem is solved by a method for detecting objects, a corresponding vehicle, a corresponding computer program and a corresponding machine-readable storage medium according to the independent claims.

[0017] This approach has the advantage of more efficiently utilizing the information captured by the lidar sensor by representing objects based on their vertices rather than their centers. This enables more accurate and realistic object detection, since the vertices are often detected more accurately by the lidar sensor than the object centers. In this way, a method according to the invention allows for the precise detection and classification of objects, which can be particularly beneficial in safety-critical applications such as autonomous driving.

[0018] Furthermore, the proposed method allows for improved object tracking. Since information is processed directly from the vertices of the objects rather than their centers, object tracking can be more efficient and accurate. This could help improve the performance of object tracking systems, especially in dynamic and complex environments.

[0019] Furthermore, the proposed approach can contribute to reducing computational complexity and thus improving processing speed. Since it eliminates the need for a second processing phase to refine the results, as required by the CenterPoint model, the described method enables faster and more efficient object detection.

[0020] Finally, an embodiment of the invention offers greater flexibility in object recognition by being able to detect and process multiple vertices of an object, thus providing a more comprehensive and accurate representation of the object. This capability could help improve the population structure of the representation space, which also benefits the efficiency and accuracy of object recognition.

[0021] Further advantageous embodiments of the invention are specified in the dependent patent claims.

[0022] The figure shows the two-dimensional map view of a point cloud captured by the lidar sensor of a vehicle turning left at an intersection.

[0023] An embodiment will now be explained using an exemplary scene captured by a lidar sensor (11) in the form of a spatial point cloud (10). This point cloud (10) represents all stationary and moving objects within the range of the lidar sensor (11) and is represented—for example, in the context of VoxelNet—by a grid of voxels, each of which carries information about a specific area in the real world obtained from the lidar impulse response.

[0024] To enable effective object detection, this three-dimensional point cloud (10) is first converted into the image-based top view. This can be achieved by projecting the points in the point cloud (10) onto a two-dimensional plane corresponding to the height of the ground or the road surface.

[0025] This top view shows several objects (12) that were detected by the lidar sensor (11). For each of these objects (12), the corner point (13) closest to the lidar sensor (11) is determined. The laser signals emitted by the lidar sensor (11) must travel the shortest distance to the respective object (12) at said corner point (13), so that this point provides the most reliable information about the object (12).

[0026] Once the relevant vertex (13) for an object (12) is located, its properties are regressed onto the associated voxel. In this way, the height, width, length, and orientation of the object relative to the lidar sensor (11) can be derived to gain a comprehensive understanding of each object and effectively detect and track it. It is understood that additional vertices of the objects (12) may be located and included in the analysis to further improve the accuracy of object detection and tracking.

[0027] Finally, each object (12) can be assigned to a specific class of road users based on its derived properties. This classification can help predict the behavior of detected objects and thus improve the safety and efficiency of autonomous driving. List of reference symbols 10 point cloud 11 Lidar sensor (of the ego vehicle) 12 objects 13 Corner point QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] KR 2023072584A

[0007] EP 3644276A1

[0008] EP 4174611A1

[0009] US 20190385457A1

[0010] WO 2019060125A1

[0011] WO 2021094398A1

[0012] Cited non-patent literature

[0000] YIN, Tianwei; ZHOU, Xingyi; KRAHENBUHL, Philipp. Center-based 3D Object Detection and Tracking. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021. S. 11784-11793

[0006]

Claims

[1] Method for detecting objects (12), characterized by following features: - a three-dimensional point cloud (10) is recorded by means of a lidar sensor (11), - for each object (12) among the objects (12), a corner point (13) of the object (12) closest to the sensor (11) is located in the point cloud (10) and - the objects (12) are recognized based on their respective corner points (13). [2] Method according to claim 1, characterized by following features: - the point cloud (10) is first converted into a top view after it has been captured and - the location of the corner points (13) is done based on the top view. [3] Method according to claim 1 or 2, characterized by following feature: - for each object (12) among the objects (12), certain properties of the object (12) are inferred from its vertex (13). [4] Method according to claim 3, characterized byat least one of the following characteristics: - the properties include dimensions of the respective object (12) or - the properties include an orientation of the respective object (12). [5] Method according to claim 3 or 4, characterized by following features: - the vertices (13) are located in a grid of isotropic voxels and - the properties are regressed onto the voxel of the grid to which the respective vertex (13) is assigned. [6] Method according to claim 5, characterized by following features: - at least one further corner point (13) of the respective object (12) is located and - the regression is carried out on all voxels to which at least one of the corner points (13) is assigned. [7] Method according to one of claims 3 to 6, characterized by following feature: - the respective object (12) is assigned to a specific class of road users based on its properties. [8] Vehicle, characterized by following features: - the vehicle has a lidar sensor (11) and - the vehicle is designed to carry out a method according to one of claims 1 to 7. [9] Computer program which is designed to carry out all the steps of a method according to one of claims 1 to 7. [10] A machine-readable storage medium having stored thereon a computer program according to claim 9.

Images