Object detection with distance measurement

Abstract

The invention relates to a method, in particular a computer-implemented method, for detecting objects (O) on the basis of environmental data acquired by means of an environment sensing sensor unit (1) and for determining a distance (d) of the detected objects (O) from the environment sensing sensor unit (1), the procedure comprises the following steps: - Receiving image data (B(U,O)) concerning an environment (U) of the environment detection sensor unit (U) in which at least one object (O) is located, - Detecting at least one object (O) in the vicinity (U) of the environment sensing sensor unit (1); - Receiving point data (P(U,O)) in the form of a point cloud, in particular a three-dimensional one, relating to the environment (U) of the environment detection sensor unit (1) in which the at least one object (O) is located, - Determining a superposition (SP(B,P)) of the point data (P(U,O)) and image data (B(U,O)), - Determining an object subset (OT(P,O)) of the point data (P(U,O)), which object subset (OT(P,O)) contains point data relating to the object (O), using the superposition (SP(B,P)); - Determining the distance (d) of the object (O) using the object subset (SP(B,P)), and - Output of the object (O) and the distance (d) of the object (O) from the environment detection sensor unit (1).

Description

[0001] The invention relates to a method, in particular a computer-implemented method, for detecting objects based on environmental data acquired by means of an environment sensing sensor unit and for determining a distance of the detected objects from the environment sensing sensor unit, a training data set comprising point data of an environment sensing sensor unit and associated object classes and / or poses, a computer program and a computer program product.

[0002] For object detection and / or object classification, input data from sensors, such as data containing information about the objects to be detected and classified, is typically examined for the presence of various predefined features, and these features are then assigned to specific object classes. Many different methods, and in particular numerous approaches from the field of machine learning, have become known for object recognition in this context. The assignment to an object class is performed, for example, using an activation function that defines specific value ranges for the detected features.

[0003] For example, object detection and / or classification is often performed in connection with advanced driver assistance systems (ADAS), where neural networks are used to process input data from various sensors or environmental sensors, such as radar, lidar, or ultrasonic sensors, as well as cameras. ADAS functions typically provided by these systems are often based on prior object classifications and can either support the driver while maintaining control of the vehicle, or, depending on the level of automation, enable fully automated driving.Examples of ADAS functions include various methods for detecting objects or obstacles on the road, methods for detecting lane boundaries and / or keeping the vehicle in a lane, methods for detecting rain on the windshield, or methods for assisting or carrying out a parking maneuver.

[0004] To determine objects and their coordinates in input data, various methods based on point cloud processing have become known, depending on the environmental sensors used. Examples include the methods described in "Real-Aug: Realistic Scene Synthesis for LiDAR Augmentation in 3D Object Detection" from 2023 by J. Zhan et al., available on arXiv, doi: arXiv:2305.12853, in "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space" from 2017 by CR Qi et al., available on arXiv, doi: arXiv:1706.02413, or in "VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection" from 2017 by Y. Zhou et al., also available on arXiv, doi: arXiv:1711.06396.

[0005] Object detectors based on point cloud input data often struggle with decreasing data density at long distances. This leads to erroneous or incorrect object detections or classifications, especially at long distances. Furthermore, spatial annotation of training datasets for these object detectors is very time-consuming and complex. In the long distance, annotation is sometimes impossible due to insufficient structural information. Moreover, methods based on point cloud input data are not robust, resulting, among other things, from the adaptation of sensor-specific scan and noise patterns during the training phase and a subsequent reduction in the generalization performance of the respective object detector.

[0006] Furthermore, methods combining input data from LiDAR sensors and cameras have become known, such as the method described in "SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection" by H. Zhang et al. from 2024, available on arXiv, doi: arXiv:2403.07284. When using grid-based features for object detection, however, a comparably high storage requirement arises, such that a trade-off typically has to be made between the desired range and the level of discretization.

[0007] Possibilities for 3D spatially oriented object detection based on 2D image-based detections were revealed, for example, in "Frustum VoxNet for 3D object detection from RGB-D or Depth images", 2019, by X. Shen, available on arXiv, doi: arXiv:1910.05483, or in "Frustum PointNets for 3D Object Detection from RGB-D Dat", 2017, by CR Qi, available on arXiv, doi: arXiv:1711.08488v2. Here, too, problems arise with detection accuracy and generalization performance, especially at long distances, comparable to approaches based purely on point clouds.

[0008] Further possibilities for 3D object recognition have been proposed, for example, by A. Mousavian et al. in “3D Bounding Box Estimation Using Deep Learning and Geometry” from 2017, available on arXiv, doi: arXiv:1612.00496v2, by J.-U. Kim et al. in “A New 3D Object Pose Detection Method Using LIDAR Shape Set” in Sensors 2018, 18(3), 882, and by M. Fürst et al. in “HPERL: 3D Human Pose Estimation from RGB and LiDAR”, available on arXiv, doi: arXiv:2010.08221.

[0009] Based on the problems that arise from the aforementioned methods, the present invention aims to provide improved object detection with output of spatial information, which functions reliably, particularly in a long range, especially at distances between the objects and an environmental detection sensor of 100m or more.

[0010] This problem is solved by the methods according to claim 1, the training data set according to claim 13, and by the computer program and the computer program product according to claim 14.

[0011] With regard to the method, the problem underlying the invention is solved by a method, in particular a computer-implemented method, for detecting objects based on environmental data acquired by means of an environmental sensing sensor unit and for determining the distance of the detected objects from the environmental sensing sensor unit. The method comprises the following process steps: - Receiving image data concerning an environment of the environmental sensing sensor unit in which at least one object is located, - Detecting at least one object in the vicinity of the environmental sensing sensor unit; - Receiving point data in the form of a point cloud, in particular a three-dimensional one, relating to the environment of the environmental detection sensor unit in which at least one object is located, - Determining a superposition of the point data and image data, - Determining an object subset of the point data, which object subset contains point data relating to the object, based on superposition; - Determining the distance of the object based on the object subset, and - Output of the object and the object's distance from the environment detection sensor unit.

[0012] The method according to the invention is based on the processing, in particular fusion, of image data and point data to detect objects and determine their distance from the respective environmental sensing sensor unit. In this way, reliable 3D localization of objects can be achieved. In particular, the proposed method advantageously exhibits high accuracy even at a distance relative to the environmental sensing sensor unit.

[0013] The environmental sensing sensor unit comprises at least one environmental sensing sensor. In particular, it comprises one or more sensors for generating image data and point data relating to the sensor's environment. For example, one or more ADAS sensors, in particular a camera, a LiDAR sensor, a radar sensor, or an ultrasonic sensor, may be used.

[0014] In this context, it is advantageous if the environmental sensing sensor unit comprises at least two sensors, wherein, in particular, a first sensor is a camera and a second sensor is a LiDAR sensor or a radar sensor. In this case, the camera serves to acquire sensor data in the form of image data for carrying out the method according to the invention, and the LiDAR sensor or the radar sensor serves to acquire the point data, in particular in the form of a three-dimensional point cloud.

[0015] It is further advantageous if parameters, for example intrinsic and / or extrinsic parameters, of the sensor(s) used are known and can be used within the framework of the method according to the invention.

[0016] The detection of at least one object in the vicinity of the environmental sensing sensor unit can be carried out using many different methods, which are known from the prior art.

[0017] It is advantageous if at least one object is detected using an object detector. This object detector can be based on a machine learning method, in particular an object detector using at least one neural network.

[0018] Preferably, a bounding box is also determined and, if necessary, specified for the object. It is then also advantageous if the object subset of the point data, which contains point data relating to the object, is also determined based on the bounding box.

[0019] According to an advantageous embodiment of the method, determining the superposition of the point data and image data involves projecting the point data onto the image data. This projection can be performed, for example, using intrinsic and / or extrinsic parameters and by comparing the detection ranges of the respective sensors used. The projection can, in particular, include a coordinate transformation into an image plane of the image data.

[0020] According to a further advantageous embodiment of the method, determining the object subset involves ascertaining a frequency distribution of the point data in the superposition as a function of a position within the superposition. The frequency distribution can be determined, for example, as a 1D or 2D representation. Preferably, the frequency distribution is determined using a histogram, in particular with equidistant bins.

[0021] In this context, it is advantageous to determine a local and / or global maximum of the frequency distribution and assign at least one object to it. The presence of an object is typically associated with a higher point data density and can therefore, according to a favorable design, be determined based on the extrema or maxima of the frequency distribution.

[0022] Another, particularly preferred embodiment of the method according to the invention involves determining the height of the object based on the object subset. In this way, not only the object's position in the plane but also its three-dimensional extent can be determined.

[0023] Another preferred embodiment of the method involves using the object, its distance from the environmental sensing sensor unit, and, in particular, its height to perform obstacle detection for a vehicle. This obstacle detection can, for example, include detecting obstacles within a vehicle's driving area where the environmental sensing sensor unit is located, or along predefined object hypotheses. This is particularly advantageous for a movable environmental sensing sensor unit.

[0024] Furthermore, determining the object's orientation is advantageous for the proposed method. Knowing the object's position and orientation allows for reliable pose determination. In particular, a three-dimensional boundary for the object can be determined and provided. The orientation preferably includes determining the roll, pitch, and yaw angles. The object's orientation can also be used for obstacle detection by a vehicle.

[0025] In this context, it is advantageous to define a multitude of virtual projection lines within a predefined angular interval, for example, an angular interval of 180°, specifying which projection lines are aligned parallel to a longitudinal direction of the superposition. This corresponds to a bird's-eye view. Furthermore, the projection lines are provided with a grid; if histograms are used, the grid can be defined by equidistant classes.

[0026] Furthermore, the object subset is projected onto each of the projection lines, and it is determined for which projection line the most points of the object subset lie in a single grid segment. The object's orientation is then determined based on the projection line for which the most points of the object subset lie in a single grid segment. The projection line for which the most points of the object subset lie in a single grid segment typically corresponds to a longitudinal or transverse axis of the respective object.

[0027] Furthermore, it is advantageous to determine the object's orientation using a neural network. This neural network should be designed to determine the object's orientation based on a subset of the object and / or at least a subset of the image data, particularly a subset of the image data that corresponds to a bounding box for the object. The neural network can be trained, for example, using a supervised, semi-supervised, or unsupervised learning process. Using the image data, especially a subset of it, can offer advantages in cases of low data density for a three-dimensional environment around the respective environmental sensing sensor unit, as a pose can be determined even with otherwise insufficient data density.

[0028] A further embodiment of the proposed method involves providing each data point of the point data, which corresponds to an object, with the label containing an object class of the object as well as a pose for the object in three-dimensional space. According to the invention, a three-dimensional indication or a three-dimensional label for the object is thus provided, which in particular contains a three-dimensional bounding box, a distance, and an object class for the object.

[0029] The problem underlying the invention is further solved by a training data set comprising point data and / or image data from an environmental sensing sensor unit and associated indications, wherein the indications include at least one object class for each contained object and an associated pose of the object, and wherein the at least one object class and / or the associated pose is determined by a method according to one of the described embodiments. The method according to the invention is therefore also suitable for generating training data sets comprising training data and three-dimensional indications or labels.

[0030] Overall, the inventive method allows for the determination and specification of three-dimensional bounding frames for image data that originally only contain two-dimensional bounding frames for objects. This enables the additional determination of the orientation of the objects under consideration and their distance from the sensor used to generate the image data. In particular, this is reliably possible even at a distance from the sensor. This also applies to instance-segmented image data. For such image data, object class information is available at a higher resolution. Furthermore, any image data containing additional information about the objects within the image data can be used to carry out the inventive method.

[0031] The problem underlying the invention is ultimately also solved by a computer program with instructions which, when the computer program is executed by a computer, cause the computer to execute the inventive method according to one of the described embodiments, and by a computer program product on which the inventive computer program is stored.

[0032] The inventive method according to one of the described embodiments is preferably used for a driver assistance system or for a vehicle that is at least partially autonomous.

[0033] In summary, the present invention offers several advantages: - The detection range for detecting objects O and their distance d from the environmental sensing sensor unit 1 can be significantly improved or increased compared to other methods. In particular, according to the invention, objects O can also be reliably detected at a distance. - Annotation of the training data used can be advantageously limited to a two-dimensional case. The method also offers a high level of abstraction and generalization performance, meaning it provides high reliability in object detection for environmental sensors other than those used to acquire the training dataset. One reason for this is that sensor-specific scan and / or noise patterns are adapted during the learning process, thus not limiting the generalization performance. - It is possible to distinguish precisely between structures in the foreground and background. - In addition, the method is characterized by high robustness, physical interpretability, especially with regard to the design for determining a complete pose, and a comparatively manageable storage requirement.

[0034] It should be noted that the configuration described in connection with the method according to the invention is also applicable mutatis mutandis to the training data set, the computer program and the computer program product according to the invention and vice versa.

[0035] The invention and its advantageous embodiments are explained in more detail with reference to the following figures. They show: Fig. 1 a block diagram relating to a first preferred embodiment of the method according to the invention; Fig. 2 the determination of a superposition and an object subset within the framework of the inventive method; Fig. 3 an exemplary frequency distribution of point data in the presence of an object in the detection range of the sensor used to detect the point data; Fig. 4. Determining the orientation of the object within the framework of the present procedure; and Fig. 5 exemplary image data together with a three-dimensional boundary frame belonging to an object, which has been determined using the method according to the invention.

[0036] In the figures, the same elements are always labelled with the same reference symbol.

[0037] Fig. Figure 1 illustrates an advantageous embodiment of the inventive method for object detection with distance determination. For the following description, an environmental sensing sensor unit 1 comprising a camera 1a and a LiDAR sensor 2 is used as an example. However, it should be noted that other environmental sensing sensor units 1 can also be used within the scope of the present invention to acquire environmental data in the form of image data B and point data P.

[0038] First, based on received image data B(U,O) concerning an environment U of the environmental sensing sensor unit, which can be located, for example, on a vehicle, in which at least one object O to be detected is located, the at least one object O is detected based on the image data B(U,O). The object detection can be carried out, for example, using a suitable object detector.

[0039] In addition to the image data B(U,O), point data P(U,O) received from the environmental sensing sensor unit 1, relating to the environment U, are used to determine or create a superposition SP(B,P) of the point data P(U,O) and image data B(U,O). From this superposition, an object subset OT(P,O) is then determined, containing the point data P(O) relating to object O. This object subset OT(P,O) is then used to determine the distance d of object O from the environmental sensing sensor unit 1, which is ultimately output along with the object.

[0040] Optionally, at least based on the object subset OT(P,O), the height h of the object O and / or orientation or of the object O can also be determined and possibly output, as indicated by the dashed lines.

[0041] The following, in particular based on Fig. 2. Individual steps of the method according to the invention are explained in more detail using an exemplary possible application: An automatic detection of road users for an environment detection sensor system 1, as described in connection with Fig. 1, described for use as reference data.

[0042] The method according to the invention enables a significant increase in object detection performance in the long range.

[0043] Based on the received image data B(U,O), as already explained, at least one object O is detected and a two-dimensional bounding box is provided as a two-dimensional object hypothesis for the object, cf. Fig. 2a. The point data P(U,O), which also received and which correspond to the same detection range e as that of camera 1b, are in Fig. Figure 2b is shown. In the representation shown here, a distinction is made between object points p and surrounding points u.

[0044] Based on the point data P(U,O) and image data B(U,O), a superposition SP(B,P) is determined according to the invention, as shown in Fig. Figure 2c illustrates this. According to an advantageous embodiment of the proposed method, determining this superposition SP(B,P) can involve projecting the point data P(U,O) onto the image data B(U,O). This projection can be performed by transforming the coordinates into the coordinate system of sensor 1b, which acquires the image data B(U,O).

[0045] The object subset OT(P,O) can in turn be determined, for example, by appropriately evaluating a frequency distribution of the point data P(U,O) in the superposition SP(B,P) as a function of a position (x) within the superposition SP(B,P).

[0046] To do this, the boundary frame BR of the superposition SP(B,P) can first be superimposed and used to determine those point data P(O, BR) which lie within the boundary frame BR, cf. the top view of the point data P(O, BR) in Fig. 2d. For the point data P(O, BR) thus determined, a frequency distribution H can then be calculated, for example in the form of a histogram with equidistant classes b (bin), where different information about the calculated point data P(U,O) can be stored for each class b, such as the number of calculated points, a running longitudinal mean, a running vertical mean, a maximum height, or other parameters. The size and number of classes b can be determined by a global optimization per object class for an object O. The calculation of such a frequency distribution H, here in the form of a histogram, is described in Fig. 2e illustrated.

[0047] To locate the object O based on the determined point data P(O, BR), for example a local and / or global maximum of the frequency distribution HH can be used. max The points are determined and assigned to at least one object O. Class b of the frequency distribution H with maximum point density represents a position on one side of the object O facing the environmental sensor unit 1. The point data within this and neighboring classes are therefore assigned to object O and form the object subset OT(P,U), while all other points are assigned to a background and possibly discarded.

[0048] The assignment of maxima in frequency distributions H of point data P(U,O) to specific objects O is further explained in Fig. 3 illustrated. Fig. 3a shows a distribution of the point data P(U,O) as a function of a distance y from a point in Fig. 3b shown environmental sensing sensor unit 1. Fig. Figure 3b shows a two-dimensional representation of the environment of the environmental sensing sensor unit 1 and an object O in the form of a truck located in the environment U. The density of the point data P(U,O) in the area of ​​object O is significantly higher than in other areas of the environment where no objects O are located. There is a concentration at the object edge, which is also reflected in the corresponding frequency distribution of the point data P(U,O), as shown in Fig. 3a is shown, is visible.

[0049] Based on the point data P(O, BR) of the object subset OT(P,U) assigned to object O, the distance d and, if necessary, the height h of object O can finally be determined.

[0050] Furthermore, it is possible to determine the orientation or of object O. In this way, a complete pose of object O can be determined and output. This is explained in more detail in connection with Fig. 4.

[0051] The orientation or of an object O can be described, among other things, by a yaw angle. The yaw angle can, in turn, be determined based on the object subset OT(P,O). For this purpose, the object subset OT(P,O) is viewed in a top view. Similarly, the orientation or can also be described by a roll angle roll – in this case, the object subset OT(P,O) is viewed in a front view – and / or by a pitch angle pitch – in which case the object subset OT(P,O) is viewed in a side view. The other angles roll and pitch can be determined analogously.

[0052] Assuming that x describes a lateral direction from left to right, y a longitudinal direction (with the environment detection sensor unit 1 arranged in the direction of travel) and z a vertical direction from bottom to top, then for various projection lines corresponding to angle α in a predefinable interval (0° ≤ α ≤ 179°) and all points (x,y,z) of the point data of the object subset OT(P,O) and for the corresponding projections x yaw , x pitch and x roll : xyaw=x cos α+y sin α xpitch=y cos α+z sin α xroll=−x cos α+z sin α

[0053] The projections x yaw , x pitch and x roll can then be accessed via three histograms (one for each of the angles) H yaw , H pitch and H roll (or another suitable frequency distribution) and, for example, H can be distributed for each of the histograms. yaw , H pitch and Hroll A number of points per class b will be stored.

[0054] Subsequently, class b with the maximum number of points or the highest point density is determined in each case. Thus, a maximum of the histograms H is identified in each case. yaw , H pitch and H roll determined and assigned to the respective angles: dyaw=max(Hyaw)N,dpitch=max(Hpitch)N,droll=max(Hroll)N, where N is the total number of all points. The principal orientations then result from the angles with the highest associated point density: yaw=argmax(dyaw);pitch=argmax(dpitch);roll=argmax(droll)

[0055] From this, a dimension of the object can also be determined, in particular a length L, width B and height H: (L,W,H)=(max(y')−min(y'),max(x')−min(x'),max(z')−min(z')), where: (x',y',z')=Rx(pitch)∗Ry(roll)∗Rz(yaw)∗(x,y,z), and R x , R y, and R z describe rotations around the x, y and z axes.

[0056] To determine the orientation or of a given object O, the object class assigned to that object O can also be considered. Specifically, the point densities and angles of a primary orientation yaw and an orthogonal orientation can be examined, and the ratio of these point densities can be calculated. Then, for example, a characteristic shape (e.g., an L-shape for objects in the object class "vehicles" or "cars") is determined, and by comparing this characteristic shape with the calculated ratio of point densities, the three-dimensional position of object O can be estimated. This yields the object's orientation or. This approach can be advantageously used to determine the orientation or of a given object O even with low data densities.

[0057] To determine the orientation or of the object O, in principle a large number of virtual projection lines p are specified in a predefinable angular interval, which projection lines p are in particular aligned parallel to a longitudinal direction of the superposition SP(B,P), and which projection lines p are each provided with an equidistant grid.

[0058] In Fig. Figure 4a shows, as an example for a first projection line p0 corresponding to an angle α=0°, the associated frequency distribution H of the point data P(U,O) projected onto this projection axis p0 for eight equidistant classes. The point data P(U,O) are uniformly distributed along this projection line. Therefore, the maximum density of point data for the projection axis p0 corresponding to α=0° is one for the simplified example shown.

[0059] Another exemplary projection line considered p 69This corresponds to an angle α=69° and is in Fig. 4b is shown. For the projection line p onto this projection line 69 For projected point data P(U,O), the projection of all displayed point data P(U,O) lies at the same grid point. The maximum density of point data for the projection axis p 69 Accordingly, α=69° is eight for the simplified example shown.

[0060] Basically, the predefined, virtual projection line p is rotated by an increment, for example by 1°, and the point data P(U,O) are projected onto the respective projection axis p. A maximum point density per raster element is then determined. In other words, the object subset OT(P,O) is projected onto each of the projection lines p, and the projection line is identified for which the most points of the object subset OT(P,O) lie in a single raster segment.

[0061] The resulting list assigns the rotation angles yaw, pitch, and roll, the maximum point data densities per raster element or class, and a primary orientation for the object can then be determined by identifying the angle with the highest point data density, as described previously. This procedure can be performed for all three different two-dimensional projections: a lateral-longitudinal projection (corresponding to the yaw angle), a projection in the longitudinal-vertical plane (pitch angle), and a projection in the lateral-vertical plane (roll angle).

[0062] The information obtained according to the invention about the object O, in particular its object class, a distance d, height h and orientation or, can then be suitably output; for example, the individual image data B(U,O) can be displayed together with a three-dimensional bounding frame BB. 3D be issued as in Fig. 5 shown.

[0063] The respective output parameters, i.e., the detected object O or the associated object class and / or the boundary frame, the determined distance d, the height h and / or possibly also the determined orientation or, can preferably be used to perform obstacle detection for a vehicle, for example within a driver assistance system or in a vehicle that is at least partially autonomous.

[0064] On the one hand, obstacles can be detected along a predefined object hypothesis. For this purpose, the frequency distribution H is used, and a traversability state is assigned to each class. The traversability state can be determined, for example, based on the number of point data assigned to the respective class, the average or maximum height of detected objects O, the respective point data, or other determined parameters.

[0065] A second possibility is to check for the presence of obstacles in the vehicle's operating area. The operating area can, for example, be divided into suitable sections, in which, similar to considering predefined object hypotheses, a specific traversability state is assigned to each section.

[0066] It is also possible to determine the orientation or of object O using a neural network (NN). The neural network is designed to determine the orientation or of object O based on suitable input data. Input data can include, among other things, the object subset OT(P,O), one or more of the described frequency distributions H, or a point density distribution along a height axis h. h, or at least a subset of the image data B(U,O), in particular a subset of the image data B(U,O) that corresponds to a bounding box for the object, can be used. Numerous other quantities are also suitable as input variables for the neural network NN. Furthermore, the neural network NN can also be used to determine the object's position or dimensions. Different features of the object O can be learned together or at least partially separately.

[0067] Furthermore, the use of a neural network (NN) offers a way to create a compact representation with a fixed number of features. The high achievable information density also allows for the use of comparatively less complex or computationally intensive architectures. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection” by H. Zhang et al from 2024, available on arXiv, doi: arXiv:2403.07284

[0006] Frustum VoxNet for 3D object detection from RGB-D or Depth images", 2019, by X. Shen, available on arXiv, doi: arXiv:1910.05483

[0007] Frustum PointNets for 3D Object Detection from RGB-D Dat“, 2017, by CR Qi, available on arXiv, doi: arXiv:1711.08488v2

[0007] A. Mousavian et al. in „3D Bounding Box Estimation Using Deep Learning and Geometry“ aus 2017, verfügbar auf arXiv, doi: arXiv:1612.00496v2

[0008] von J.-U. Kim et al in „A New 3D Object Pose Detection Method Using LIDAR Shape Set“ in Sensors 2018, 18(3), 882

[0008] von M. Fürst et al. in „HPERL: 3D Human Pose Estimation from RGB and LiDAR“, verfügbar auf arXiv, doi: arXiv:2010.08221

[0008]

Claims

[1] Method, in particular a computer-implemented method, for detecting objects (O) based on environmental data acquired by means of an environment sensing sensor unit (1) and for determining a distance (d) of the detected objects (O) from the environment sensing sensor unit (1), wherein the method comprises the following process steps: - Receiving image data (B(U,O)) concerning an environment (U) of the environment detection sensor unit (U) in which at least one object (O) is located, - Detecting at least one object (O) in the vicinity (U) of the environment sensing sensor unit (1); - Receiving point data (P(U,O)) in the form of a point cloud, in particular a three-dimensional one, relating to the environment (U) of the environment detection sensor unit (1) in which the at least one object (O) is located, - Determining a superposition (SP(B,P)) of the point data (P(U,O)) and image data (B(U,O)), - Determining an object subset (OT(P,O)) of the point data (P(U,O)), which object subset (OT(P,O)) contains point data relating to the object (O), using the superposition (SP(B,P)); - Determining the distance (d) of the object (O) using the object subset (SP(B,P)), and - Output of the object (O) and the distance (d) of the object (O) from the environment detection sensor unit (1). [2] Method according to claim 1, wherein the environment sensing sensor unit (1) comprises at least two sensors, in particular wherein a first sensor (1a) is a camera and wherein a second sensor (1b) is a LiDAR sensor or a radar sensor. [3] Method according to claim 1 or 2, wherein the at least one object (O) is detected by means of an object detector. [4] Method according to any of the preceding claims, wherein determining a superposition (SP(B,P)) of the (P(U,O)) and image data (B(U,O)) involves a projection of the point data (P(U,O)) into the image data (B(U,O)). [5] Method according to any of the preceding claims, wherein determining the object subset (OT(P,O)) includes determining a frequency distribution (H) of the point data (P(U,O)) in the superposition (SP(B,P)) as a function of a position within the superposition (SP(B,P)). [6] Method according to claim 5, wherein a local and / or global maximum of the frequency distribution (H) is determined and assigned to at least one object (O). [7] Method according to one of the preceding claims, wherein a height (h) of the object (O) is determined on the basis of the object subset (OT(P,O)). [8] Method according to one of the preceding claims, wherein the object (O), the distance (d) of the object (O) from the environment detection sensor unit (1) and in particular the height (h) of the object (O) are used to perform obstacle detection for a vehicle. [9] Method according to any of the preceding claims, wherein an orientation (or) of the object (O) is determined. [10] Method according to claim 9, wherein a plurality of virtual projection lines (p) are specified in a predefinable angular interval, which projection lines (p) are in particular aligned parallel to a longitudinal direction of the superposition (SP(B,P)), and which projection lines (p) are provided with a grid, where the object subset (OT(P,O)) is projected onto each of the projection lines (p), where it is determined for which projection line (p) most points of the object subset (OT(P,O)) lie in a single raster section, and where the orientation (or) of the object (O) is determined on the basis of the projection line (p) for which most points of the object subset (OT(P,O)) lie in a single grid section. [11] Method according to claim 9 or 10, wherein the orientation (or) of the object (O) is determined by means of a neural network (NN) which neural network (NN) is designed to determine the orientation (or) of the object (O) based on the object subset (OT(P,O)) and / or based on at least a subset of the image data (B(U,O), in particular a subset of the image data (B(U,O)) which corresponds to a bounding frame (BR) for the object (O). [12] Method according to any of the preceding claims, wherein each point of the point data (P(U,O) corresponding to an object (O) is provided with a label, the label containing an object class of the object and a pose for the object (O) in three-dimensional space. [13] Training data set comprising point data (P(U,O) and / or image data (B(U,O)) of an environment sensing sensor unit (1) and associated indications, wherein the indications include at least one object class for each contained object (O) and an associated pose of the object (O), and wherein the at least one object class and / or the associated pose of the object (O) is determined by a method according to any of the preceding claims. [14] Computer program with instructions which, when the computer program is executed by a computer, cause the computer to execute the method according to any one of claims 1-12, or computer program product on which the computer program is stored. [15] Use of the method according to at least one of claims 1-12 for a driver assistance system or for a vehicle that is at least partially autonomous.

Images