Method, computer program, device and storage medium for providing three-dimensional feature map of roadway surface

Abstract

The invention relates to a method for providing a three-dimensional feature map of a roadway surface, comprising the following steps: providing (101) a camera image; extracting (102) a feature of the camera image; defining (103) a three-dimensional feature space (3); determining (104) at least three surface hypotheses (6) for the roadway surface (5) in the three-dimensional feature space (3); projecting (105) the extracted features onto the determined surface hypothesis (6); calculating (106) an intersection (8) of the line-of-sight ray (7) with the determined surface hypothesis (6); assigning (107) each extracted feature to each voxel (4); the three-dimensional feature map is provided (108) on the basis of the associated extracted features. The invention further relates to a computer program, a device and a storage medium for this purpose.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method for providing a three-dimensional feature map of a carriageway surface. Furthermore, the invention also relates to a computer program, a device and a storage medium for this purpose. BACKGROUND

[0002] Deep learning based approaches for lane marking and traffic line detection can use an architecture with two or more components. Here, one component forms for example a neural network for feature extraction (“backbone”), while a second component (“detection head”) can be used for the actual recognition and geometric estimation of lane markings and traffic lines from the extracted features. In the simplest case, the features can be directly learned and extracted from the input image in the camera perspective by the backbone, which is then processed by the detection head for detection. This basic architecture is used in particular for 2D line detection, since the learned image feature space is directly represented in the camera perspective, in which the lines to be detected should also be modeled. In particular for 3D line detection, a further component - a so-called inverse perspective mapping (IPM) - can be used. The IPM is in particular a homography-based transformation, by means of which the image features are projected from the plane of the camera perspective onto a flat ground plane. Here, the ground plane should preferably reflect the geometry of the road surface. Since for 3D detection, the traffic lines markings can be exhibited in a 3D coordinate system determined by the course of the road surface, the image features should preferably be exhibited in a feature space that is as similar as possible to the road surface.

[0003] Some approaches make use of the IPM component in order to project the input image onto a flat ground plane. However, most methods for 3D detection make use of the IPM component as an intermediate step between the backbone and the detection head in order to not project the image features directly, but rather the features learned by the backbone onto the ground plane. The detection head then processes these extracted features, which have been geometrically projected onto the ground plane, in order to estimate the 3D geometry of the underlying lines therefrom.

[0004] The limitation of the approach so far is that the inverse perspective mapping only describes the transformation from the image plane in the camera perspective to a flat ground plane. Therefore, it is implicitly assumed that the road surface corresponds to a flat world, thus does not contain height profiles deviating from the zero plane. If this assumption is not true, the feature space is not well suited to represent the scene of the road environment. Thus, for scenes with steep uphill, some features in the input image can no longer be represented, and for scenes with steep downhill, large areas can be contained which do not have features useful for the detection task. However, for the problem of 3D line detection, these height differences are exactly the scenes of interest and important. SUMMARY

[0005] The subject matter of the present invention is a method, a computer program, a device and a machine-readable storage medium for providing a three-dimensional feature map of a road surface. Further features and details of the invention result from the corresponding description and the drawings. Herein, the features and details described in connection with the method according to the invention of course also apply to the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, respectively, so that in the disclosure of the invention always also mutual references can be made.

[0006] The subject matter of the present invention is especially a method for providing a three-dimensional feature map of a road surface, comprising the following steps, wherein the steps can be executed repeatedly and / or sequentially.

[0007] In a first step, preferably a camera image is provided, wherein the camera image comprises a representation of the road surface and is derived from a detection of a camera. The camera image is especially an image of a front camera perspective. In other words, the camera is especially arranged at a front side of the vehicle during the detection of the camera image.

[0008] In a further step, preferably features of the camera image are extracted, wherein the extracted features are specific to the road surface. For this purpose, preferably a machine learning model, especially a backbone of a machine learning model, is used. The machine learning model or the backbone of the machine learning model is for example a convolutional neural network and can be used for feature extraction using its properties. Herein, a two-dimensional feature map can also be generated by further layers, in which each two-dimensional point contains a multi-dimensional feature. The feature map is especially equivalent to the front camera perspective.

[0009] In a further step, preferably a three-dimensional feature space is defined, wherein the three-dimensional feature space has a plurality of voxels. The three-dimensional feature space is present in particular in the shape of a cuboid grid. The three-dimensional feature space is preferably unfolded by x, y and z coordinates and is defined by corresponding intervals. Additionally, a resolution can be determined for each dimension of the three-dimensional feature space, so that a grid in the shape of a cuboid is formed inside the three-dimensional feature space. Each voxel thus formed can be uniquely described by one three-dimensional space coordinate. The number of voxels is defined in particular by the corresponding intervals and the resolution. The voxels are in particular three-dimensional pixels for the presentation of volume data.

[0010] In a further step, preferably at least three surface hypotheses are determined in the three-dimensional feature space for the carriageway surface. The surface hypotheses are in particular planes of different orientations in the three-dimensional feature space, which respectively represent a possible course of the carriageway surface. The surface hypotheses can be arbitrary shaped surface hypotheses, which can advantageously reflect prior knowledge about typical road surface courses or carriageway surface courses. The surface hypotheses can contain inclines and declines and can be arbitrarily defined in terms of their number and orientation, wherein a more precise mapping of the actual carriageway surface can be provided as the number increases.

[0011] In a further step, preferably the extracted features are projected onto the determined surface hypotheses. Within the scope of this step, it can be provided that for each voxel a multi-dimensional feature is uniquely assigned, which represents the corresponding extracted feature of the voxel, wherein each point on the surface hypotheses is associated with a multi-dimensional feature.

[0012] In a further step, preferably the intersection points of the sight lines with the determined surface hypotheses are calculated, wherein the sight lines run from the camera device through the respective center of all voxels of the plurality of voxels. In a subsequent process, preferably only such intersection points are considered which lie inside the three-dimensional feature space.

[0013] In a further step, preferably the individual extracted features are assigned to the individual voxels, wherein the extracted feature assigned to the corresponding voxel is obtained by a weighted sum of the extracted features projected onto the surface hypotheses, wherein the weights of the weighted sum correspond to the distance of the voxel center of the corresponding voxel to the calculated intersection point of the sight line with the surface hypotheses. Here, a large distance leads in particular to a small weight. The weighted extracted features can be added along their sight lines, thus forming a weighted multi-dimensional feature of the belonging voxel. The corresponding weight of the intersection point can advantageously act as a balance between the geometric position of the extracted feature with respect to the belonging voxel and the quality of the feature of the intersection point.

[0014] In a further step, a three-dimensional feature map is preferably provided on the basis of the assigned extracted features. There are different possibilities to provide the three-dimensional feature map. One possibility consists in a simple combination of the features along the height dimension of the three-dimensional feature space. Here, for example, the weighted extracted features can be concatenated or accumulated.

[0015] Another possibility consists in selecting features from the three-dimensional feature space such that the features are taken along the surface run of the carriageway surface. For this purpose, in a further step, a probability value is determined for each voxel inside the corresponding vertical voxel column, preferably on the basis of the assigned extracted features of each voxel, wherein the probability value represents the probability of the carriageway surface run passing through the corresponding voxel.

[0016] The determination of the probability value can be implemented by means of a machine learning model which is based on a training by means of an implicit learning method (English: "self-supervised") or also by means of an explicit learning method (English: "supervised"). For this purpose, each voxel column of the three-dimensional feature space can be regarded as a classification problem. For further processing, preferably for the feature assignment, a single best voxel is classified in each voxel column, wherein the height dimension of the three-dimensional feature space can be reduced to a size of 1. By minimizing a 3D detection cost function, the weights of the machine learning model can be adapted in the training process such that the resulting probability distribution reflects the true height profile of the carriageway surface. Thus, the carriageway surface can be implicitly learned. For the calculation of the surface probability, the probability of the carriageway surface run passing through the corresponding voxel can be estimated for each voxel in the voxel column. Thereby, in particular a probability distribution along the height is generated for each voxel column. By applying this classification on each voxel column, in particular a new feature map is formed which has the dimensions of the xy plane of the three-dimensional feature space and which comprises the most valuable features of the three-dimensional feature space corresponding to the surface run. Subsequently, this feature map can be additionally encoded by an encoding block which can comprise a plurality of convolvable neural layers. The classification problem along the voxel column can likewise be optimized by explicit supervised learning (English: "supervised"). For this purpose, preferably an explicit cost function is formulated which is suitable for multiple classification. This cost function forms in particular a part of a total cost function which is minimized by the machine learning model in the training process.

[0017] In a further step, a geometric representation of the road surface can be derived based on the determined probability values. To this end, for example, the voxel with the highest probability value can be selected for each voxel column. Subsequently, a three-dimensional feature map can be provided for the voxels of the geometric representation. Subsequently, the geometric representation can be used for the detection of traffic lane markings, for example by a detection head of a machine learning model. Here, it can also be provided that the extracted features are combined based on the derived probability values.

[0018] It is furthermore conceivable that the method further comprises the following steps:

[0019] - calculating a second distance of each voxel to the nearest one of the at least three surface hypotheses, wherein

[0020] The extracted features for each voxel are further weighted based on the calculated second distances. Within the scope of the present invention, this further weighting can also be referred to as geometric prior weighting. The weight is for example calculated by the second distance, which represents the minimum distance of the voxel along the corresponding view ray to the surface hypothesis, wherein a distance of zero yields the highest weight of 1. Here, the minimum weight is for example the number 0.

[0021] Preferably, it can be provided within the scope of the present invention that the projection is performed using a sampling grid, wherein also bilinear interpolation is used in order to obtain the features for each point on the surface hypothesis. This can have the advantage that the accuracy of the surface construction is improved, since the sampling grid can provide a higher resolution than the actual camera resolution. Furthermore, also the bilinear interpolation can be used to improve the quality of the features obtained for each point on the surface hypothesis. By combining these techniques a more precise and more detailed geometric representation of the road surface can be provided.

[0022] It is also optionally conceivable that the method is applied in a vehicle, wherein the vehicle comprises a camera and the method further comprises the following steps:

[0023] - starting a detection of the camera image,

[0024] - using the provided three-dimensional feature map for at least partially automated driving of the vehicle. Thereby, it can be advantageously possible to determine the road surface from the geometric representation directly in the vehicle, and in turn the subsequent processes, like traffic lane detection, can be performed more precisely in order to improve the at least partially automated driving of the vehicle. Here, the provided three-dimensional feature map can for example be used in a system function of the vehicle, like a lane keeping assistant. Here, the three-dimensional feature map can first be used to better recognize traffic lane markings, which in turn can be used for the environmental perception of the vehicle for at least partially automated driving.

[0025] Furthermore, it can be advantageous within the scope of the application that the origin of the three-dimensional feature space is located in a Cartesian reference coordinate system of the carriageway surface exhibited by the camera image, wherein the vertical voxel column represents the height dimension in the three-dimensional feature space. Accordingly, deviations in the height of the carriageway surface, such as an incline or decline or also a pothole in the carriageway surface, can advantageously be taken into account by the method according to the application. The reference coordinate system of the three-dimensional feature space is in particular a spatial representation of the vehicle environment, wherein the vehicle, preferably the rear axle of the vehicle, describes the origin of the coordinate system. Here, the camera image is in particular exhibited in a further coordinate system having the camera center as the origin and exhibits the carriageway surface. Within the scope of the method according to the application, features from the camera image are preferably transferred into the three-dimensional feature space, but the two coordinate systems are in particular not identical.

[0026] The method according to the application can be used in a vehicle. The vehicle can be configured, for example, as a motor vehicle and / or as a passenger vehicle and / or as an autonomous vehicle. The vehicle can have a vehicle facility, for example, for providing an autonomous driving function and / or have a driving assistance system. The vehicle facility can be implemented for at least partially automatically controlling and / or accelerating and / or braking and / or steering the vehicle.

[0027] Likewise, the subject matter of the application is also a computer program, in particular a computer program product, which comprises instructions which, when the computer program is implemented by a computer, cause the computer to implement the method according to the application. The computer program according to the application therefore has the same advantages as are described in detail with regard to the method according to the application.

[0028] Likewise, the subject matter of the application is also a device for data processing, which is set up to implement the method according to the application. For example, a computer which implements the computer program according to the application can be set up as the device. The computer can have at least one processor for implementing the computer program. A non-volatile data memory can also be provided, in which the computer program can be saved and from which the computer program can be read out by the processor for implementation.

[0029] Likewise, the subject matter of the application can also be a computer-readable storage medium, which has the computer program according to the application and / or comprises instructions which, when the instructions are implemented by a computer, cause the computer to implement the method according to the application. The storage medium is configured, for example, as a data memory, such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can be integrated, for example, in the computer.

[0030] In addition, the method according to the application can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or executed automatically.

[0031] Further advantages, features and details of the present application can be gathered from the subsequent description, in which embodiments of the present application are explained in more detail with reference to the drawings. Herein, the features mentioned in the present application can be of essential significance for the application, respectively, individually or in arbitrary combination. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings show:

[0033] Figure 1 Schematic visualizations of a method, a vehicle, a device, a storage medium and a computer program according to embodiments of the present application,

[0034] Figure 2 Schematic diagram of a three-dimensional feature space with surface assumptions according to embodiments of the present application,

[0035] Figure 3 Schematic diagram of a three-dimensional feature space and a voxel column according to embodiments of the present application. DETAILED DESCRIPTION

[0036] In Figure 1 the accompanying drawings, a method 100, a vehicle 1, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the present application are schematically shown.

[0037] Figure 1Especially a method 100 for determining a geometric representation of a carriageway surface 5 is shown. In a first step 101, a camera image is provided, wherein the camera image comprises a representation of the carriageway surface 5 and is derived from a detection of the camera 2. In a second step 102, features of the camera image are extracted, wherein the extracted features are specific to the carriageway surface 5. In a third step 103, a three-dimensional feature space 3 is defined, wherein the three-dimensional feature space 3 has a plurality of voxels 4. In a fourth step 104, at least three surface hypotheses 6 are determined in the three-dimensional feature space 3 for the carriageway surface 5. In a fifth step 105, the extracted features are projected onto the determined surface hypotheses 6. In a sixth step 106, intersection points 8 of sight lines 7 with the determined surface hypotheses 6 are calculated, wherein the sight lines 7 run from the camera 2 through a respective center of all voxels 4 of the plurality of voxels 4. In a seventh step 107, each extracted feature is attributed to a respective voxel 4, wherein the extracted feature attributed to a respective voxel 4 is obtained by a weighted sum of the extracted features projected onto the surface hypotheses 6, wherein the weights of the weighted sum correspond to a distance of a voxel center of the respective voxel 4 to the calculated intersection point 8 of the sight line 7 with the surface hypotheses 6. In an eighth step 108, a three-dimensional feature map is provided based on the attributed extracted features.

[0038] Figure 2 A three-dimensional feature space 3 with voxels 4 is schematically shown. In this feature space 3, a course of three surface hypotheses 6 and of the actual carriageway surface 5 is represented. Furthermore, intersection points 8 of sight lines 7 of the camera 2 with the surface hypotheses 6 are represented as intersections.

[0039] Figure 3 A schematic view of a three-dimensional feature space 3 with a plurality of voxels 4 and a voxel column 11 is shown. Here, a probability value W is determined for each voxel 4 along the height H for the voxel column 11.

[0040] The present invention is especially based on a concept for a deep learning based 3D detection of traffic lane markings. The method is for example used to improve approaches for 3D line detection for the environmental perception of driver assistance systems (partially automated to fully automated driving functions). Since the invention presented here builds on camera based detection approaches, at least one camera mounted on the vehicle during detection and pointing in the forward direction is for example required as a sensor. Generally, the method according to the embodiments is not limited to a single camera sensor and can be extended with little adjustment effort so that multiple camera sensors can be utilized. For the sake of simplicity, only the case of a single camera is described in the following.

[0041] The image data acquired by the camera is preferably processed by a computing unit, on which the method according to the embodiments can also be implemented as software. The present application is based, inter alia, on learning-based detection approaches that use machine learning models, such as neural networks, for line detection. Since the neural networks used here must be trained, inter alia, on (preferably large) data sets, it can be necessary to provide the image data acquired by the aforementioned camera. In addition, it is also possible to provide the image data with labels that are supposed to describe the so-called true 3D geometry ("Ground Truth") of the traffic lines visible in the corresponding images. This description of the 3D geometry of individual line instances can be implemented, for example, as an ordered list consisting of 3D point coordinates ("Polyline").

[0042] According to the embodiments, the present application describes a new component of a machine learning model for traffic line detection for which the assumption of a "flat road height profile" does not have to be fulfilled, which is rarely the case. Instead of an image feature representation corresponding to a ground plane, inter alia, a three-dimensional feature representation is taken, in such a way that the image features can be modeled and processed in a 3D voxel grid of cuboid shape, i.e. in a three-dimensional feature space. Instead of taking the "flat surface assumption", a surface that can be of arbitrary shape and thus possibly contain ups and downs is preferably implicitly modeled and used for feature selection in the three-dimensional feature space. In order to project the image features into the three-dimensional feature space, different surface assumptions can be used that are supposed to map the space with a high probability of occurrence of road surfaces or, respectively, carriageway surfaces, and thus of 3D traffic line markings.

[0043] In summary, by means of the application according to the embodiments the following advantages, for example, result. By projecting and processing the features into a three-dimensional feature space, three-dimensional features can be learned, whereas feature projections onto a unique flat surface only learn two-dimensional features. With this three-dimensional feature representation the three-dimensional geometry of the lines can be learned better than in the case of a two-dimensional feature representation. By utilizing different surface hypotheses prior knowledge about the three-dimensional line geometry can be used. Since the surface hypotheses can be modeled in such a way that a large part of the height course of the three-dimensional traffic lane can be approximated by these surface hypotheses, for example, the coarse surface geometry does not have to be learned explicitly from the data. When projecting by means of IPM under the assumption of a "flat road surface", if the height course of the traffic lane marking to be learned has an uphill, important image features are not represented, among others. Conversely, these uphill slopes can be represented by the upper voxels in the three-dimensional feature space. On the other hand, if the height course of the line to be learned has a downhill, important image features are only represented compressed, among others. Conversely, these downhill slopes can be represented by the lower voxels in the three-dimensional feature space. Thus, by projecting the image features into a three-dimensional feature space, the space necessary for the detection is modeled geometrically meaningfully. By implicitly modeling the road surface by means of the learned surface probability, the three-dimensional feature space can become more interpretable, whereby the learning process of the machine learning model can be better understood.

[0044] The first technical feature of the application according to the embodiments is a cuboid-shaped 3D voxel grid, i.e. a three-dimensional feature space, which is fixedly defined by the interval and the resolution, which is unfolded in the Cartesian coordinate system of the road. Each voxel of the three-dimensional feature space is defined by a unique 3D coordinate of the voxel center, among others, and preferably represents a multi-dimensional feature, which can be used for three-dimensional traffic lane recognition. The second technical feature of the application according to the embodiments is surface hypotheses of arbitrary shape, which can contain uphill and downhill slopes, which can be unfolded inside the three-dimensional feature space. The surface hypotheses are used, for example, for feature aggregation by means of projection of the feature map, in turn forming the feature basis of each individual voxel, wherein the feature occupancy can be higher than in the case of projection by means of IPM. The third technical feature of the application according to the embodiments is the modeling of the surface inside the three-dimensional feature space by means of the surface probability and the method for learning this surface probability. By classifying the height position along each voxel column of the three-dimensional feature space, the real course of the carriageway surface can be estimated explicitly, whereby only valuable features can be used for further processing.

[0045] The overall architecture of the machine learning model can be described as a deep neural network with different processing components, including multiple layers of a convolutional neural network (“convolutional neural network”). The entire machine learning model is trainable and optimizable end-to-end. The present invention particularly relates to a component for learning three-dimensional (3D) features. Other components of the machine learning model include, for example, a backbone and a probe head that outputs traffic line markings in an arbitrary representation (e.g., based on anchor points or through a continuous function, such as a spline). According to embodiments, the present invention can be used in combination with any backbone, probe head, and line representation.

[0046] For example, a camera image of a front camera perspective is used as an input to the present method. This camera image is preferably further processed by a backbone. The backbone is, for example, a convolutional neural network and can be used for feature extraction using its properties. Thus, a two-dimensional (2D) feature map is preferably generated by further layers, in which each two-dimensional point comprises a multi-dimensional feature. This feature map corresponds, inter alia, to the camera perspective in front.

[0047] In a next step, a three-dimensional feature space is preferably defined in the shape of a cuboid grid. The origin of the three-dimensional feature space is located in the Cartesian reference coordinate system of the vehicle traffic environment, among other things. The three-dimensional feature space is preferably unfolded by means of x, y and z coordinates and is defined by means of corresponding intervals. Additionally, a resolution can be determined for each dimension of the three-dimensional feature space, such that a grid in the shape of a cuboid is formed inside the three-dimensional feature space. Each voxel thus formed can be uniquely described by means of one three-dimensional space coordinate. The number of voxels is defined by means of the corresponding intervals and the resolution, among other things. Furthermore, a multi-dimensional feature ("feature vector") is preferably also uniquely assigned to each voxel, by means of which multi-dimensional feature 3D information can be learned in the training process. Thus, a seamless connection between the features and the 3D geometry is formed, among other things. In order to be able to assign features to the individual voxels, according to an embodiment of the application, a plurality of arbitrarily shaped surface hypotheses are unfolded inside the three-dimensional feature space, which can reflect prior knowledge about typical road courses. These surface hypotheses can contain inclines and declines and can generally be defined arbitrarily. The features extracted from the backbone or the two-dimensional feature map can be projected onto these surface hypotheses, such that each point on the surface hypotheses is associated with a transformed multi-dimensional feature. Furthermore, in order to keep the machine learning model end-to-end trainable, specific transformation layers can be used for the transformation, which allow a differentiable transformation. Here, for example, a sampling grid is used, which projects the extracted features of the two-dimensional feature map onto the individual surface hypotheses. Furthermore, in order to obtain a feature for each point on the surface hypotheses, a bilinear interpolation is preferably used. In Figure 2 Three surface hypotheses 6 are mapped in simplified form in Fig. 6, which describe planes with different angles, respectively.

[0048] For feature aggregation, the intersection points of the line-of-sight rays, which run from the camera through the individual voxel centers, respectively, with all surface hypotheses are preferably calculated. Thus, there are, among other things, exactly as many line-of-sight rays as there are voxel centers. The number of intersection points preferably corresponds to the product of the number of voxels and the number of surface hypotheses. In the subsequent process, only such intersection points located inside the three-dimensional feature space are preferably taken into account. Figure 2 A three-dimensional feature space 3 in the shape of a cuboid grid according to an embodiment of the application is shown. Inside the three-dimensional feature space 3, the surface hypotheses 6 shown are unfolded into angled planes, among other things. In Figure 2In this context, for three different voxel centers, the view ray 7 and the respective intersection 8 with the surface hypothesis 6 are displayed. In a next step, the distance of the intersection 8 along the view ray 7 to the respective voxel center is preferably calculated for each view ray 7. Subsequently, the multidimensional features belonging to the intersection 8 can be weighted therefrom, wherein a large distance preferably leads to a small weight. Then, in a further step, the weighted features are preferably added along their view rays 7, thus forming a weighted multidimensional feature of the belonging voxel. Here, the respective weight of the intersection 8 acts as a balance between the geometric position of the belonging voxel 4 and the feature quality of the intersection 8, among others. In a further step, each voxel feature can additionally be weighted by a geometric prior weight. This weight is calculated, for example, by the minimum distance of the voxel 4 along the respective view ray to the surface hypothesis 6, wherein a distance of zero yields the highest weight of 1. Here, the minimum weight is, for example, the number 0. In summary, a three-dimensional feature space 3 is formed, in particular by the steps indicated, in which each voxel 4 is uniquely described by a 3D coordinate and a weighted aggregated multidimensional feature.

[0049] After the line features have been represented in the three-dimensional feature space thus formed, these three-dimensional features can be combined or selected in different ways in order to be further processed by a subsequent probe head for detection. There are different possibilities to provide the probe head with the three-dimensional features. One possibility consists in simply combining the features along the height dimension of the three-dimensional feature space. Here, for example, the features can be concatenated or accumulated. Another possibility consists in selecting the features from the three-dimensional feature space such that they are taken along the surface run of the carriageway surface. In the following, one approach is described how a surface can be modeled and learned in the feature space in the form of a geometric representation:

[0050] According to the embodiment, one aspect of the application is to learn the surface run by a discrete probability distribution along the height dimension. This can be achieved by a machine learning model which is based on a training by means of an implicit learning method ("self-supervised") or also by means of an explicit learning method ("supervised"). For this purpose, each voxel column 11 of the three-dimensional feature space 3 can be regarded as a classification problem. For further processing, preferably for feature assignment, a single best voxel 4 is classified in each voxel column 11, whereby the height dimension of the three-dimensional feature space 3 can be reduced to a size of 1.

[0051] By minimizing the 3D detection cost function the weights of the machine learning model can be adjusted during the training process such that the resulting probability distribution reflects the true height profile of the road surface 5. Thus, the road surface 5 can be learned implicitly. For calculating the surface probability a probability value W can be estimated for each voxel 4 in a voxel column 11, which represents the probability that the road surface 5 runs through the corresponding voxel 4. Thereby, a probability distribution along the height H is generated for each voxel column 11, in particular (see Figure 3 , right). The voxel 4 indicating the highest probability inside the voxel column 11 is preferably chosen for feature assignment in the next step (see Figure 3 , right). By applying this classification on each voxel column 11 a new feature map is formed, in particular, having the dimensions of the xy-plane of the three-dimensional feature space 3, which comprises the most valuable feature of the three-dimensional feature space 3 - the corresponding surface orientation. Subsequently, this feature map can be encoded by an encoding block comprising multiple convolutional neural layers. The classification problem along the voxel column 11 can also be optimized by explicit supervised learning. For this purpose, an explicit cost function for multiple classification is preferably formulated. This cost function forms part of a total cost function, which is minimized over the training process. If no real 3D surface orientation is available for training, sparse available real 3D line coordinates can be utilized for the estimation method of the real surface.

[0052] As a further possible step, the three-dimensional feature map can be passed to a detection head, which is responsible for the real recognition and geometric estimation of markings and traffic lines. Here, the present application can be combined with any detection head. For example, a lane detection head comprises multiple convolutional neural layers, by which the newly obtained feature map can be processed. From this, parameters can be output, which describe the traffic line markings with any line representation.

[0053] The above explanations of embodiments only describe the present application in the scope of examples. Of course, individual features of the embodiments can be freely combined with each other as far as this is technically meaningful without leaving the scope of the present application.

Claims

1. Method (100) for providing a three-dimensional feature map of a carriageway surface (5), comprising the following steps: - providing (101) a camera image, wherein - capturing (101) a camera image of the carriageway surface (5) by means of a camera (2), - the camera image comprising a representation of the carriageway surface (5) and being derived from a detection of the camera (2), - extracting (102) features of the camera image, wherein the extracted features are specific to the carriageway surface (5), - defining (103) a three-dimensional feature space (3), wherein the three-dimensional feature space (3) has a plurality of voxels (4), - determining (104) at least three surface hypotheses (6) for the carriageway surface (5) in the three-dimensional feature space (3), - projecting (105) the extracted features onto the determined surface hypotheses (6), - calculating (106) intersection points (8) of a line of sight ray (7) with the determined surface hypotheses (6), wherein the line of sight ray (7) runs from the camera (2) through a respective center of all voxels (4) of the plurality of voxels (4), - attributing (107) each extracted feature to a respective voxel (4), wherein the extracted feature attributed to a respective voxel (4) is obtained by a weighted sum of the extracted features projected onto the surface hypotheses (6), wherein the weights of the weighted sum correspond to a distance of a voxel center of the respective voxel (4) to the calculated intersection point (8) of the line of sight ray (7) with the surface hypotheses (6), - providing (108) the three-dimensional feature map based on the attributed extracted features.

2. Method (100) according to claim 1, characterized in that a geometrical representation of the carriageway surface (5) is further derived by the following steps: - determining a probability value (W) for each voxel (4) inside a respective vertical voxel column (11) based on the attributed extracted features of each voxel (4), wherein the probability value (W) represents a probability (W) of the carriageway surface (5) running through the respective voxel (4), - deriving the geometrical representation of the carriageway surface (5) based on the determined probability values (W).

3. Method (100) according to any of the preceding claims, characterized in that the method (100) further comprises the following steps: - calculating a second distance of each voxel (4) to a nearest one of the at least three surface hypotheses (6), wherein the extracted features of each voxel (4) are further weighted based on the calculated second distance.

4. Method (100) according to any of the preceding claims, characterized in that a multi-dimensional feature is uniquely attributed to each voxel (4), the multi-dimensional feature representing the respective extracted features of the voxel (4), wherein each point on the surface hypotheses (6) is associated with a multi-dimensional feature.

5. Method (100) according to any of the preceding claims, characterized in that the multi-dimensional feature is a vector. The projection (105) is performed using a sampling grid, wherein also bilinear interpolation is used in order to obtain the features for each point on the surface hypothesis (6).

6. The method (100) according to any one of the preceding claims, characterized in that The method (100) is applied in a vehicle (1), wherein the vehicle (1) comprises the camera (2) and the method (100) further comprises the steps of: - initiating detection of the camera image, - using the provided three-dimensional feature map for at least partially automated driving of the vehicle (1).

7. The method (100) according to any one of the preceding claims, characterized in that The origin of the three-dimensional feature space (3) is located in a Cartesian reference coordinate system of the road surface (5) as represented by the camera image, wherein the vertical voxel column (11) represents the height dimension in the three-dimensional feature space (3).

8. A computer program (20) comprising instructions which, when the computer program (20) is implemented by a computer (10), cause the computer to implement the method (100) according to any one of the preceding claims.

9. An apparatus (10) for data processing, arranged for implementing the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions which, when implemented by a computer (10), cause the computer to implement the steps of the method (100) according to any one of claims 1 to 7.

Images