Method and device for identifying areas that can be travelled on in an environment of a vehicle

Abstract

The invention relates to a method and a device for identifying areas (2) that can be travelled on in an environment of a vehicle (1), wherein recordings of the environment of the vehicle (1) are captured by means of a sensor system, and wherein areas (2) that can be travelled on are identified in the recordings, wherein a spatial phase imaging sensor system is used as the sensor system for capturing the environment of the vehicle (1), wherein areas (2) that can be travelled on are distinguished from areas (3) that cannot be travelled on on the basis of normal information and on the basis of distance information relating to detected areas.

Description

[0001] Method and device for identifying drivable areas in the environment of a vehicle

[0002] The invention relates to a method for identifying drivable areas in the environment of a vehicle according to the preamble of claim 1 and to a device for identifying drivable areas in the environment of a vehicle according to the preamble of claim 10.

[0003] Navigating a vehicle, especially one with autonomous driving, can be a significant challenge in traffic, especially when there are no marked road sections. This is often the case on country roads or quiet side streets. Modern vehicles, thanks to their sensors, can detect obstacles that have a certain spatial extent. This is the case, for example, with objects such as other vehicles, trees, people, or even bollards and beacons.

[0004] However, the detection of freely available areas (freespace) is also desirable.

[0005] It is well known to detect objects in the environment and infer the available space from their position. This is usually achieved using point clouds and their extent, but it does not work for large, flat structures, such as traffic islands or roundabouts without a central elevation.

[0006] WO 2021 / 030454 A1 describes, in one general aspect, a data management system for spatial phase imaging. A data management system for spatial phase imaging comprises: a storage engine configured to receive and store input data in a dataset format, wherein the input data comprises: first-order pixel-level primitives generated based on electromagnetic (EM) radiation received from an object located within a field of view of an image sensor device; and second-order pixel-level primitives generated based on the first-order primitives.The data management system further comprises: an analysis engine configured to determine a plurality of features of the object based on the first-order pixel-level primitives and the second-order pixel-level primitives; and an access engine configured to provide a user with access to the plurality of features of the object determined by the analysis engine and to the input data stored by the storage engine.

[0007] The invention is based on the object of providing a novel method and a novel device for identifying drivable areas in the environment of a vehicle.

[0008] The object is achieved according to the invention by a method for identifying drivable areas in the environment of a vehicle having the features of claim 1 and by a device for identifying drivable areas in the environment of a vehicle having the features of claim 10.

[0009] Advantageous embodiments of the invention are the subject of the subclaims.

[0010] A method for identifying drivable areas in the surroundings of a vehicle is proposed, wherein images of the surroundings of the vehicle are captured using a sensor system, and wherein drivable areas are identified in the images. According to the invention, a spatial phase imaging sensor system is used as the sensor system for detecting the surroundings of the vehicle, wherein drivable areas are differentiated from non-driftable areas based on normal information and distance information of detected areas.

[0011] In one embodiment, the normal information includes a maximum allowable normal vector inclination. The distance information may include a minimum height and a maximum height.

[0012] In one embodiment, a course of the drivable area is determined taking into account gradient and incline. In one embodiment, an environmental map is modeled from identified drivable areas and non-driftable areas. The images from the spatial phase imaging sensors are optionally fused with data from a conventional detection of freely available areas to model the environmental map. The conventional detection of freely available areas includes the detection of objects in images of the environment and their position, from which inferences about freely available areas are made.

[0013] In one embodiment, the normal information and the distance information are processed classically or using machine learning and / or deep learning.

[0014] In one embodiment, the surroundings map is transferred to a navigation system, to at least one driver assistance system and / or to another system of the vehicle whose calculations require knowledge of a surroundings map.

[0015] In one embodiment, a comparison is made with highly accurate map data in which the road categories motorway, secondary road and rough road are provided.

[0016] In one embodiment, when bad roads are detected, an adjustment of an operating strategy of a drive train and / or chassis settings of the vehicle is carried out.

[0017] In one embodiment, the collected data is reported back to a backend, which merges the information with map data and makes it available to other vehicles.

[0018] According to one aspect of the present invention, a device for identifying drivable areas in the surroundings of a vehicle is proposed, comprising a sensor system for capturing images of the surroundings of the vehicle, wherein the device is configured to identify drivable areas in the images. According to the invention, the sensor system for capturing the surroundings of the vehicle is designed as a spatial phase imaging sensor system, in particular comprising at least one top-view camera and / or at least one mirror-replacement camera with spatial phase imaging functionality, wherein the device is configured to distinguish drivable areas from non-driftable areas based on normal information and distance information. The solution according to the invention enables the reliable detection of freely available, i.e., drivable areas in the surroundings of a vehicle. This allows for increased safety and accuracy.

[0019] Embodiments of the invention are explained in more detail below with reference to a drawing.

[0020] It shows:

[0021] Fig. 1 is a schematic view of a vehicle located on a drivable surface next to which there are non-driftable surfaces.

[0022] The sole Figure 1 is a schematic view of a vehicle 1 located on a drivable surface 2, in particular a roadway, next to which there are non-drivable surfaces 3, for example meadows.

[0023] The present invention is based on spatial phase imaging, abbreviated to SPI. Spatial phase imaging is a technique that records the phase information of light waves to gain insights into the properties of objects or systems under investigation. This phase information describes how the light waves are positioned in relation to each other in time and space and can reveal important properties such as the thickness, refractive index, and topological structure of objects.

[0024] One of the most common methods of spatial phase imaging is interferometry, in which the light from an object is compared with a reference wave to determine the phase shift. There are different types of interferometry, such as Mach-Zehnder interferometry, Michelson interferometry, Twyman-Green interferometry, Fourier transform interferometry, and half-draw-back interferometry.

[0025] Another spatial phase imaging method is digital holography, in which the phase information of light is captured by generating and analyzing holograms. Here, the light from the object is crossed with a reference wave to create an interference pattern, which is then recorded with a camera. This interference pattern contains the phase information of the light and can be processed to create a digital image of the object. There are other spatial phase imaging methods that may be suitable depending on the application, such as modulation of the diffraction index (holographic optical tweezers, ptychography), which enables phase recording directly in a single measurement (but without interference).

[0026] In summary, spatial phase imaging is a technique that allows the capture of the phase information of light waves, which is derived from various types of interferometry and digital holography. Spatial phase imaging is a tool in applications such as microscopy, wavefront sensing, and optical measurements, as it can reveal important properties of objects that are difficult to access using other methods.

[0027] According to the present invention, the surroundings of a vehicle 1 are detected using spatial phase imaging sensors. By detecting the surroundings using SPI, algorithms can be used to clearly identify a drivable surface 2, and the course of this drivable surface 2 can be determined, taking into account gradients and inclines.

[0028] The drivable surface 2, i.e. the road, appears green, for example, since this surface forms the same or at least a very similar normal vector at every point, while the meadow areas as non-driftable surfaces 3 next to the drivable surface 2 appear in mixed colors, for example, since the light is reflected differently by individual blades of grass.

[0029] Data from this SPI-based method can be fused with data from a conventional open space detection method to model a highly accurate environmental map. Conventional open space detection can involve the detection of objects in the environment and their position, from which the available space can be inferred. This can be accomplished using point clouds and their extents.

[0030] The method according to the invention is very well suited for the identification of surfaces and edges.

[0031] In one embodiment of the invention, a camera with SPI functionality is arranged at a position on the vehicle 1 where it has the best possible view of the surroundings. For example, at least one downward-looking camera or at least one mirror-replacement camera (Mirrorcam) with SPI functionality can be provided.

[0032] In the SPI-based method according to the invention, information about surfaces is extracted from the acquired image data. An algorithm is provided that uses normal information and distance information of the detected surfaces to determine whether the surface is a drivable open space (calculating a minimum and maximum height, maximum permissible normal vector inclination). This generates a freespace map.

[0033] Limits for such a defined open space can be specified, for example, by a maximum normal vector inclination, so that only a certain change in the normal vector per square meter is permissible. This ensures that continuous but excessively steep surfaces, such as a roadside ditch and general transitions into the terrain, are detected as impassable. Furthermore, a permissible height difference is also checked for detected surfaces, so that, for example, a sidewalk separated from the roadway by a curb and at a different height is not recognized as a passable surface.

[0034] The finite small surface elements, which do not contain the properties mentioned above, are linked together to create the open space map.

[0035] The information can be processed classically or using machine learning and / or deep learning.

[0036] The information from the open space map can be fused with the data from the classic approach of detecting freely available areas.

[0037] The open space map or a complete set of information resulting from the fusion with the classical approach can be transferred to a navigation system, to at least one driver assistance system and / or to other systems whose calculations require knowledge of a map of the surrounding area.

[0038] Furthermore, a comparison can be made with highly accurate map data, which, for example, include the road categories motorway, secondary road, and rough road. If rough roads are detected, an adjustment of the drivetrain operating strategy and / or the chassis settings of vehicle 1 can be made.

[0039] The collected data can be fed back to a backend, which can then merge the information with map data. This opens up the possibility for automatic map updates and support for vehicles without SPI-based sensors.

[0040] The vehicle 1 can be, for example, a commercial vehicle or a bus or a passenger car.

[0041] List of reference symbols Vehicle accessible area Non-accessible area

Claims

Patent claims 1. Method for identifying drivable surfaces (2) in the surroundings of a vehicle (1), wherein images of the surroundings of the vehicle (1) are captured by means of a sensor system, and wherein drivable surfaces (2) are identified in the images, characterized in that a spatial phase imaging sensor system is used as the sensor system for detecting the surroundings of the vehicle (1), wherein drivable surfaces (2) are differentiated from non-driftable surfaces (3) on the basis of normal information and on the basis of distance information of detected surfaces.

2. Method according to claim 1, characterized in that the normal information comprises a maximum permissible normal vector inclination and / or that the distance information comprises a minimum height and a maximum height.

3. Method according to one of the preceding claims, characterized in that a course of the drivable surface (2) is determined taking into account gradient and inclination.

4. Method according to one of the preceding claims, characterized in that an environmental map is modeled from identified drivable areas (2) and non-drivable areas (3), wherein the images from the spatial phase imaging sensors are optionally fused with data from a classic detection of freely available areas in order to model the environmental map, wherein the classic detection of free areas comprises the detection of objects in images of the environment and their position, from which conclusions are drawn about freely available areas.

5. Method according to one of the preceding claims, characterized in that the normal information and the distance information are processed classically or with the aid of machine learning and / or deep learning.

6. Method according to one of the preceding claims, characterized in that the map of the surroundings is transferred to a navigation system, to at least one driver assistance system and / or to another system of the vehicle (1), the calculations of which require knowledge of a map of the surroundings.

7. Method according to one of the preceding claims, characterized in that a comparison is carried out with highly accurate map data in which the road categories motorway, secondary road and rough road are provided.

8. Method according to claim 7, characterized in that when bad roads are detected, an adjustment of an operating strategy of a drive train and / or of chassis settings of the vehicle (1) is carried out.

9. Method according to one of the preceding claims, characterized in that the recorded data is reported back to a backend, which merges the information with map data and makes it available to other vehicles.

10. Device for identifying drivable surfaces (2) in the surroundings of a vehicle (1), comprising a sensor system for capturing images of the surroundings of the vehicle (1), wherein the device for identifying drivable surfaces (2) is configured in the images, characterized in that the sensor system for capturing the surroundings of the vehicle (1) is designed as a spatial phase imaging sensor system, in particular comprising at least one top view camera and / or at least one mirror replacement camera with spatial phase imaging functionality, wherein the device is configured to differentiate drivable surfaces (2) from non-driftable surfaces (3) on the basis of normal information and distance information.

Images