Method for creating a three-dimensional environment map for a 360° view in a motor vehicle and motor vehicle

Abstract

The invention relates to a method for creating a three-dimensional environment map (2) for a 360° view in a motor vehicle (1). The method comprises capturing the environment around the motor vehicle (1) using sensors (3) of the motor vehicle (1), wherein the sensors (3) comprise at least environment cameras (4), and outputting a textured, rendered, and virtual road surface (8) together with an applied optical highlighting (9) to a user together with preprocessed and / or segmented and / or rendered image data as a three-dimensional environment map (2) via a display device of the motor vehicle (1). The invention also relates to a motor vehicle (1).

Description

[0001] The invention relates to a method for creating a three-dimensional environment map for a 360° view in a motor vehicle. The invention also relates to a motor vehicle configured to perform such a method.

[0002] US Patent 2023 / 0306693A1 describes a system and a procedure for experiencing augmented virtual reality in a vehicle.

[0003] EP 3 376 163 A1 discloses a method and a system for determining an environmental map for a wearer of an active device attached to his head.

[0004] DE 10 2022 129 249 A1 discloses a computer with a processor and a memory with executable instructions, which is set up to receive a multitude of first images of an environment under a first lighting condition, to classify their pixels and to mask selected pixels, to generate a three-dimensional representation of the environment on this basis and to create a second image of the environment under a second lighting condition using one of the first images.

[0005] DE 10 2021 211 309 A1 discloses a method for generating a representation of the environment of a mobile platform, in which a two-dimensionally generated image of the environment is provided, semantically segmented and used to identify object-related image areas, the respective position of the objects relative to the mobile platform is determined, a representation of an adjacent three-dimensional space is generated and the identified object image areas are projected onto corresponding planes at the object positions in order to generate the environment representation of the mobile platform.

[0006] US Patent 2021 / 0148713A1 discloses systems and methods for improved presentation of navigation instructions in augmented reality, in which a live camera image is captured, navigation instructions are generated, and the relative position between the viewpoint of a device and recognized landmarks is determined using an image processing-based positioning algorithm in order to determine the position or shape of the visual presentation of the navigation instructions.

[0007] Technologies such as "Top View" or "Surround View" use external cameras mounted on the vehicle to create environmental maps. These maps of the vehicle's surroundings are then displayed on a screen, such as a head-up display. According to current technology, these environmental maps can be two-dimensional, meaning the stitched camera images are projected onto a flat surface, for example, to display a top-down view of the vehicle's surroundings. Three-dimensional elements or objects in the environment, such as rocks or road irregularities like potholes, are displayed as a two-dimensional, flat image because their depth information is not available.

[0008] One disadvantage of these two-dimensional maps is that they do not provide a realistic and / or immersive representation of the surroundings. Furthermore, they lack the ability to augment or enrich these three-dimensional maps with additional digital content to display enhanced information to a user or driver, such as obstacles, bottlenecks, terrain gradients and inclines, and optimized route guidance in real time. These limitations can reduce the driver's situational awareness and prevent a fully immersive driving experience.

[0009] The object of the invention is to provide a driver of a motor vehicle with a digital, three-dimensional representation of his surroundings and a motor vehicle equipped to carry out such a method.

[0010] The problem is solved by the subject matter of the independent patent claims. Advantageous embodiments of the invention are described by the dependent patent claims, the following description, and the figures.

[0011] A first aspect of the invention relates to a method for creating a three-dimensional environmental map for a 360° view in a motor vehicle, comprising the following method steps: capturing the environment around the motor vehicle using sensors of the motor vehicle, wherein the sensors capture at least environmental cameras. In particular, it is provided that the environmental cameras are arranged on the motor vehicle such that they capture the environment in a 360° range around a vertical axis of the motor vehicle. For each image section or each capture position, two environmental cameras can be provided in order to each capture or record a slightly offset image section of the environmental area, comparable to the distance between human eyes for the three-dimensional capture of the environment."Slightly offset" here means that 60-100 percent of the two image sections of a camera pair overlap, for example, at a detection position. The ambient cameras can be designed according to the state of the art. In a preferred subsequent image preprocessing step, the image data acquired at least by the sensors, especially the ambient cameras, which together can represent the ambient view, are removed from the image data due to the recording or detection process. Such an error can, for example, be caused by dirt on a lens of the ambient cameras. Specifically, in the software-based image preprocessing, each captured image, i.e., the captured image data, is processed in such a way that a clear image is produced.Subsequently, a semantic segmentation of the preprocessed image data is performed to identify a road surface or ground area—that is, a surface on which the vehicle is intended to drive—located in the vicinity of the vehicle, particularly in its immediate surroundings. The immediate surroundings are defined as an area between 1 and 5000 meters from the vehicle, especially along the Earth's surface. A machine learning algorithm, such as an artificial neural network trained with preselected and / or evaluated environmental image data, can be used for semantic segmentation. In other words, the semantic segmentation process aims to select one or more image areas for further processing.Irrelevant or irrelevant image areas or parts of the preprocessed image data can thus be filtered out to avoid further processing and save computing power. The relevant area is a road surface or ground area—that is, an area on which the vehicle could theoretically drive. Therefore, during semantic segmentation, parts of the image data that, for example, describe the sky are filtered out and excluded from further processing. The road surface or ground area can be a road, a path, and / or a carriageway, and / or terrain not open to road traffic, such as open land like a meadow or a field. Subsequently, recognized and / or distinctive image features are extracted from the segmented image data, such as the ground area, for a subsequent depth estimation.As is known from the prior art, prominent image features can be edges, points, and / or lines with strong contrast. These image features are also referred to as "features." In other words, image feature extraction aims to identify and / or capture image areas or features that, for example, due to strong contrast, are particularly easy to recognize or locate within an image or segmented image data for subsequent depth estimation. Depth estimation here means estimating the distance of image points or features in the image data caused by spatial conditions in the real environment.For image feature extraction, so-called "SIFT" (Scale-Invariant Feature Transform) and / or SURF descriptors can be used. The SIFT descriptors are extracted from a collection of orientation scale spaces (the so-called "SIFT feature vectors") at each point in an image analysis. Subsequently, the extracted image features are to be recognized for depth estimation in temporally sequential and / or perspectively offset image data, particularly in already segmented image data. If each image section—where many image sections or the captured image sections can be combined to form the environment map—is captured by pairs of environmental cameras, the detection ranges of the two cameras in a camera pair are intended to partially overlap (comparable to the detection range of two human eyes).The image data captured by each camera in such a camera pair is understood here as being perspectively offset relative to the other camera in the pair. After recognizing the image features, their disparity or parallax is calculated; that is, the distance to the observer or the vehicle is calculated from their respective offset in two different images, which are captured perspectively and / or temporally offset. The offset can result from the movement of the vehicle and / or from the distance between the cameras. Therefore, a paired arrangement of surrounding cameras is not strictly necessary for this offset calculation.The disparity is calculated between the matching, i.e., recognized, image features. From this, for example in a vehicle's control unit, depth information or distance to the vehicle is calculated for each pixel of an image feature and / or the segmented image data. Alternatively, the distance of a point in the environment to the vehicle can be understood, where the point is represented and recognized in the image data, for example as part of an image feature. Subsequently, the depth information is converted, for example by a control unit, into a three-dimensional point cloud, as is known from the prior art. The points of the point cloud can describe at least pixels in the segmented image data with their position in the environment relative to the vehicle.In other words, the advantage of the point cloud is that at least some or all of its points correspond to the image data and each describes depth information that represents a distance relative to the vehicle.

[0012] In the next step, a preliminary, and in particular digital, visual map of the surroundings is created. This involves combining the relevant depth information with the image data assembled for the map, i.e., the preprocessed image data and the segmented image data, such as their pixels. Simultaneously, and specifically for calculating a perspective on the three-dimensional map of the surroundings to be created subsequently, or immediately thereafter, the vehicle is located on the preliminary map, or its position relative to the points in the point cloud that contain depth information is determined. The vehicle serves as the reference point for the three-dimensional map of the surroundings, particularly for its virtual representation.By continuously capturing the vehicle's surroundings using its sensors, and in particular by simultaneously localizing the vehicle in conjunction with the respective depth information of the pixels in the at least pre-processed image data, the three-dimensional environment map can be modified even when the vehicle is stationary. This means that if something is moving in the vehicle's real-world environment, it will be represented by the three-dimensional environment map. If the vehicle is moving, the three-dimensional environment map, or initially the map itself, is correctly displayed relative to the vehicle, for example, by adjusting the image data previously captured relative to a previous position of the vehicle and measured using depth information to the current or subsequent position of the moving vehicle.In other words, the position of a pixel, or of the image data as a whole, in the real environment can be known if the depth information is relative to a known position of the vehicle. This image data can then be adapted to a changing perspective, for example, when the vehicle is moving, so that a correct representation of the environment is achieved through adjusted or newly calculated depth information in conjunction with appropriately selected, especially current, image data.

[0013] The simultaneous creation of the preliminary map and localization of the vehicle can be performed using a so-called SLAM method (Simultaneous Localization and Mapping). Additionally or alternatively to the SLAM method, the three-dimensional environmental map and / or the preliminary map creation can be performed using a spatial computing method.

[0014] In a further process step, a virtual, continuous road surface or potential road surface is created from the preliminary map or point cloud, i.e., based on depth information, using one or more specially designed algorithms, such as machine learning and / or a so-called "Poisson Surface Reconstruction" or "Marching Cube" algorithm. This virtual road surface can take the form of a network or a so-called "3D mesh," bounded by points from the point cloud or by depth information. The virtual road surface thus represents or describes a real ground area around the vehicle. It should depict all objects located on or adjacent to the assumed roadway of the vehicle.For example, if the vehicle is traveling straight ahead, the road surface is defined as an area or ground area in front of the vehicle. If the vehicle is traveling in reverse, the area behind it is represented as the roadway, or virtual road surface. The objects can be, for example, stones and / or potholes located directly in front of the vehicle (within the width of its lane when traveling forward) or directly adjacent to the vehicle's path. In other words, the points on the preliminary map that lie on or near the roadway are virtually combined into a single surface, namely the virtual road surface.When grouping objects, a distinction is made between whether they are combined into a single part of the road surface or into two separate objects, such as those that differ three-dimensionally or are spaced apart. For example, two stones that are so close together that they are recognized as a single object are grouped together as one object on the road surface or ground. If the stones exceed a predetermined distance, they can be recognized as two separate objects, and the road surface will treat them as two distinct objects. In other words, the virtual road surface displays a bulge or a cap under which both stones are grouped, or alternatively, two caps, each containing or representing one of the two stones.The level of detail or resolution of the virtual road surface should be high enough for a driver or user to identify potential hazards for their vehicle. This means that potholes up to a certain depth or stones up to a certain size need not be included in the rendering or creation of the road surface. The virtual road surface thus represents a set of points on the preliminary map, combined with depth information or a three-dimensional point cloud. To display the virtual road surface on the three-dimensional environment map, it must be textured so that the driver or user can recognize it as part of the surroundings.Immediately after the virtual road surface is created, it consists only of pixels that are not yet perceptible or understandable to a driver. In other words, there is no color information yet for the points of the virtual road surface indicating which color these points should be displayed with in the three-dimensional map. Therefore, in a further process step, the virtual road surface is textured using the preprocessed image data by backprojecting this image data onto the virtual road surface. In other words, the points of the virtual road surface are "searched for"—that is, identified—in the preprocessed image data or the preliminary map and assigned to the corresponding pixels there, without necessarily having to use the segmented image data.

[0015] To make the virtual road surface clearly distinguishable for the driver or user in the displayed three-dimensional environment map, a visual highlight is applied to the virtual road surface, at least along its boundaries. This visual highlight is applied either before or after texturing the virtual road surface. The road surface can be a continuous area, particularly one oriented according to the current driving direction and / or speed, for example, depending on the steering angle and the vehicle's direction of travel. Naturally, the virtual road surface should not be completely covered with the visual highlight, as this would make it difficult to distinguish from the surroundings and would no longer be recognizable three-dimensionally or in perspective within the three-dimensional map.Therefore, the visual highlighting of the virtual road surface is applied at least along its boundaries. For better orientation or visibility for the driver, the visual highlighting can extend along the surface between the boundaries or the boundaries of the road surface itself, for example, in the form of a network and / or grid structure. The visual highlighting can take the form of lines that, for example, display a signal color. In particular, the lines can have a complementary color to the pixels surrounding them or to a combination thereof.

[0016] To visually improve the overall appearance, i.e., to ensure particularly good perception or processing of the three-dimensional environment map by the driver, the three-dimensional environment map, which includes the virtual road surface and its associated textures or colors, is rendered using state-of-the-art technology. The environment, captured three-dimensionally using depth information, is projected onto a two-dimensional image for output by the vehicle's display. To enable the user to perceive the three-dimensionality, a technique called "ray tracing" or light ray tracking can be employed, allowing for the display of particularly realistic shadows and / or reflections.The textured, rendered, and virtual road surface, along with the pre-processed and / or segmented and / or rendered image data, is then displayed to the driver as a three-dimensional environment map via a display device, representing at least a two-dimensional image. The display device can be integrated into the vehicle. Naturally, the virtual road surface itself can only be displayed on an image area or part of the image data that is not excluded from further processing—that is, the segmented part of the image data or the segmented image data itself—because the pixels corresponding to the points of the virtual road surface are known in this case.The image data that was removed or segmented can be reintegrated into the three-dimensional map so that the user can see the sky and thus orient themselves more effectively, at least vertically. Overall, this method offers the advantage of providing the user with a three-dimensional representation of their surroundings, in which a potential or planned route is visually highlighted. This visual highlighting and its three-dimensional rendering clearly indicate obstacles along the route or virtual and / or real road surface, thus enabling safer vehicle operation, especially when the driver cannot see through the vehicle's windows.

[0017] According to the invention, during image feature extraction, one or more anchor points are defined for the three-dimensional environment map. These anchor points can be provided with one or more additional pieces of information describing at least one three-dimensional feature in the environment, and this information is displayed to the driver along with the three-dimensional environment map. The at least one anchor point can thus represent the three-dimensional feature, whereby the additional information or a label attached to the three-dimensional environment map is perceptible to the driver, thus indicating this feature.Such a three-dimensional feature can be an obstacle on the real and / or virtual road surface, which is displayed on the virtual road surface, and / or a low or too low obstacle relative to the vehicle height that could damage the vehicle if driven under or through it, such as a traffic sign or a branch. Anchor points, as known from the prior art for methods of so-called "Augmented Reality" or "Augmented Virtual Reality", can be displayed with so-called labels or notes when output in the three-dimensional environment map.The anchor points according to the invention, or at least one anchor point, represent particularly distinctive, recognizable, and / or clear image features in the segmented image data that at least partially represent or reproduce a three-dimensional situation, which can be recognized by a machine learning algorithm. A three-dimensional situation can be a stone, a pothole, and / or a narrow passage. This offers the advantage that damage to the vehicle can be avoided particularly efficiently.

[0018] A first alternative according to the invention provides that the additional information describes a gradient and / or incline as a given condition. A particularly steep gradient and / or incline can pose a danger, especially in winter when the road surface is slippery. This further development advantageously allows the driver to be warned of such a danger.

[0019] A second alternative according to the invention provides that the additional information describes a route guidance system, incorporating at least one elevation data point about the surroundings, for bypassing impassable areas. This further development is particularly advantageous when the motor vehicle is moving off-road, i.e., not on a road intended for motor vehicle use. "Impassable" here means, for example, that a stone too large for the motor vehicle to drive over and / or a particularly deep pothole is located in the current trajectory or roadway of the motor vehicle, i.e., in its actual surroundings.For route guidance, a drivable virtual road surface can be determined and visually highlighted on the three-dimensional map, then displayed or suggested to the driver. The driver can then follow the route shown on the map, thus ensuring that the route is also followed in the real-world environment, avoiding obstacles. This allows for particularly fast progress in off-road terrain, and can especially replace "scouts" who drive ahead of a convoy to scout and display routes, particularly when using an off-road vehicle.

[0020] According to a third alternative of the invention, the first and the second alternative are combined, wherein the invention necessarily comprises one of the three alternatives.

[0021] According to the invention, in addition to the first, second, or third alternative, the supplementary information can describe a narrow passage and / or an obstacle as a three-dimensional feature. This offers the advantage of warning the driver of such features that could cause damage to the vehicle if it were driven over or collided with them.

[0022] The invention comprises further developments or embodiments that offer further advantages.

[0023] A further development of the method involves generating a stereo effect in the image data, which is captured by only one surrounding camera, through the vehicle's own movement. Depth information is then generated or calculated using this stereo effect. Each surrounding camera can capture a section of the environment, and all these sections can be combined to create a three-dimensional map or a preliminary map. Instead of capturing each section with two cameras positioned directly next to each other to create the stereo effect by offsetting the positions of the two cameras in a pair, the stereo effect can be generated by the vehicle's own movement.This offers the advantage that the respective image data for each image section can be captured with just one surround-view camera, yet a stereo effect is still generated, allowing depth information to be calculated by determining the disparity. Here too, the vehicle incorporates multiple surround-view cameras, but each camera captures only one image section of the surroundings for each section. Such an image section could be, for example, the rear of the vehicle, the side, the area in front of it, or the ground.

[0024] Utilizing the stereo effect requires that image areas from two images taken sequentially during the vehicle's movement overlap. The image features in these two images are recognized, a process also known as "matching," and the depth information is calculated by analyzing their disparity.

[0025] A further development of the procedure stipulates that, to increase the accuracy of creating the preliminary map, and thus the three-dimensional environmental map, and / or the localization of the vehicle, the sensors include a GPS receiver (Global Positioning System), an IMU (Inertial Measurement Unit), and / or at least one LIDAR (Light Detection and Ranging) sensor. Using the respective data acquired from the environment—that is, the real environment around the vehicle—the estimation of depth information is improved compared to depth information obtained solely from the environmental cameras. The GPS receiver and / or the IMU can support motion detection or self-motion detection, for example, by improving or increasing the signal-to-noise ratio in the motion data and / or by fusing the data from the GPS receiver and / or the IMU with other sensor data from the vehicle.The "other sensor data" includes, for example, the vehicle's instantaneous speed in conjunction with its steering angle, from which the vehicle's current heading can be calculated. LiDAR can provide depth information for pixels and / or the point cloud, which can be more accurate than depth information obtained from image data. Additionally or alternatively, it is planned that depth information from LiDAR sensors and image data can be fused together to replace missing data in one set of depth information with the other (if available). This offers the advantage of creating a particularly informative or detailed preliminary map.

[0026] A further development of the method stipulates that the area to be monitored includes a space beneath the vehicle, which is detected by sensors mounted on the vehicle's underbody. In other words, sensors, such as underbody cameras (at least one camera) and / or at least one or more LiDAR sensors, should be mounted on the vehicle's underbody to monitor the area beneath it. This offers the advantage that obstacles located beneath the vehicle can also be detected, and the driver can be warned of them if necessary.

[0027] A further development of the method provides for the display device to be configured as XR (Extended Reality) glasses and / or AR (Augmented Reality) glasses and / or VR (Virtual Reality) glasses. The corresponding glasses can be designed as known from the prior art. XR glasses can be a combination of AR glasses and VR glasses. Each pair of glasses can include position sensors, such as an IMU, to detect the direction of gaze and / or head position of the driver or the wearer of the glasses. Depending on the driver's direction of gaze, the glasses can display a corresponding section of the three-dimensional environment map, in particular one that would be visible to them looking out of the vehicle with that same direction of gaze.This offers the advantage of enabling the driver to have a particularly high level of situational awareness, meaning that he can perceive the surroundings of his vehicle very well and thus recognize dangers or obstacles particularly quickly.

[0028] A second aspect of the invention relates to a motor vehicle designed to carry out a method, in particular according to the first aspect of the invention, wherein the motor vehicle comprises sensors, a control unit and at least one display device.

[0029] The motor vehicle can be designed as a passenger car, truck or motorcycle, powered by an internal combustion engine and / or an electric traction machine, for example an electric motor.

[0030] Semantic segmentation, the identification and / or determination of at least one anchor point, can be performed using a machine learning algorithm, for example, a neural network, particularly a convolutional neural network (CNN). Such a neural network for semantic segmentation can be trained, for instance, with image data captured by the vehicle's surround-view cameras.

[0031] The proposed invention aims to enable a comprehensive and immersive, transparent view through a vehicle. This is achieved through the integration of three-dimensional maps created using SLAM methods and / or spatial computing. Various sensors, such as environmental cameras, lidar, and / or depth cameras, continuously acquire image and / or depth data, which are combined to form a seamless three-dimensional environmental map.

[0032] This map, or three-dimensional environment map, can be dynamically expanded for each time step and displayed in AR and / or XR glasses. Spatial computing allows additional XR elements, such as bottlenecks, obstacles, gradients, and / or inclines, as well as real-time route guidance, to be displayed or provided on the three-dimensional environment map. This can improve safety, particularly the driver's situational awareness, and offers an immersive driving experience, especially on uneven terrain—that is, on a road or road surface not designed for motor vehicle use.

[0033] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0034] The invention also includes a control device for the motor vehicle, which is configured to execute one of the described methods. The control device can comprise a data processing device or a processor circuit configured to perform an embodiment of the method according to the invention. For this purpose, the processor circuit can comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can comprise program code configured to perform the embodiment of the method according to the invention when executed by the processor circuit.The program code can be stored in a data memory of the processor device. The processor device can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0035] The invention also includes further developments of the method according to the invention, which have features already described in connection with the further developments of the motor vehicle according to the invention. For this reason, the corresponding further developments of the method according to the invention are not described again here.

[0036] As a further solution, the invention also includes a computer-readable storage medium comprising program code which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code, assembly code, source code of a programming language (e.g., C), or a program script (e.g., Python). Alternatively, the computer-readable storage medium can be implemented as a signal containing computer-readable data, such as a time-varying voltage signal or a radio signal.

[0037] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0038] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematic representation of a three-dimensional environment map with three-dimensional features; Fig. 2 Schematic representation of a three-dimensional environment map with an optically highlighted, virtual road surface.

[0039] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0040] In the figures, identical reference symbols denote functionally equivalent elements.

[0041] Fig. Figure 1 shows a schematic representation of a three-dimensional environment map 2 with three-dimensional features 11. The real environment can be detected by means of sensors 3 of the vehicle 1, for example, by means of surround-view cameras 4 and / or underbody cameras 5. In other words, the environment around the vehicle 1 is to be detected, at least partially, by means of the sensors 3. The three-dimensional environment map can be a SLAM-based dynamic three-dimensional map and can be displayed to a driver of a vehicle 1 on a display device. The display device can, in particular, be the one described in Fig. The glasses shown in Figure 1 (7) could be XR glasses, AR glasses, and / or VR glasses. Three-dimensional objects (11) could include, for example, a stone, as in... Fig. 1. The area shown is to the right of the vehicle, and / or it may be a bottleneck. The three-dimensional features, such as the bottleneck shown here, can be supplemented with additional information that is displayed to the driver, i.e., included in the three-dimensional environment map. This additional information can also be referred to as a label and, as in Fig. Figure 1 warns of a bottleneck. The warning can, for example, include an oversize indicating by which the bottleneck is wider than the vehicle, or an undersize indicating by which the bottleneck is narrower than the vehicle's width. Figure 11 augmenting the three-dimensional conditions can be created or provided using spatial computing.

[0042] Fig. Figure 2 shows a schematic representation of a three-dimensional environment map 2 with an optically highlighted, virtual road surface 8. The optical highlighting 9, as shown in Fig. 2 can be depicted as a network, with at least one boundary of the virtual road surface 8 highlighted. Through this visual highlighting 9, the virtual road surface 8 can also be suggested to a driver as a route or guide 10 in the three-dimensional environment map 2.

[0043] As in Fig. As shown in Figure 2, the visual highlighting can represent a route guidance 10, i.e., an area of ​​a road surface or in the vicinity of the motor vehicle 1, on or along which it is to be or can be moved, in particular for driving around obstacles. Depending on the speed of the motor vehicle 1, the highlighting can emphasize a longer (at high speed) or shorter (at low speed) route segment or area of ​​the virtual road surface.

[0044] Overall, the examples show how a three-dimensional environment map can be provided for all-round visibility in vehicles.

Claims

[1] Method for creating a three-dimensional environment map (2) for a 360° view in a motor vehicle (1), comprising the following method steps: • Detection of the environment around the motor vehicle (1) by means of sensors (3) of the motor vehicle (1), wherein the sensors (3) include at least environment cameras (4); • Segmentation, in particular semantic segmentation, of image data obtained from the sensors to identify a ground area in the vicinity of the motor vehicle (1); • Image feature extraction from the segmented image data of detected and / or prominent image features in the segmented image data; • Recognition of image features in image data captured sequentially and / or with shifted perspectives; • Calculation of a disparity between matching and / or recognized image features and conversion of the disparity into depth information for the segmented image data; • Conversion of depth information into a three-dimensional point cloud; • Creation of a preliminary map of the surroundings by combining the pre-processed and / or segmented image data with the respective depth information to create the map and current localization of the motor vehicle (1) on the map; • Creation of a virtual, continuous road surface (8) from the three-dimensional point cloud using the identified ground area; • Texturing of the virtual road surface (8) using the image data by backprojecting this image data onto the virtual road surface (8); • Imprinting an optical highlight onto the virtual road surface (8) at least along its boundary; • Rendering of the three-dimensional environment map (2); and • Output of the textured, rendered and virtual road surface (8) together with the optical highlighting (9) applied to it to a user together with the preprocessed and / or segmented and / or rendered image data as a three-dimensional environment map (2) via a display device of the motor vehicle (1), wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which at least one additional piece of information (6) describes a slope and / or inclination as a three-dimensional feature (11), or wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a slope and / or an incline and which describes at least one additional piece of information (6) a route guidance (10) including at least one height information about the surroundings for bypassing impassable areas and / or a three-dimensional feature (11), or wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a bottleneck and / or an obstacle and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a slope and / or an incline, or wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a bottleneck and / or an obstacle, and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a slope and / or an incline, and which describes at least one additional piece of information (6) as a route guidance (10) including at least one elevation information about the surroundings for bypassing impassable areas and / or a three-dimensional feature (11), or wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which describes at least one additional piece of information (6) a route guidance (10) including at least one elevation information about the surroundings for bypassing impassable areas and / or a three-dimensional feature (11), or wherein at least one anchor point for the three-dimensional environment map (2) is defined during image feature extraction, which is provided with at least one additional piece of information (6) that describes at least one three-dimensional feature (11) in the environment and which is output together with the three-dimensional environment map (2) and which describes at least one additional piece of information (6) as a three-dimensional feature (11) a bottleneck and / or an obstacle and which describes at least one additional piece of information (6) a route guidance (10) including at least one height information about the surroundings to bypass impassable areas and / or a three-dimensional feature (11). [2] Method according to claim 1, wherein a stereo effect is generated in the image data, which are each captured by only one environment camera (4), by means of a self-movement of the motor vehicle (1) with respect to the environment, and the depth information is calculated by means of the stereo effect. [3] Method according to one of the preceding claims, wherein in the creation of the preliminary map and / or the localization the sensors (3) comprise a GPS receiver, an IMU and / or at least one LIDAR, by means of whose respective acquisition data of the environment around the motor vehicle (1) an estimation of the depth information is supported. [4] Method according to one of the preceding claims, wherein the environment to be detected comprises an area under the motor vehicle (1) which is detected by means of sensors (3) arranged on or in the direction of travel in front of or behind the underbody of the motor vehicle (1). [5] Method according to any of the preceding claims, wherein the display device (7) is designed as XR and / or AR and / or VR glasses. [6] Motor vehicle glasses (7) designed to carry out a method according to one of the preceding claims, comprising sensors (3), a control unit and at least one display device.

Images