Method for displaying an augmented vehicle environment, control device, computer program, computer-readable storage medium and vehicle

Abstract

The invention relates to a method for displaying an augmented vehicle environment (22) of a vehicle (10) by means of a display device (12), wherein recording data (20) of at least one area of ​​the vehicle environment (22) is recorded by a vehicle environment detection device (14) of the vehicle (10); a virtual vehicle environment (22) of the vehicle environment (22) is reconstructed by a control device (24) from the recording data (20); position data (26) describing a position of a display device (12) within the vehicle (10) is received by the control device (24); a field of view (28) of the display device (12) is determined by the control device (24) based on the position of the display device (12); and display data (30) for displaying the virtual vehicle environment (22) in the field of view (28) is generated by the control device (24).The control device (24) provides the display data (30) to the display device (32); the field of view (28) of the display device (12) is superimposed on the virtual vehicle environment (22) by the display device (32).

Description

[0001] The invention relates to a method for displaying an augmented vehicle environment of a vehicle by means of a display device, a control device, a computer program, a computer-readable storage medium, and a vehicle.

[0002] A driver can see the area around the vehicle through the windows. However, the driver's view can be obstructed by vehicle components such as pillars or other objects, creating an area the driver cannot see, also known as a blind spot. These obstructions to the driver's field of vision can be particularly problematic when parking, maneuvering, or maneuvering into a parking space.

[0003] To assist the driver during parking maneuvers, vehicles are equipped with vehicle environment detection devices, which may have one or more cameras that record the unseen area of ​​the vehicle's surroundings and display it to the driver on a screen.

[0004] A disadvantage of the described setup is that the driver has to look at the screen, thus restricting their view of the vehicle's surroundings. Furthermore, the available area is usually fixed.

[0005] According to the current state of technology, solutions exist that eliminate these disadvantages.

[0006] For example, DE 10 2015 007 246 A1 describes a method for operating a display system and a display system for a vehicle. It is provided that a field of view of electronic data glasses arranged in the motor vehicle is determined and, if at least one predefined element of the motor vehicle is located within the determined field of view, an area of ​​the surroundings of the motor vehicle located behind the predefined element is displayed by means of the electronic data glasses, whereby the element is superimposed on the displayed area of ​​the surroundings from the predefined viewing angle.

[0007] DE 10 2015 201 642 A1 describes a device and a method for displaying the surroundings of a vehicle. The device comprises at least one camera for capturing the surroundings of the vehicle, a gaze direction detector configured to detect the gaze direction of a user, and a computing unit that generates a video image based on the captured surroundings of the vehicle, which depicts at least a section of the surroundings of the vehicle corresponding to the detected gaze direction of the user.

[0008] DE 10 2017 209 802 A1 describes a method and a device for operating a display system with data glasses.

[0009] The present invention is based on the objective of providing a solution for the provision of a vehicle environment of a vehicle by means of a display device.

[0010] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention are described by the dependent patent claims, the following description, and the figures.

[0011] A first aspect of the invention relates to a method for displaying an augmented vehicle environment of a vehicle by means of a display device.

[0012] A key concept of the invention is to provide the driver or occupants of the vehicle with an augmented vehicle environment. This means that the vehicle environment visible to the driver is not replaced by a virtual vehicle environment, but rather supplemented by the virtual vehicle environment.

[0013] The vehicle in question may be, in particular, a motor vehicle, for example a passenger car or a truck.

[0014] The display device can be designed, in particular, as glasses or a headset for wearing on the driver's head. The display device includes a display unit for showing a message. The display unit can include screens positioned in a wearing position in front of the driver's eyes.

[0015] The display device is designed to show the display according to the received display data provided to it. The display device can also be configured as a transparent screen positioned in front of the driver's eyes, allowing visible light to pass through. This allows the driver to continue seeing at least part of the vehicle's surroundings directly with their eyes while wearing the display device. In this case, the display device can be configured to project the display onto the screen, thus overlaying the display onto the driver's field of vision.

[0016] The method involves recording data of at least one area of ​​the vehicle's surroundings using a vehicle environment sensing device. This device can include one or more cameras configured to record a specific area of ​​the vehicle's surroundings and generate the corresponding data. These cameras could be, for example, wide-angle cameras mounted at various positions on the vehicle.

[0017] It is intended that a control device will reconstruct a virtual vehicle environment from the respective recording data. In other words, the recording data will be provided to the control device. The control device is configured to generate the virtual vehicle environment from the recording data, which represents at least one area of ​​the vehicle's environment. The reconstruction can, for example, include a stitching process, in which the recordings from the individual camera systems are combined. The reconstruction can also include a rectification process, in which distortions in the recordings caused by the lens and / or the camera's mounting position are corrected.

[0018] The virtual vehicle environment can, in particular, be a 360-degree view of the vehicle's surroundings reconstructed from the recordings. For example, the virtual vehicle environment might represent a 360-degree view of the vehicle's environment. Alternatively, the virtual vehicle environment could be a 3D reconstruction of the vehicle's surroundings, which could be reconstructed from 3D volumes using Structure from Motion techniques derived from the recording data.

[0019] In a further step, the control device is designed to receive position data describing the position of a display device within the vehicle. The position of the display device within the vehicle can include a translational and a rotational component relative to a reference position. For example, the translational component can describe a displacement of the display device relative to the reference position within the vehicle, and the rotational component can describe the direction in which the display device is oriented. The position data can be provided, for example, by the display device itself and / or a bearing sensor.

[0020] In a further step, the control device is designed to determine the display's field of view based on its position. In other words, the position of the display within the vehicle can be used to determine the orientation of its field of view. This allows the control device to identify the portion of the virtual vehicle environment that lies within the display's field of view. The control device can be provided with an angular range covered by the display for determining this field of view.

[0021] In a further step, the control device is intended to generate display data for showing the virtual vehicle environment according to the user's field of vision. In other words, the control device is intended to describe the portion of the virtual vehicle environment that lies within the user's field of vision in the display data. The control device generates the display data for representing the virtual vehicle environment in the user's field of vision, taking into account any distortion of the vehicle environment depending on the position of the display device.

[0022] In a further step, the display data is provided to the display device. This can be done via a wired or wireless connection. The display device overlays the virtual vehicle environment within its field of view onto the vehicle's actual environment. In other words, the display device's field of view, as described in the display data, is transparently projected into the virtual environment within the driver's field of vision to provide a natural view. Alternatively, the field of view described in the display data can be overlaid with additional content, such as that provided by a first-person camera system, which describes the content of the driver's field of view as recorded by the camera system.

[0023] The display device is equipped with an ego camera unit that records the user's field of view and provides ego recording data of that field of view. In other words, the display device includes the ego camera unit, which is configured to record the display device's field of view. The method provides that the display device presents a superimposed display based on the recorded data and the ego recording data. In other words, the ego camera unit provides the ego recording data that describes the field of view. This ego recording data is then fused with the recording data provided by the environment sensing device.The fusion can, for example, provide for an overlay of the content of the field of view, captured by the display device, with the virtual vehicle environment within the field of view, whereby the virtual vehicle environment can be mapped transparently over the real vehicle environment.

[0024] The invention offers the advantage that the representation of the virtual vehicle environment does not replace the driver's view of a real vehicle environment, but rather expands upon it. This results in the advantage that latency effects and adverse representations of the vehicle environment can be avoided through a purely virtual display.

[0025] A further development of the invention provides that the control device receives motion data from the vehicle, describing a change in the vehicle's position from a previous position to the current position. In other words, the vehicle has, for example, acceleration sensors that can detect its movement. The vehicle environment sensing device can also be configured to generate the motion data based on changes in the vehicle's environment over time. By tracking these changes in the vehicle's environment, the vehicle's movements from a previous position to the current position can be determined, for example, by tracking individual points.

[0026] The control device reconstructs at least a substitute area of ​​the virtual vehicle environment for the current vehicle position from the recorded data of the vehicle environment at the previous vehicle position. In other words, it is intended to generate the substitute area of ​​the virtual vehicle environment for the current vehicle position, at least partially, from the recorded data captured by the vehicle environment detection device of the vehicle environment at the previous vehicle position. For example, a front camera of the vehicle may capture a section of the road in front of the vehicle. The motion data can describe that the vehicle moved straight ahead from the previous vehicle position to the current vehicle position and is located above the section of the road captured from the previous vehicle position.The vehicle environment detection device can be configured such that it cannot record the area beneath the vehicle and therefore cannot directly reconstruct the vehicle environment in that area from current recording data of the vehicle's current position. Instead, the control device can provide a substitute area that maps the corresponding area beneath the vehicle. This reconstructed substitute area is based on the recording data of the previous position, taking into account the vehicle's movement as described by the motion data. This improved design offers the advantage that areas of the vehicle environment not previously recorded can be reconstructed based on previous recording data.

[0027] A further development of the invention provides that the at least one substitute area reconstructed from the recording data of the vehicle's surroundings at the previous vehicle position is located outside the detection range of the vehicle's surroundings detection device. In other words, it can be provided that the substitute area of ​​the vehicle's surroundings lies outside the detection range of the vehicle's surroundings detection device and therefore cannot be detected by a camera unit of the vehicle's surroundings detection device. This could, for example, be an area that is not recorded by any of the camera units. In particular, it could be an area under the vehicle. This further development of the invention offers the advantage that areas of the vehicle's surroundings that are located outside the detection range can be reconstructed as a substitute.

[0028] A further development of the invention provides that at least some of the position data is generated by the display device and made available to the control device. In other words, the display device is configured to determine its position within the vehicle completely or at least partially. For example, the display device may include camera units or sensor units configured to track known reference points within the vehicle and, based on these tracked reference points, to determine the position of the display device within the vehicle.

[0029] A further development of the invention provides that at least some of the position data is generated by a bearing detection device and made available to the control device. In other words, the bearing detection device is arranged inside the vehicle. The bearing detection device can, for example, be configured to track reference points on the display device and, based on the tracked reference points, to determine the position of the display device in the vehicle.

[0030] A further development of the invention provides that the display device presents the display data on a transparent display element of the display device. In other words, the display device has a transparent display element through which the driver can see. The display data is presented by the display device within the transparent display element. This can be achieved, for example, by projecting the display onto the transparent display area. This further development offers the advantage that the driver's actual vision in the field of view is not replaced.

[0031] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0032] A second aspect of the invention relates to a control device which is configured to carry out the process steps of the control device of a method according to the first aspect of the invention.

[0033] The control device is configured to receive recording data of at least one area of ​​the vehicle's environment, recorded by a vehicle environment sensing device. The control device is configured to reconstruct a virtual vehicle environment from this recording data. The control device is configured to receive position data describing the location of a display device within the vehicle and, based on this location, to determine the display device's field of view. The control device is configured to generate display data for showing the virtual vehicle environment within this field of view and to provide this display data to the display device.

[0034] The control device can include a data processing device or a processor circuit configured to perform an embodiment of the method according to the invention. For this purpose, the processor circuit can include at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can include program code configured to perform the embodiment of the method according to the invention when executed by the processor circuit. The program code can be stored in a data memory of the processor circuit. The processor circuit can, for example,based on at least one circuit board and / or at least one SoC (System on Chip).

[0035] A third aspect of the invention relates to a computer program, comprising program code, which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The program code can be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g., C) and / or as a program script (e.g., Python).

[0036] A fourth aspect of the invention relates to a computer-readable storage medium comprising a computer program according to the third aspect of the invention.

[0037] As a further solution, the invention also includes a computer-readable storage medium comprising program code which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code, assembly code, source code in a programming language (e.g., C), or a program script (e.g., Python). Alternatively, the computer-readable storage medium can be implemented as a signal containing computer-readable data, such as a time-varying voltage signal or a radio signal.

[0038] A fifth aspect of the invention relates to a vehicle which has a control device according to the second aspect of the invention, as well as a vehicle environment detection device.

[0039] The vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0040] The invention also includes further developments of the control device, the computer program, the computer-readable storage medium, and the vehicle according to the invention, which have features already described in connection with the further developments of the method according to the invention. For this reason, the corresponding further developments of the control device, the computer program, the computer-readable storage medium, and the vehicle according to the invention are not described again here.

[0041] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0042] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematic representation of a vehicle which has a vehicle environment detection device and a control device; Fig. 2 a schematic representation of a display described by the display data, which represents the field of view; and Fig. 3 a schematic representation of the sequence of a procedure for displaying an augmented vehicle environment of a vehicle by a display device.

[0043] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0044] In the figures, identical reference symbols denote functionally equivalent elements.

[0045] Fig. Figure 1 shows a schematic representation of a vehicle 10 which has a vehicle environment detection device and a control device.

[0046] Vehicle 10 can, for example, be configured as a passenger car or a truck.

[0047] The vehicle environment detection device 14 of the vehicle 10 can include camera units 16, which can be arranged at various installation positions on the vehicle 10. The camera units 16 can be configured to capture images of respective detection areas 18 of a vehicle position 36 and record data 20 of their respective detection area 18 of the vehicle environment 22. The camera units 16 of the vehicle environment detection device 14 can provide the recording data 20 to the control device 24. The control device 24 can be configured to reconstruct a virtual vehicle environment 22 from the recording data 20.

[0048] It may be possible that an area of ​​the vehicle environment 22 cannot be recorded by the vehicle environment detection device 14. This could, for example, be the underside of the vehicle 10. The control device 24 can be configured to provide a corresponding substitute area 38 in the virtual vehicle environment 22. For this purpose, the control device 24 can be configured to receive motion data 34, which may be provided, for example, by a motion detection device 44 of the vehicle 10 or determined by the control device 24 based on the recording data 20. This allows a change in the vehicle position 36 to be detected, which can enable the reconstruction of the substitute area 38 of the current vehicle position 36 based on the recording data 20 of a previous vehicle position 36.

[0049] The display device 12 may be configured to detect its position within the vehicle 10. Additionally or alternatively, the position of the display device 12 may be determined by a corresponding position detection device 40 of the vehicle 10. Corresponding position data 26 can be provided to the control device 24. Based on the position data 26, a viewing area 28 of the display device 12 can be determined. From the virtual vehicle environment 22, a display for the viewing area 28 can be determined by the control device 24 and provided to the display device 12. The display device 12 may be configured to extend the viewing area 28 with the display described by the display data 30.This can be achieved by overlaying ego-recording data, which was captured by an ego-camera device 42 of the virtual display device 12, with the display data 30 and displaying it in the display device 32. Alternatively, the display can be projected onto a transparent screen of the display device 32.

[0050] Fig. Figure 2 shows a schematic representation of a display described by the display data, which represents the field of view.

[0051] The illustration shows the interior fittings 46 of vehicle 10. The interior fittings 46 can be displayed superimposed on the virtual vehicle environment 22. This makes it possible to see through the interior fittings 46 of vehicle 10 using the display device 12.

[0052] Fig.Figure 3 shows a schematic representation of a process for displaying an augmented vehicle environment of a vehicle by means of a display device.

[0053] A first step S1 can include recording a vehicle environment 22 by a vehicle environment detection device 14 of the vehicle 10, wherein recording data 20 is provided by the vehicle environment detection device 14.

[0054] In a second step S2, a virtual vehicle environment 22 of the vehicle environment 22 can be reconstructed from the recording data 20 by a control device 24.

[0055] In a third step S3, position data 26 can be received by the control device 24, which describe a position of the display device 12 within the vehicle 10.

[0056] In a fourth step S4, the control device 24 can determine a viewing area 28 of the display device 12 based on the position of the display device 12.

[0057] In a fifth step S5, the control device 24 can generate display data 30 to display the virtual vehicle environment 22 in the field of vision 28.

[0058] In a sixth step S6, the display data 30 of the display device 32 can be provided by the control device 24.

[0059] In a seventh step S7, the field of view 28 of the display device 12 can be superimposed with the virtual vehicle environment 22 by the display device 32.

[0060] Current transparent bonnet concepts use displays, such as a head-up display, to show areas of a vehicle concealed by the bodywork. This creates the impression of a transparent hood.

[0061] Other embodiments aim to display an underbody view on a screen. However, these technologies offer only a limited view and are restricted to certain areas of the vehicle.

[0062] A limitation of these solutions is that visibility is restricted to the areas where the display is mounted. For example, the hood only appears transparent where the head-up display is positioned. If the driver moves outside the field of vision, visibility is no longer possible. When the surroundings are displayed on an in-vehicle display, the driver has to mentally transfer the information to know which area of ​​the vehicle they are currently looking at. This creates increased stress in critical situations.

[0063] The new idea aims to further develop this concept to enable a completely transparent view through the vehicle using AR, VR, or XR glasses. This means that the XR glasses should allow users to see inside the vehicle in all directions, through any part of the vehicle. This provides a more comprehensive and immersive view of the vehicle's surroundings. Vehicle occupants can intuitively perceive their immediate surroundings and locate them accurately. This helps the driver navigate precisely and allows passengers to get a sense of the vehicle's movement even without looking out the window.

[0064] The procedure can be designed as follows.

[0065] The vehicle's surround-view cameras, such as top-view cameras with wide-angle lenses, record a complete picture of the vehicle's surroundings.

[0066] The camera images are rectified and stitched together using a stitching algorithm to create a map of the environment. With the help of in-vehicle sensors, such as speedometers, yaw rate and acceleration sensors, as well as matching algorithms, the camera images are extended into the past as the vehicle moves.

[0067] A 6DOF AR or XR headset will be used for in-vehicle display. This headset locates itself within the vehicle using inside-out tracking, or using vehicle interior sensors and outside-in tracking, or a combination of both methods. The AR / VR / XR headset displays the calculated environmental map in the correct orientation relative to the vehicle, creating the impression of a transparent view.

[0068] Surround-view cameras, such as top-view cameras with wide-angle lenses, record the vehicle's surroundings. The camera images are stitched together using an algorithm to create a map of the environment. This map is displayed in AR / VR / XR glasses located inside the vehicle. Obscured areas can be revealed within the vehicle, enhancing safety and situational awareness. This creates the impression of an unobstructed view of the surroundings, thus improving the driving experience.

[0069] The technology can reduce motion sickness because the felt vehicle movement and the visually perceived vehicle movement are identical.

[0070] Similar to the transparent hood method, the surrounding images can also be projected like a map underneath the vehicle to create a simulated underbody view. This is based on the continuation of the camera images as the vehicle moves.

[0071] Another option is to add additional cameras to the topview cameras, such as underbody cameras in or under the vehicle's floor, in order to reduce the proportion of statically continued areas in the environment map and thus obtain more up-to-date views.

[0072] Overall, the examples show how a transparent all-round view can be provided in vehicles using AR / VR / XR glasses.

Claims

[1] Method for displaying an augmented vehicle environment (22) of a vehicle (10) by means of a display device (12), wherein - recording data (20) of at least one area of ​​the vehicle environment (22) are recorded by a vehicle environment detection device (14) of the vehicle (10), - a virtual vehicle environment (22) of the vehicle environment (22) is reconstructed from the recording data (20) by a control device (24), - the control device (24) receives position data (26) that describe the position of a display device (12) within the vehicle (10), - by means of the control device (24) a viewing field (28) of the display device (12) is determined based on the position of the display device (12), - the control device (24) generates display data (30) to display the virtual vehicle environment (22) in the field of vision (28), - the control device (24) provides the display data (30) to a display device (32) of the display device (12), and - the field of view of the display device (12) is superimposed on the virtual vehicle environment (22) by the display device (32), characterized by , that - ego recording data of the field of view (28) are captured by an ego camera device (42) of the display device (12), - the ego recording data are fused with the recording data (20), the fusion involving an overlay of a content of the field of view (28) captured by the display device (12) with the virtual vehicle environment (22) within the field of view (28), and - according to the display data (30) and the ego recording data, a superimposed display is shown on the display device (32) of the display device (12). [2] Method according to claim 1, characterized by , that - motion data (34) of the vehicle (10) are received by the vehicle environment detection device (14), which describe a change in a vehicle position (36) from a previous vehicle position (36) to a current vehicle position (36), and - by means of the control device (24) at least a replacement area (38) of the virtual vehicle environment (22) of the vehicle environment (22) of the current vehicle position (36) is reconstructed from the recording data (20) of the vehicle environment (22) of the previous vehicle position (36). [3] Method according to claim 2, characterized by , that at least one replacement area reconstructed from the recording data (20) of the vehicle environment (22) of the previous vehicle position (36) lies outside a detection area (18) of the vehicle environment detection device (14). [4] Method according to any one of the preceding claims, characterized by, that at least some of the position data (26) are generated by the display device (12) and provided to the control device (24). [5] Method according to any one of the preceding claims, characterized by , that at least some of the position data (26) are generated by a position detection device (40) and provided to the control device (24). [6] Method according to any one of the preceding claims, characterized by , that the display device (12) displays a display on a transparent display unit (32) of the display device (12) according to the display data (30). [7] Control device (24), configured to carry out the steps of a method according to one of the preceding claims to be carried out by a control device (24). [8] Computer program comprising program code which, when executed by a control device (24), causes the control device (24) to perform the steps of a method according to any one of claims 1 to 6. [9] Computer-readable storage medium comprising a computer program according to claim 8. [10] Vehicle (10) comprising a control device (24) according to claim 7.

Images