Method for autostereoscopic display in a vehicle and device therefor

Abstract

The invention relates to a method in which a stereo image or a sequence of stereo images with a multitude of perspectives is autostereoscopically displayed to a viewer in a vehicle in such a way that the viewer has the same impression as with a rearview mirror and the so-called convergence-accommodation conflict is avoided. The method can be supplemented by brightening the displayed stereo image in darkness. Furthermore, a device for carrying out the method and a vehicle with the device are disclosed.

Description

[0001] The invention relates to a method in which a stereo image or a sequence of stereo images with at least two perspectives is autostereoscopically displayed to a viewer by means of a display device. According to a preferred embodiment, the method is supplemented by process steps which, in darkness, brighten the displayed stereo image or sequence of stereo images for dark pixels. The method can be used, for example, in a rearview mirror replacement system or as a reversing camera system in a vehicle. Furthermore, a device that performs the method and a vehicle equipped with the device are claimed. State of the art

[0002] Conventional rearview mirrors allow for three-dimensional vision, but their placement within the vehicle is dependent on specific positions. These positions must adhere to the principle that the angle of incidence equals the angle of reflection, and the mirror's field of vision must not be obstructed by passengers, luggage, cargo, or dirty windows. Vehicles without a rear window should also be able to be equipped accordingly. Conventional rearview mirrors require adjustment to the driver's anatomy and / or seating position by rotating or tilting the mirror. This should no longer be necessary.

[0003] For the purposes of this revelation, side mirrors (sometimes also called exterior mirrors) are also rearview mirrors, and what is said about rearview mirrors also applies to the side mirror(s) / exterior mirror(s).

[0004] Conventional rearview mirrors cannot produce a brightened image when driving at night. If it's dark behind the vehicle, it will also be dark in the rearview mirror. Furthermore, the headlights of a vehicle behind can cause glare for the driver. Therefore, it's essential that a rearview mirror replacement system brightens the displayed image while simultaneously eliminating glare, for example, from the high beams of a vehicle behind.

[0005] Camera-based mirror replacement systems with displays are already in use for cars and trucks. These systems are installed as replacements for rearview mirrors or to detect a lateral blind spot, for example in trucks. However, these are not based on stereo cameras or an autostereoscopic display system.

[0006] In Abad Garcia et al. (US 2023 / 0234510 A1), a 2D rearview mirror replacement system for a side mirror is described, in which the camera's viewing direction can be controlled by the driver. This results in added convenience, which with a conventional rearview mirror can only be achieved by changing the position of the viewer's eyes relative to the mirror. However, dark areas in the image are not illuminated, and an autostereoscopic display device is not disclosed.

[0007] J. Eichenlaub (US 5457574 A) describes an autostereoscopic display device in which the displayed image is continuously adjusted to the driver's current position via head tracking. This is intended to ensure that the driver always sees a three-dimensional image without any visual field defects. Furthermore, the light rays are aligned parallel by a lens system to increase the light intensity for the driver. However, only one perspective is displayed for one pair of eyes. Brightening of dark received images is also not mentioned.

[0008] Methods have been proposed to improve night vision, which rely on an additional information source provided by thermal imaging cameras. However, these are not necessary according to the method of the invention.

[0009] In Gignac et al. (WO 2013 / 013296 A1), an autostereoscopic rearview mirror replacement system is described which uses an autostereoscopic display with two perspectives and can also display additional information for the driver. The method does not eliminate the convergence-accommodation conflict, nor can the brightness of the captured image be enhanced for the displayed image.

[0010] The object of the invention is to develop a rearview mirror replacement system that enables three-dimensional vision without the convergence-accommodation conflict and does not require adjustment to the viewer's eye position. Furthermore, as an option, it should be possible to brighten the stereo image or stereo image sequence of the autostereoscopic display device in order to give the rearview mirror replacement system improved night vision capability. Summary of the invention

[0011] The process involves capturing an image with at least two perspectives using a stereo camera or two or more cameras. This image is then analyzed to identify the disparities between the leftmost and rightmost perspectives. Next, the brightness of the captured perspectives is calculated and increased, specifically for each perspective individually and potentially by a different factor, or by a uniform factor if the captured perspectives fall below a given brightness threshold. Finally, a multitude of intermediate perspectives between the leftmost and rightmost perspectives are generated and displayed. This creates the so-called super-multiview effect, which eliminates the convergence-accommodation conflict.

[0012] The method according to the invention provides for the acquisition of a stereoscopic image using at least two simultaneously acquired images with different perspectives with one stereo camera or two or more cameras. From this, a stereoscopic image with a leftmost and a rightmost perspective is obtained and displayed on an autostereoscopic display device. - a disparity map is created by searching between the leftmost perspective and the rightmost perspective, or vice versa, for each pixel in the leftmost perspective, the corresponding pixel in the rightmost perspective, or vice versa, and occlusions are marked, and - Intermediate perspectives between the leftmost perspective and the rightmost perspective are generated using the disparity map, with more than 32 intermediate perspectives being generated, and - a multitude of different intermediate perspectives are displayed simultaneously to each eye of a viewer of the autostereoscopic display device by the autostereoscopic display device.

[0013] According to one embodiment of the method, the brightness value LH is calculated for the leftmost perspective and the brightness value RH for the rightmost perspective, where LH is the sum of all brightness values ​​of the leftmost perspective and RH is the sum of all brightness values ​​of the rightmost perspective, and if LH < S or RH < S, where S is a predetermined threshold, all pixels of the perspectives are multiplied by a scaling factor before or after the calculation of the disparity map such that the sum of their new brightness values ​​is above the threshold S.

[0014] In a further embodiment of the method, additional distance values ​​are acquired in the form of a depth map, wherein the distance values ​​are preferably acquired with at least one lidar sensor, and the depth map is used to support the calculation of the disparity map. The relationship between depth and disparity is described, for example, by Olivier Faugeras (“Three-Dimensional Computer Vision”, MIT Press, page 176, 1993). Acquiring the depth values ​​accelerates the calculation of the disparity map.

[0015] In another embodiment of the method, the position of the generated viewing zones is adjusted to the viewer's eye position so that the eye is located in the center of the viewing zone, without moving the display device. For this purpose, the method according to C. van Berkel ("Image Preparation for 3D-LCD", SPIE 1990) is applicable, for example.

[0016] The above method is also part of a device as described in claims 5 to 8. Furthermore, a vehicle with the device according to claim 9 is also provided.

[0017] The stereo camera(s) and the autostereoscopic display device are part of a vehicle. The vehicle is mobile and is, for example, driven by a driver, or a viewer uses the display device. Vehicles include, for example, motor vehicles, ships, boats, motorcycles, bicycles, helicopters, drones, trains, or tractors, each manned or unmanned.

[0018] Normally, the viewer is the driver, but it is also possible that the display device and the viewer are located outside the vehicle, e.g. because the viewer is controlling a drone or parking a car in a parking space without being in the car themselves. Detailed description of the invention

[0019] Since the invention relates to a method or device with an autostereoscopic display device on which the received stereoscopic images are displayed with a horizontal resolution of I x and vertical I y Such a display device, which can be displayed, is characterized by certain optical parameters.

[0020] At the beginning of the process, the perspective assignment to the pixels of the display device is calculated based on the optical parameters and the desired number k of perspectives to be displayed.

[0021] Once this has been done, for each stereo image received from a stereo camera or from a camera system consisting of two or more cameras with at least two perspectives, the following procedure is performed: in one step, preferably the first step, the received stereo image is analyzed to identify the disparities between the pixels of the leftmost perspective and the rightmost perspective, as well as left and right occlusions.

[0022] Various approaches for this are known in the literature. These include, for example, methods from the Heinrich Hertz Institute and the Fraunhofer Society by N. Brandenburg and P. Kauff: “Method for correspondence analysis of image features in corresponding video images in real time” (DE10105423 C1) or the Optical Flow method by BKP Horn and BG Schunck (“Determining Optical Flow”, Artificial Intelligence, 17, pp. 185-203, 1981).

[0023] However, other methods can also be used. The result of each method is always a disparity map in which corresponding image points are assigned to each other from the left perspective to the right perspective, or vice versa. Corresponding image points are those image points that can be assigned to the same spatial point from both the leftmost and the rightmost perspectives. Typically, this is achieved by selecting a sub-image consisting of n vertically and m horizontally adjacent image points around each leftmost image point B(i, j) in the leftmost perspective. The image point B(i, j) currently being processed is located at the center of this sub-image. Starting from the image point B'(i, j) in the rightmost perspective, the most similar sub-image is then searched for within a given area. The most similar corresponding image point is then defined as B(i*,j*):=minl=−tl=+t(∑r,sn,m(B(i+r,j+s)−B'(i+r+l,j+s))2 where t is the search area in B' around B'(i, j) and B(i*, j*) is the found corresponding image point of B(i, j) in B'.

[0024] Occlusions are defined such that there is no corresponding pixel in the other perspective. This process can be accelerated and simplified by incorporating a depth map generated by an external sensor alongside the stereo images. Since depth is inversely proportional to pixel disparity, this would facilitate and speed up the calculation of the disparity map. A suitable sensor for this purpose is, for example, a lidar sensor.

[0025] In a second step, the brightness of the received perspectives is calculated and this brightness value is increased if the received perspectives fall below a given brightness threshold or decreased to avoid glare effects such as those caused by headlights.

[0026] For this purpose, the sum of all brightness values ​​of the subpixels (RGB) is calculated in each perspective (left and right): LH:=Sum(Ri,j,Gi,j,Bi,j) / Bleft and RH:=Sum(Ri,j,Gi,j,Bi,j) / Bright. LH denotes the brightness of the leftmost perspective and RH the brightness of the rightmost perspective, as well as B. left the number of left pixels and B right the number of right pixels.

[0027] If either LH < S or RH < S, where S denotes a predefined brightness threshold, then scaling factors F are used. left and F right calculated: Fleft:=S / LH and Fright:=S / RH.

[0028] The corrected pixels are then calculated as follows: B(i,j):=B(i,j)*Fleft and B'(i,j):=B'(i,j)*Fright.

[0029] This ensures that both perspectives preferably have the same brightness, preventing the viewer from favoring one perspective over the other. The approach proposed here does not use a thermal imaging camera as an additional information source, but relies solely on the images from the connected stereo camera or camera systems.

[0030] This step can also be performed before calculating the disparities.

[0031] In a third step, a large number of intermediate perspectives between the leftmost perspective and the rightmost perspective are then generated and displayed.

[0032] For this purpose, based on the calculated disparity map and the matrix of perspective assignments to the subpixels of the output image created at the beginning of the process, the corresponding subpixels from the leftmost and rightmost perspectives are transferred into the output image. This is done according to the method of R.-D. Naske: “Method and device for real-time multiview generation” (WO 2009 / 082990 A1).

[0033] The number k of perspectives to be displayed must always be large enough to enable a super-multiview technology. This means that each eye simultaneously receives subpixels from a multitude of different perspectives. This creates the so-called super-multiview effect (see H. Mizushina and J. Nakamura et al., "3D-Displays reduce conflict between accommodative and vergence response," SID 1071-0922 / 16 / 2412-0520) and H. Mizushina and Y. Takaki, "Vergence and Accommodation Responses to Super Multi-View Display," Proceedings of 3DSA2013, 2013), which eliminates the convergence-accommodation conflict. However, to also achieve real-time capability, the method by R.-D. Naske (WO2009 / 082990 A1) is preferably used. In each perspective, only the subpixels that need to be displayed are calculated.A number of k > 32 simultaneously displayed perspectives can then also be handled in real time.

[0034] Another component of the method and the display device is the ability to adjust the position of the viewing zones so that the viewer is always precisely in the center of such a viewing zone. This is achieved by changing the calculated perspective assignment to the subpixels and is particularly advantageous in cases where the display device is fixed in place. It then allows the viewer, as with a mirror, to look behind objects in the foreground by moving their head left or right within the viewing zone. This capability is known as the "walk-around effect" and is only possible with displays without head tracking. Installation of the display device on an adjustable or movable mount is not required. This is a further advantage of the method according to the invention.

Claims

A method for capturing a stereoscopic image using at least two simultaneously acquired images with different perspectives with one stereo camera or two or more cameras, acquiring the stereoscopic image with a leftmost perspective and a rightmost perspective, and displaying the stereoscopic image on an autostereoscopic display device, comprising at least the following steps: - Generating a disparity map by finding, for each pixel in the leftmost perspective and the rightmost perspective, a corresponding pixel in the rightmost perspective, or vice versa, and marking occlusions; - Generating intermediate perspectives between the leftmost perspective and the rightmost perspective using the disparity map.wherein more than 32 intermediate perspectives are generated and a multitude of different intermediate perspectives are simultaneously displayed to each eye of a viewer of the autostereoscopic display device by the autostereoscopic display device, wherein at least the stereo camera or the two or more cameras is part of a mobile vehicle. The method of claim 1, wherein a brightness value LH is calculated for the leftmost perspective and a brightness value RH is calculated for the rightmost perspective, wherein LH is the sum of all brightness values ​​of the leftmost perspective and RH is the sum of all brightness values ​​of the rightmost perspective, and if LH < S or RH < S, where S is a predetermined threshold, all pixels of the leftmost perspective and the rightmost perspective are multiplied by a scaling factor before or after the calculation of the disparity map such that the sum of their new brightness values ​​is above the threshold S. Method according to claim 1 or 2, wherein distance values ​​are further recorded in the form of a depth map, wherein the distance values ​​are preferably recorded with at least one lidar sensor, and the depth map is used to support the calculation of the disparity map. Method according to at least one of claims 1 to 3, wherein the eyes of the viewer have an eye position and the viewing zones generated by the autostereoscopic display device have a position, and the position for the viewer can be adjusted to his eye position such that the eye position is in the middle of one of the generated viewing zones without moving the autostereoscopic display device. Device for displaying a supplied stereoscopic image comprising a stereo camera or two or more cameras for capturing at least two perspectives, a processor unit comprising one or more processors, and an autostereoscopic display device, wherein the processor unit is configured to: - generate a disparity map from the perspectives by searching for a corresponding pixel in the rightmost perspective for each pixel in the leftmost perspective and the rightmost perspective, or vice versa, and to identify occlusions; - generate intermediate perspectives between the leftmost perspective and the rightmost perspective using the disparity map, generating more than 32 intermediate perspectives; and - the autostereoscopic display device is configured such thatthat each eye of a viewer of the autostereoscopic display device is simultaneously shown a multitude of different intermediate perspectives by the autostereoscopic display device. Device according to claim 5, wherein the processor unit is configured to calculate a brightness value LH for the leftmost perspective and a brightness value RH for the rightmost perspective before generating the disparity map, wherein LH is the sum of all brightness values ​​of the leftmost perspective and RH is the sum of all brightness values ​​of the rightmost perspective, and, if LH < S and / or RH < S, wherein S is a predetermined threshold, to multiply all pixels of the leftmost perspective and the rightmost perspective by a scaling factor before or after calculating the disparity map, such that the sum of their new brightness values ​​is above the threshold S. Device according to at least one of claims 5 or 6, wherein the device has at least one further sensor for detecting distance values, preferably a lidar sensor, and the processor unit is configured to determine distance values ​​and provide the perspectives with a depth map which is used in the calculation of the corresponding pixels and occlusions. Device according to at least one of claims 5 to 7, wherein the eyes of the viewer have an eye position and the processor unit is configured to generate vision zones and to align the generated vision zones without movement of the autostereoscopic display device so that the eye position of the viewer is in the center of one of the generated vision zones. Vehicle with a device according to at least one of claims 5 to 8, wherein the vehicle is movable and at least the stereo camera or the two or more cameras are arranged in or on the vehicle, preferably also a processor unit and an autostereoscopic display device.

Images