Method for reducing parallax, computer program product, storage medium, data carrier signal, device and vehicle

Abstract

The present invention relates to a computer-implemented method 230 for reducing parallax that occurs when viewing display devices 105 such as transparent A-pillars or head-up displays. The eyes 140, 145 of a driver 135 of a vehicle 100 are detected, and a perspective of the right eye 335 and a perspective of the left eye 340 are calculated. Image data 275 to be displayed on the display device 105 are previously transformed into the respective perspectives 335, 340 of the eyes 140, 145. The resulting right-hand transformed image data 345 are displayed by the display device 105 so that they are perceptible only to the right eye 140. The left-hand transformed image data 350 are displayed by the display device 105 so that they are perceptible only to the left eye 145. Also disclosed are a computer program product 225, a computer-readable storage medium 215, a data carrier signal, a device 200, and a vehicle 100.

Description

Technical area

[0001] The present invention relates to the field of image processing. In particular, the present invention relates to a computer-implemented method for reducing parallax. Furthermore, the present invention relates to a computer program product, a computer-readable storage medium, a data carrier signal, a device, and a vehicle. Technical background and task

[0002] Improving safety measures for drivers is an important goal for all vehicle manufacturers. Unsafe traffic situations can arise from opaque structures in the vehicle body, which can obscure traffic-relevant events for the driver. Such opaque structures are usually parts of the vehicle body that ensure the mechanical rigidity required for occupant safety in the event of an accident. Examples of such opaque structures are the A, B, and C pillars, as well as the dashboard and trunk. These opaque structures disrupt the driver's field of vision, creating blind spots. Traffic-safety-relevant events or objects, such as a pedestrian crossing the road, a cyclist, or an overtaking car, can be obscured and may not be recognized by the driver in time.The conflicting requirements of high rigidity of the vehicle body on the one hand and a large field of vision for the driver on the other hand usually require a compromise.

[0003] Transparent A-pillars, hoods, or tailgates have been proposed as solutions. These displays, for example in the form of liquid crystal displays (LCDs) or organic light-emitting diodes (OLEDs), are placed inside the vehicle between the opaque structure and the driver. They display images from a camera located outside the vehicle, whose field of view captures the obscured area caused by the structure. Such an arrangement creates the illusion that the structure is transparent.

[0004] For example, EP 1 878 618 B1 discloses a support unit for capturing an image of an area that is not visible to a driver and that is created due to the presence of a pillar of a vehicle.

[0005] The solutions disclosed in the prior art have the disadvantage that the displayed image exhibits parallax when viewed with both eyes, as the two eyes view the display from slightly different perspectives. This effect is also known as disparity and also occurs in head-up displays (HUDs). This effect can confuse the driver, as the image is perceived as unclear or diffuse, and can even lead to headaches. It is therefore desirable to have devices and methods available that reduce the parallax effect.

[0006] It is therefore the object of the invention to provide a computer-implemented method for reducing parallax. Further objects of the invention are to provide a computer program product, a computer-readable storage medium, a data carrier signal, a device, and a vehicle. Disclosure of the invention

[0007] The object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be found in the subclaims.

[0008] According to a first aspect of the invention, a computer-implemented method for reducing parallax when viewing a display device by a driver of a vehicle comprises a step of obtaining image data generated from an image data perspective.

[0009] The vehicle can be a motor vehicle, a truck, an aircraft, a helicopter, a train, or the like. The image data originates, for example, from an exterior camera when a transparent A-pillar is to be realized. The image data perspective is often known in such cases because it originates from a camera permanently installed on the vehicle, whose position and orientation are known. In the case of a head-up display (HUD), the image data can also be artificially generated image content such as navigation arrows, text, or the like. In this case, a known perspective was used to generate the images and is therefore known.

[0010] The parallax, which is at least reduced, occurs because the driver views the display from different perspectives, namely from the perspective of their right and left eyes. The inventor recognized that parallax can be reduced or even eliminated by displaying different images for the left and right eyes. Therefore, a 2D display is not suitable for the display, but rather an autostereoscopic display such as a screen with a parallax barrier or lenticular screen, or a 3D display based on polarization filter technology, interference filter technology, or the like. However, the latter often requires glasses to function.

[0011] The method further includes a step in which the right position of the driver's right eye is obtained. This information can be generated, for example, by an eye tracker. An interior camera that can simultaneously measure distances can also serve as the basis for this information. In general, any distance imaging camera (TOF camera; English: time-of-flight) can be used for this purpose. The position is known with respect to a vehicle-fixed coordinate system, for example, with respect to the camera position or a specific point on the vehicle itself.

[0012] The method further includes a step in which a right perspective is determined from the right position. This can be achieved, for example, using the position of the corneal reflection. It is also possible to define the perspective as a direction vector from the eye to an object detected in the image data or to a point (e.g., the center) of the display device.

[0013] The method further comprises a step in which right-transformed image data is generated by transforming the image data from the image data perspective to the right-transformed perspective. Those skilled in the art are familiar with various common methods for transforming image data from one perspective to another. For example, US 10 832 372 B2 discloses a method for adapting an image based on the perspective and on the shape of a display device for a motor vehicle.

[0014] Furthermore, the method comprises a step in which a left position of a left eye of the driver is obtained.

[0015] Furthermore, the method comprises a step in which a left perspective is determined from the left position.

[0016] Furthermore, the method comprises a step in which left transformed image data is generated by transforming the image data from the image data perspective into the left perspective.

[0017] These steps are therefore analogous for the left eye to those for the right eye.

[0018] The method further includes a step in which the right-hand transformed image data and the left-hand transformed image data are displayed on the vehicle's display device in such a way that the right-hand transformed image data is perceived by the right eye and the left-hand transformed image data by the left eye. The images for the different eye positions and thus eye perspectives are thus presented to only one eye at a time. Under ideal conditions (no installation tolerances in the vehicle, error-free detection of the sensors involved, etc.), parallax no longer occurs.

[0019] In an advantageous embodiment, an object position is used to generate the image data perspective, with the object position being determined using a first depth map. The object can be detected in the image data, for example, using known object detection methods. If the image data is correlated with distance data in the form of a depth map, the object's position can be easily determined. A lidar sensor, for example, can serve as the basis for such a depth map.

[0020] In an advantageous implementation, the right and left positions are determined using a second depth map. Like the object position, the eye positions can be determined using a depth map. A point projector or an infrared sensor, for example, is suitable for this purpose.

[0021] In an advantageous embodiment, the right perspective corresponds to a right vector from the right position to the object position, and the left perspective corresponds to a left vector from the left position to the object position. In other words, the perspective corresponds to the viewing direction toward an object detected in the image data. This does not necessarily correspond to the driver's actual viewing direction.

[0022] In an advantageous embodiment, a right gaze direction of the right eye is detected, and the right perspective corresponds to the right gaze direction. A left gaze direction of the left eye is detected, and the left perspective corresponds to the left gaze direction. Here, the driver's actual gaze direction is detected, which defines the respective eye perspectives. This can be achieved, for example, using additional image recognition algorithms that can detect the corneal reflection and the eye center and extract the gaze direction from them. Such methods are known to those skilled in the art. For example, US 10 314 484 B2 discloses a system for determining a gaze direction of at least one eye of a user.

[0023] According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to execute a method as described above. The computer program product can be written in a programming language, for example, Python or C++.

[0024] According to a third aspect of the invention, a computer-readable storage medium comprises instructions that, when the program is executed by a computer, cause the computer to perform a method as described above. The computer-readable storage medium can be implemented, for example, as an SSD (solid-state disk) or as a flash memory. The computer-readable storage medium can also store other data, for example, sensor data and / or data that is (temporarily) stored during the execution of the method.

[0025] According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above. The data carrier signal can be transmitted via a cable, for example, via a CAN bus (controller area network) or Ethernet cable, or wirelessly via Wi-Fi, Bluetooth, or the like.

[0026] According to a fifth aspect of the invention, a device for reducing parallax when a driver of a vehicle views a display device comprises an evaluation unit configured to carry out a method as described above. The evaluation unit itself can, for example, comprise a processor and volatile and / or non-volatile storage media. The individual components of the evaluation unit can be implemented in a single component, for example, in a central processing unit (HPC, English: high-performance computer). However, the individual components can also be distributed decentrally; for example, storage media can be outsourced to a server.

[0027] The device further comprises an image generation unit that is communicatively connected to the evaluation unit. The image generation unit can be implemented, for example, by an external camera on an exterior mirror of the vehicle. The image generation unit can also be implemented, for example, by a computing unit that can generate artificial image data such as navigation arrows, maps, or the like.

[0028] The device also includes an interior camera that is communicatively connected to the evaluation unit and positioned in the vehicle in such a way that it can capture the driver's eyes. Examples of such interior cameras include eye trackers.

[0029] The device further comprises a display device that is communicatively connected to the evaluation unit. The display device is capable of displaying two different images simultaneously but in different directions. For example, these can be autostereoscopic displays or 3D displays.

[0030] A CAN bus or Ethernet cable, for example, is suitable for the communicative connection between the individual components of the evaluation unit.

[0031] In an advantageous embodiment, the display device corresponds to an autostereoscopy display.

[0032] In an advantageous embodiment, the display device corresponds to a head-up display.

[0033] According to a sixth aspect of the invention, a vehicle comprises a device such as the one described above. Individual components of the device can be used to provide multiple functions of the vehicle. The interior camera, for example, can also be used to detect driver fatigue. Summary of the characters

[0034] The invention is explained in more detail below using exemplary embodiments and figures. The figures show: Fig. 1: A first scene from a driver's cab to explain the desired result; Fig. 2: A plan view of a vehicle to explain the problem underlying the invention; Fig. 3: A second scene from a driver's cab to explain the problem underlying the invention; Fig. 4: An embodiment of a device for reducing parallax; Fig. 5: A flowchart of a computer-implemented method for reducing parallax; and Fig. 6: A flowchart of an embodiment of the computer-implemented method of Fig. 5. Detailed description of the characters

[0035] Fig. 1 shows a first scene from a driver’s cab to explain the desired result.

[0036] In Fig. 1 shows a scene from a vehicle 100. The vehicle 100 has a display device 105 arranged on an A-pillar. The A-pillar blocks the view of a driver 135 of the vehicle 100 of people 120 and objects 130 located behind the A-pillar from the perspective of the driver 135. The purpose of the display device 105 is to make these people 120 and objects 130 visible to the driver 135. The digital representations 125, 170, 175 displayed on the display device 105 should appear synchronous with the environment. In other words, the visible area around the display device 105 should appear to the driver 135 as if it were closely connected to it. Fig. 1, the display on the display device 105 is environmentally synchronous, there is no offset or scaling error between the digital representation of the person 125 and the person 120, which is still within the area visible to the driver 135.

[0037] The vehicle 100 has an exterior camera 110, which, with its field of view 115, at least partially captures the area invisible to the driver 135. The image data 275 from the exterior camera 110 is displayed on the display device 105. They are previously transformed, for example, to account for an approximate perspective of the driver 135 and a curvature of the display device 105.

[0038] Fig. 2 shows a plan view of a vehicle 100 to explain the problem underlying the invention.

[0039] Fig. 2 shows a scene essentially identical to Fig. 1, but from above. The driver 135 of the vehicle 100 is prevented by the A-pillar from fully viewing an object 130 or a person 120. The driver 135 has a right eye 140 and a left eye 145, which view the object 130 or the person 120 from different perspectives because they are spaced apart. This results in different distances from the right eye to the object 150 and from the left eye to the object 155. The different perspectives of the eyes 140, 145 also lead to different positions on the display device 105, where the object 130 or the person 120 would have to be displayed in order to be perceived by the respective eye 140, 145 with the correct perspective and in synchronization with the environment. For the right eye 140, an object position for the right eye 160 would have to be used, and for the left eye 145, a different object position for the left eye 165 would have to be used.However, this is not done in currently available systems. The implications of this deficiency are discussed with reference to . Fig. 3 discussed.

[0040] Fig. 3 shows a second scene from a driver's cab to explain the problem underlying the invention.

[0041] In Fig. Figure 3 shows how the display of a digital representation of the person 125 is perceived when the display device 105 is viewed with both eyes 140, 145. Due to the lack of correction for the different perspectives of the right eye 140 and the left eye 145, the digital representation for the right eye 170 and the digital representation for the left eye 175 are perceived as being spaced apart. This parallax or disparity can confuse the driver 135 or lead to headaches upon prolonged viewing.

[0042] In the following, devices 200 and methods 230 are discussed which at least lead to a reduction of this parallax.

[0043] Fig. 4 shows an embodiment of a device 200 for reducing parallax.

[0044] The device 200 comprises an external camera 110, an internal camera 180, and a display device 105. Furthermore, the device 200 comprises an evaluation unit 205. This, in turn, comprises a processor 210, a volatile memory 220, and a non-volatile memory 215. A computer program product 225 is stored on the non-volatile memory 215. The computer program product 225 comprises instructions which, when executed by the processor 210, cause the processor to perform a method 230 for reducing parallax. The computer program product 225 is written, for example, in the Python programming language. All elements are communicatively connected, for example, via an Ethernet cable.

[0045] Fig. 5 shows a flowchart of a computer-implemented method 230 for reducing parallax.

[0046] In method 230, image data 275 is obtained in a first acquisition step 235. In a second acquisition step 240, a right position 320 of a right eye 140 of a driver 135 is obtained. In a third acquisition step 245, a left position 325 of a left eye 145 of the driver 135 is obtained.

[0047] In a first determination step 250, a right perspective 335 is determined from the right position 320. In a second determination step 260, a left perspective 340 is determined from the left position 325.

[0048] In a first generation step 255, right-hand transformed image data 345 is generated. This is achieved on the basis of the image data 275 by transforming it from the image data perspective 310 to the right-hand perspective 335. Similarly, in a second generation step 265, left-hand transformed image data 350 is generated by transforming it from the image data perspective 310 to the left perspective 340.

[0049] Finally, the right transformed image data 345 and the left transformed image data 350 are displayed on a display device 105 of the vehicle 100. The right transformed image data 345 is perceived by the right eye 140, and the left transformed image data 350 is perceived by the left eye 145. By correctly considering the perspectives 335, 340 and the eye-dependent display, the parallax that would otherwise occur is at least reduced, if not completely eliminated.

[0050] Fig. 6 shows a flowchart of an embodiment of the computer-implemented method 230 of Fig. 5.

[0051] In Fig. Figure 6 shows an outdoor camera 110 that, in addition to image capture, also has distance measurement capabilities thanks to its LIDAR (light detection and ranging) system. Therefore, the outdoor camera 110 has LIDAR data 280 in addition to outdoor camera image data 275.

[0052] Furthermore, Fig. 6 shows an interior camera 180 which, in addition to image capture, also has distance measurement capabilities, as it has a dot projector. Therefore, the interior camera 180 has dot projector data 290 in addition to interior camera image data 285.

[0053] Fig. 6 shows an evaluation unit 205 with a processor 210, a volatile memory 220, and a non-volatile memory 215. The elements 210, 215, 220 of the evaluation unit 205 can communicate bidirectionally with each other if necessary. A computer program product 225 is stored on the non-volatile memory 215. This is written in one or more programming languages ​​such as Python, C, C++, or the like and describes commands that enable the method 230 to be executed. Fig. 5. The instructions are executed by the processor 210 and are explained in more detail below. Furthermore, Fig. 6 shows a display device 105. This is designed as an LCD display with a parallax barrier, so that certain pixels can only be seen from an angular range corresponding to the right eye 140 and other pixels can only be seen from an angular range corresponding to the left eye 145.

[0054] First, the lidar data 280 of the external camera 110 is processed in the processor 210 to create an external camera depth map 295. This map correlates each pixel of the external camera image data 275 with a depth. After an object is detected in the external camera image data 275, an object position 300 can be assigned to a relevant object 130, for example, a person 120. Since the external camera 110 is arranged such that an area invisible to the driver 135 of the vehicle 100 is captured, vulnerable road users or obstacles, in particular, are considered relevant objects 130.

[0055] The object position 300 is typically initially present in relation to the external camera 110, but can be converted to any other arbitrary reference using an external camera coordinate system 305, which is defined, for example, in relation to a vehicle-fixed coordinate system. The image data perspective 310 is then calculated as a vector pointing from the external camera 110 to the detected object 120 (see Fig. 2, reference numeral 131). However, the image data perspective 310 can also be assumed to be static and determined by the orientation of the external camera 110.

[0056] In parallel, the point projector data 290 are processed in the processor 210 to form an interior camera depth map 315. The head of the driver 135 and in particular his eyes 140, 145 are recorded in the interior camera image data 285. For this purpose, the interior camera 180 can, for example, be positioned as shown in Fig. 1 and Fig. 2 may be arranged in an instrument panel to capture the eyes 140, 145 with its interior camera field of view 185. Other arrangements, such as in the rearview mirror, are also feasible.

[0057] Using the interior camera depth map 315 and object detection in the interior camera image data 285, which can detect the right eye 140 and the left eye 145, a position of the right eye 320 and a position of the left eye 325 can be found. Typically, these positions are relative to an interior camera coordinate system 330. However, its relationship to a vehicle-fixed coordinate system is typically known, so it can be converted to another coordinate system if necessary.

[0058] From the position of the right eye 320, a perspective of the right eye 335 can now be determined. This can be done, for example, by simple geometric considerations, in which a vector is calculated from the position of the right eye 320 to the detected object 130. In the example of Fig. 6, however, a corneal reflection is detected in the interior camera image data 285. This can be used to determine the perspective of the right eye 335, which is known to the person skilled in the art, for example, from US 2019 / 0206082 A1.

[0059] Analogously, a perspective of the left eye 340 is determined from the position of the left eye 325.

[0060] The external camera image data 275 is now prepared for display for the right eye 140. For this purpose, it is transformed from the image data perspective 310 into the perspective of the right eye 335 using mathematical transformations such as cropping, rotating, twisting, and the like. The result is right-transformed image data 345. Left-transformed image data 350 is also created analogously.

[0061] Subsequently, the right transformed image data 345 and the left transformed image data 350 are displayed on the display device 105. This occurs in such a way that those pixels of the display device 105 that are visible only to the right eye 140 through the parallax barrier display the right transformed image data 345. The pixels that are visible only to the left eye 140 through the parallax barrier display the left transformed image data 350. List of reference symbols 100 vehicles 105 Display device 110 outdoor camera 115 field of view outdoor camera 120 people 125 Digital representation of the person 130 objects 131 Distance from external camera to object 135 drivers 140 Right Eye 145 Left Eye 150 Distance right eye to object 155 Distance left eye to object 160 Object position for right eye 165 Object position for left eye 170 Digital representation for right eye 175 Digital representation for left eye 180 interior camera 185 field of view interior camera 190 Distance right eye to interior camera 195 Distance left eye to interior camera 200 device 205 Evaluation unit 210 processor 215 Non-volatile memory 220 Volatile Memory 225 Computer program product 230 procedures 235 First conservation step 240 Second conservation step 245 Third conservation step 250 First determination step 255 First generation step 260 Second determination step 265 Second generation step 270 display steps 275 image data outdoor camera 280 Lidar data 285 Image data interior camera 290 dot projector data 295 Depth map outdoor camera 300 Object Position 305 Outdoor camera coordinate system 310 Image Data Perspective 315 Depth map interior camera 320 Position right eye 325 Position left eye 330 Interior camera coordinate system 335 perspectives right eye 340 perspectives left eye 345 Rights transformed image data 350 Left transformed image data QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 1 878 618 B1

[0004] US 10 832 372 B2

[0013] US 10 314 484 B2

[0022] US 2019 / 0206082 A1

[0058]

Claims

[1] Computer-implemented method (230) for reducing parallax when viewing a display device (105) by a driver (135) of a vehicle (100), comprising the steps: a) Obtaining (235) image data (275) generated from an image data perspective (310), b) Maintaining (240) a right position (320) of a right eye (140) of the driver (135), c) Determining (250) a right-wing perspective (335) from the right-wing position (320), d) Generation (255) of right transformed image data (345) by transforming the image data (275) from the image data perspective (310) into the right perspective (335), e) Maintaining (245) a left position (325) of a left eye (145) of the driver (135), f) Determining (260) a left perspective (340) from the left position (325), g) Generation (265) of left transformed image data (350) by transforming the image data (275) from the image data perspective (310) into the left perspective (340), and h) Display (270) the right transformed image data (345) and the left transformed image data (350) on the display device (105) of the vehicle (100) such that the right transformed image data (345) is perceived by the right eye (140) and the left transformed image data (350) is perceived by the left eye (145). [2] Computer-implemented method according to claim 1, characterized in that an object position (300) is used to generate the image data perspective (310), wherein the object position (300) is determined by means of a first depth map (295). [3] Computer-implemented method according to claim 1 or 2, characterized in that the right position (320) and the left position (325) are determined using a second depth map (315). [4] Computer-implemented method according to claim 2, characterized in that the right perspective (335) corresponds to a right vector from the right position (320) to the object position (300) and the left perspective (340) corresponds to a left vector from the left position (325) to the object position (300). [5] Computer-implemented method according to one of the preceding claims, characterized in that a right gaze direction of the right eye (140) is detected and the right perspective (335) corresponds to the right gaze direction and that a left gaze direction of the left eye (145) is detected and the left perspective (340) corresponds to the left gaze direction. [6] Computer program product (225) comprising instructions which, when the program is executed by a computer, cause it to execute a method (230) according to one of the preceding claims [7] Computer-readable storage medium (215) comprising instructions which, when the program is executed by a computer, cause it to execute a method (230) according to any one of claims 1 to 5. [8] Data carrier signal which transmits the computer program product (225) according to claim 6. [9] Device (200) for reducing parallax when viewing a display device (105) by a driver (135) of a vehicle (100), comprising: a) An evaluation unit (205) which is configured to execute a method (230) according to any one of claims 1 to 5, b) An image generation unit (110) which is communicatively connected to the evaluation unit (205), c) An interior camera (180), which is communicatively connected to the evaluation unit (205), arranged in the vehicle (100) in such a way that it can capture the eyes (140, 145) of the driver (135), and d) A display device (105) which is communicatively connected to the evaluation unit (205). [10] Device according to claim 9, characterized in that the display device (105) corresponds to an autostereoscopy display. [11] Device according to claim 9, characterized in that the display device (105) corresponds to a head-up display. [12] Vehicle (100) which has a device (200) according to any one of claims 9 to 11.

Images