Electronic mirror system and electronic mirror display method

Abstract

An electronic mirror system comprises: an electronic mirror including a display element and a microlens array provided on the front surface of the display element; an imaging device for capturing an image of a subject to be displayed on the display element; a sensor for acquiring position information on both eyes of a person viewing the electronic mirror; and a control unit for generating an image to be displayed on the display element on the basis of the image of the subject captured by the imaging device. The control unit generates an image to be displayed on the display element so as to form an image of the subject at a focal position deeper than the display element. The image to be displayed on the display element includes a right-eye image formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and a left-eye image formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

Description

Electronic mirror system and electronic mirror display method

[0001] This application claims priority to Japanese Patent Application No. 2023-140535, filed on August 30, 2023, the contents of which are incorporated herein by reference.

[0002] In-vehicle electronic mirrors have been known as devices for enabling drivers to check the road ahead. Unlike the previously mainstream optical inner mirrors, in-vehicle electronic mirrors display images from a camera installed at the rear of the vehicle to capture the rear view on a display device mounted in a position visible to the driver. Optical inner mirrors have a problem with limited visibility due to obstructions from vehicle structures such as C-pillars and D-pillars. However, in-vehicle electronic mirrors can ensure sufficient rear visibility. Furthermore, in-vehicle electronic mirrors automatically reduce glare, ensuring rear visibility even at night without being dazzled by the lights of vehicles behind. Such in-vehicle electronic mirrors can be considered safe driving support devices that support safe driving.

[0003] In-vehicle electronic mirrors are typically installed 60 to 80 cm (centimeters) from the driver's viewpoint. Middle-aged and elderly drivers (i.e., drivers with presbyopia) who have difficulty adjusting their diopter settings widely face the problem of being unable to focus on an electronic mirror installed at such a position, even before ensuring a clear rearward view. In compact vehicles, electronic mirrors are sometimes installed even closer, making the problem of being unable to focus on an electronic mirror even more pronounced for middle-aged and elderly drivers who have difficulty adjusting their diopter settings widely. Therefore, various electronic mirror manufacturers have been studying ways to improve the distance viewing angle of electronic mirrors, but no effective method has yet been proposed. Patent Documents disclosing technologies for improving the visibility of electronic mirrors include, for example, Patent Document 1 and Patent Document 2.

[0004] JP 2021-162431 A JP 2022-178013 A

[0005] According to the technology described in Patent Document 1, a concave mirror is used as a display unit that displays an image from a camera installed at the rear of the vehicle, thereby displaying a virtual image in front of the vehicle. Adopting such a configuration may contribute to improving visibility. However, the technology described in Patent Document 1 has the problem of significantly increasing the size of the housing.

[0006] Furthermore, according to the technology described in Patent Document 2, an in-vehicle electronic mirror like an optical mirror is provided by superimposing an image from a camera installed at the rear of the vehicle and an image from the rear of the vehicle cabin. However, the image displayed by this method is a real image, and there is a problem in that it is not possible to display a virtual image as a distant viewpoint.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an electronic mirror system and an electronic mirror display method that are easily visible even when installed close to the driver where wide visibility adjustment is difficult.

[0008] [1] One aspect of this embodiment is an electronic mirror system comprising: an electronic mirror having a display element and a microlens array provided in front of the display element; an imaging device that captures an image of a subject to be displayed on the display element; a sensor that acquires position information of both eyes of a person viewing the electronic mirror; and a control unit that generates an image to be displayed on the display element based on the image of the subject captured by the imaging device, wherein the control unit generates an image to be displayed on the display element so that the image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

[0009] [2] Furthermore, one aspect of this embodiment is that in the electronic mirror system described in [1] above, the image of the subject captured by the imaging device includes a common area used to generate the image for the right eye and the image for the left eye, a right-eye image area that is used to generate the image for the right eye but not used to generate the image for the left eye, and a left-eye image area that is used to generate the image for the left eye but not used to generate the image for the right eye.

[0010] [3] Furthermore, one aspect of this embodiment is that in the electronic mirror system described in [2] above, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the image for the right eye, and the number of pixels in the height direction of the image for the left eye are all the same, and the control unit determines the number of pixels in the width direction of the image area for the right eye and the number of pixels in the width direction of the image area for the left eye based on position information of both eyes of the person viewing the electronic mirror.

[0011] [4] Furthermore, one aspect of this embodiment is an electronic mirror system as described in any one of [1] to [3] above, wherein the imaging device images the direction opposite to the direction of travel of the moving body, and the electronic mirror is fixed in a position where the operator of the moving body can confirm the direction opposite to the direction of travel.

[0012] [5] Furthermore, one aspect of this embodiment is an electronic mirror display method for displaying an image of a subject on an electronic mirror having a display element and a microlens array provided in front of the display element, the electronic mirror display method comprising: an imaging step of imaging the subject to be displayed on the display element; an acquisition step of acquiring position information of both eyes of a person viewing the electronic mirror; and a control step of generating an image to be displayed on the display element based on the image of the subject captured in the imaging step, wherein the control step generates an image to be displayed on the display element so that the image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

[0013] According to this embodiment, it is possible to provide an electronic mirror system and an electronic mirror display method that are easily visible even when installed in a location close to the driver where wide visibility adjustment is difficult.

[0014] 1 is a diagram for explaining an outline of the configuration of an electronic mirror system according to the present embodiment. FIG. 2 is a diagram for explaining an outline of the configuration of an electronic mirror according to the present embodiment. FIG. 3 is a first diagram for explaining an outline of the light field optical technology adopted in the electronic mirror according to the present embodiment. FIG. 4 is a second diagram for explaining an outline of the light field optical technology adopted in the electronic mirror according to the present embodiment. FIG. 5 is a diagram comparing the fields of view of an optical mirror and an electronic mirror to explain the left and right fields of view of the electronic mirror according to the present embodiment. FIG. 6 is a diagram for explaining changes in the eye point and the field of view in a display method for an electronic mirror according to the present embodiment. FIG. 7 is a diagram showing an example of a change in the rear view field due to a change in the field of view in a display method for an electronic mirror according to the present embodiment. FIG. 8 is a diagram for explaining clipping of an image captured by an imaging device according to the present embodiment. FIG. 9 is a functional configuration diagram showing an example of the functional configuration of an electronic mirror system according to the present embodiment. FIG. 10 is a first diagram for explaining an example of calculation that takes into account the tilt of the mirror for the electronic mirror system according to the present embodiment. FIG. 11 is a second diagram for explaining an example of calculation that takes into account the tilt of the mirror for the electronic mirror system according to the present embodiment. FIG. 12 is a diagram for explaining a modified example of the electronic mirror system according to the present embodiment.

[0015] A preferred embodiment of a digital mirror system and a digital mirror display method according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is merely an example, and the embodiments to which the present invention is applicable are not limited to the following embodiments. Furthermore, the term "based on XX" in this application means "based on at least XX" and includes cases where the system is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is directly used, but also includes cases where the system is based on XX after calculation or processing. "XX" is an arbitrary element (e.g., arbitrary information). Furthermore, in the following drawings, the scale and number of elements in each structure may differ from the scale and number of elements in the actual structure in order to make each configuration easier to understand.

[0016] [Embodiment] Fig. 1 is a diagram for explaining an outline of the configuration of an electronic mirror system according to an embodiment. First, the configuration of the electronic mirror system 1 will be briefly explained with reference to the diagram. The electronic mirror system 1 includes at least an image capture device 2, an electronic mirror 3, and a vehicle 4. The electronic mirror system 1 transmits an image of the situation behind the vehicle 4 to a driver 5 who is driving the vehicle 4 by displaying an image based on a video signal captured by the image capture device 2 on the electronic mirror 3 in real time.

[0017] Here, when the driver 5 is young, the crystalline lens of the eye has sufficient elasticity and is capable of adjusting the diopter over a wide range. Therefore, even if the digital mirror 3 is installed close to the driver 5, the young driver 5 can see the image displayed on the digital mirror 3. However, when the driver 5 reaches a certain age (for example, in his / her 40s or older), the elasticity of the crystalline lens hardens, making it difficult to adjust the diopter over a wide range (i.e., he / she develops presbyopia).

[0018] In order to enable even such a driver 5 to easily view the image displayed on the electronic mirror 3, it is conceivable to install the electronic mirror 3 at a position far from the driver 5. However, there may be limitations on the location where the electronic mirror 3 is installed due to vehicle space. Since the driver 5 drives while looking ahead of the vehicle 4, the electronic mirror 3 needs to be installed in front of the driver 5. However, in the case of a vehicle 4 in which sufficient vehicle space cannot be secured, such as a compact car, the electronic mirror 3 needs to be installed close to the driver 5, and it may be difficult to secure a sufficient distance between the electronic mirror 3 and the driver 5.

[0019] When driving a vehicle, which requires quick decisions, it is extremely dangerous if the driver is unable to instantly grasp the situation behind the vehicle 4. Therefore, in this embodiment, the electronic mirror system 1 provides an image that can be easily viewed by the driver 5, even if the electronic mirror 3 is installed close to the driver 5.

[0020] In the illustrated example, the vehicle 4 is an automobile moving on land. However, the electronic mirror system 1 according to this embodiment is not limited to the example of application to the vehicle 4. The electronic mirror system 1 according to this embodiment can be applied to a wide range of moving objects operated by a driver 5. The moving object operated by the driver 5 may be, for example, an airplane or helicopter moving in the air, a ship moving on the water, or a submarine moving underwater.

[0021] The imaging device 2 captures an image of a subject to be displayed on the electronic mirror 3. Specifically, the imaging device 2 is a camera including an image sensor capable of capturing video. The imaging device 2 is installed in a position where it can capture an image in the direction opposite to the traveling direction of the vehicle 4 (or moving object). An example of a position where it can capture an image in the direction opposite to the traveling direction is the rear of the vehicle 4, as shown in the figure. The imaging device 2 is fixedly installed in a position where it can capture an image in the direction opposite to the traveling direction of the vehicle 4. Installing the imaging device 2 in such a position solves the problem of the optical inner mirror, where the field of view is obstructed by vehicle structures such as C-pillars and D-pillars, resulting in a narrow field of view.

[0022] The electronic mirror 3 is fixed in a position where the driver 5 of the vehicle 4 (or the operator of the mobile body) can simultaneously check the direction in which the vehicle 4 is traveling and the direction opposite to the direction of travel. The electronic mirror 3 is typically installed above the windshield of the vehicle 4 (where an optical inner mirror has conventionally been installed) and faces the driver 5. By installing the electronic mirror 3 in such a position, the driver 5 can simultaneously view the direction in which the vehicle 4 is traveling and the direction opposite to the direction of travel.

[0023] 2 is a diagram for explaining an outline of the configuration of the digital mirror according to this embodiment. The outline of the configuration of the digital mirror 3 will be explained with reference to the same figure. The digital mirror 3 includes at least an ECU (Electronic Control Unit) 31, a TFT (Thin Film Transistor Liquid Crystal) driver 32, a liquid crystal display panel 33, and a wedge-shaped reflector 34.

[0024] The ECU 31 acquires a video signal of the area behind the vehicle from the imaging device 2. The ECU 31 performs image processing on the acquired video signal to generate a video signal to be displayed to the driver 5. The ECU 31 controls the entire electronic mirror system 1. In the following description, the ECU 31 may also be referred to as a control unit.

[0025] The TFT driver 32 drives the liquid crystal display panel 33 to display the video signal generated by the ECU 31. Specifically, the TFT driver 32 controls the display by applying a voltage to each display cell of the liquid crystal display panel 33 based on the video signal generated by the ECU 31. The TFT driver 32 converts the constant voltage video signal output from the ECU 31 into a voltage level required by the liquid crystal display panel 33 and supplies the voltage to the liquid crystal display panel 33.

[0026] The liquid crystal display panel 33 includes a TFT panel 331, a backlight 332, and a microlens array 333. The TFT panel 331 and the backlight 332 are driven by the TFT driver 32. The TFT panel 331 includes a plurality of display cells (display elements) (not shown) and displays an image based on a video signal generated by the ECU 31. The backlight 332 guides or diffuses light obtained by emission of light-emitting diodes (LEDs) (not shown) to generate a surface light source. The generated surface light source illuminates the liquid crystal from behind the TFT panel 331, thereby displaying an image to the driver 5.

[0027] The microlens array 333 is disposed close to the image emission side of the TFT panel 331. The microlens array 333 has a plurality of microlenses arranged in a grid pattern. Each microlens corresponds to one or more display cells provided in the TFT panel 331. The microlens array 333 controls the light distribution of the video signal displayed on the TFT panel 331, and provides a suitable image to the driver 5. The pitch, aperture, lens focal length, etc. of the microlens array 333 are designed so that the light is selectively distributed so that the focus is at the appropriate position.

[0028] The wedge-shaped reflecting mirror 34 is configured as part of the mirror housing so as to seal the components contained in the liquid crystal display panel 33, including the TFT panel 331, backlight 332, and microlens array 333. The wedge-shaped reflecting mirror 34 has a structure covered with a wedge-shaped glass mirror having different thicknesses at the top and bottom. The wedge-shaped reflecting mirror 34 is half-mirror coated on the microlens array side, so that when the TFT panel 331 is in a non-driven state, the electronic mirror 3 can be used as a normal optical mirror.

[0029] According to this embodiment, by using a liquid crystal display panel 33 that requires a backlight 332, it is possible to selectively use the electronic mirror 3 as either a normal optical mirror or a conventional electronic mirror. However, this embodiment is not limited to the example of using a liquid crystal, and a wide variety of other display methods can be adopted. For example, instead of the liquid crystal display panel 33 that uses a backlight, a self-luminous organic EL display or the like may be used. When a self-luminous organic EL display or the like is used, it is possible to select whether the electronic mirror 3 is used as a normal optical mirror or a conventional electronic mirror by controlling the on / off of the light emission.

[0030] The electronic mirror system 1 according to this embodiment employs the above-described configuration and uses light field optical technology to provide a display screen that is easily visible even when installed close to the driver 5. The principle of distant viewpoint using light field optical technology will be described with reference to FIGS. 3 and 4.

[0031] FIG. 3 is a first diagram illustrating an outline of the light field optical technology employed in the digital mirror according to this embodiment. The TFT panel 331 includes multiple display elements 35. The display elements 35 may be light modulation elements for optically modulating liquid crystal. The multiple display elements 35 are arranged corresponding to color filters for color display. The display element 35 corresponding to the red color filter is referred to as the red display element 35R, the display element 35 corresponding to the blue color filter is referred to as the blue display element 35B, and the display element 35 corresponding to the green color filter is referred to as the green display element 35G. These display elements 35 may be arranged, for example, in a stripe array or a mosaic array. The same figure shows, as an example, a set of the red display element 35R, the blue display element 35B, and the green display element 35G.

[0032] The orientation of light rays is controlled by a microlens array 333 arranged on the light emission surface of the TFT panel 331. As a result of controlling the orientation of light rays, the image displayed on the TFT panel 331 is observed as a virtual image. The TFT panel 331 does not display the image from the camera as is, but displays the image so that the image from the camera is observed as a virtual image.

[0033] FIG. 4 is a second diagram for explaining an outline of the light field optical technology employed in the electronic mirror according to this embodiment. As shown in the figure, the light field element image displayed on the TFT panel 331 reaches the lens 71 in the eyeball 72 of the driver 5 as a result of selectively controlling the light distribution by the microlens array 333. As shown in the figure, the image displayed by the electronic mirror 3 is formed as a virtual image at a far-point display point 61. For the sake of explanation, the image formed by the red display element 35R is referred to as a far-point red display point 61R, the image formed by the blue display element 35B as a far-point blue display point 61B, and the image formed by the green display element 35G as a far-point green display point 61G. After reaching the eyeball 72 of the driver 5, these light rays are focused on the retina. The image focused on the retina is a virtual image whose focus is not on the TFT panel 331 of the electronic mirror 3, but on the far-point display point 61, which is a point further away from the back surface of the electronic mirror 3.

[0034] According to this embodiment, such light field optical technology is used to achieve a far-point view. Therefore, the image displayed on the TFT panel 331 is aligned with the position of the eyeballs 72 of the driver 5. The positions of the eyeballs 72 of the driver 5 are different for the right eye and the left eye. Therefore, according to this embodiment, images are formed on the right eye and the left eye, respectively, so the image signal for the right eye and the image signal for the left eye are different.

[0035] Next, the difference between a normal optical mirror and an electronic mirror will be briefly described with reference to Fig. 5. Fig. 5 is a diagram comparing the fields of view of an optical mirror and an electronic mirror to explain the left and right fields of view of the electronic mirror according to this embodiment. Fig. 5(A) is a diagram showing the difference between the left and right views of an optical mirror. Meanwhile, Fig. 5(B) is a diagram showing the field of view of an electronic mirror.

[0036] As shown in Figure 5(A), in a normal optical mirror, the rear view of the vehicle that can be seen through the optical inner mirror 91 appears to be the same image with both eyes, but in fact the left and right eyes see different images. In the same figure, the rear image seen by the right eye through the mirror is shown as the rear view right eye field of view FR. Also, the rear image seen by the left eye through the mirror is shown as the rear view left eye field of view FL. In this way, the driver 5 may think that they are seeing the same rear view with both eyes, but in fact the images are different. Specifically, the right eye sees an image shifted to the rear left by the difference G1 between the left and right fields of view, and the left eye sees an image shifted to the rear right by the difference G2 between the left and right fields of view.

[0037] As shown in FIG. 5B, a conventional electronic mirror 92 displays an image captured by a camera installed at the rear of the vehicle directly on a TFT panel 93. Therefore, the driver 5 sees the same image whether he or she looks at the electronic mirror 92 with his or her right eye or left eye. The electronic mirror system 1 according to this embodiment is an electronic mirror, but is capable of individually displaying the differences in the visual fields of both eyes obtained by an optical mirror. According to this embodiment, by adopting such a configuration, it is possible to achieve the visibility of an optical mirror with an electronic mirror.

[0038] FIG. 6 is a diagram for explaining the change in eye point and field of view in the display method of the digital mirror according to this embodiment. The change in eye point and field of view will be explained with reference to the same figure, and the concept of convergence will also be explained. In the illustrated example, a linear system without reflection is used to simplify the explanation. However, when considering reflection by the digital mirror 3, it is possible to adapt the specular reflection condition.

[0039] Here, when the eye point (pupil) of the driver 5 shifts, the field of view changes. In the example shown in the figure, the position where the driver 5 shifts to the right is shown as driver 5R, and the position where the driver 5 shifts to the left is shown as driver 5L. At this time, the change in the field of view of the driver 5 is dominated by the diameter of the electronic mirror 3, and therefore is optically called a field stop. In this embodiment, by displaying rear view images (fields of view) corresponding to the left and right eyes at the position of this field stop, the electronic mirror 3 can ensure visibility similar to that of an optical mirror, even though it is a so-called electronic mirror.

[0040] Here, if light field technology is simply used to display images with different fields of view to each eye without any appropriate standard, and the image displayed on the electronic mirror 3 is displayed as a virtual image in the distance, the focal position due to the far viewpoint and the far viewpoint position due to binocular parallax may differ, resulting in poor fusion during binocular vision. If fusion does not proceed smoothly in this way, the driver 5 may experience physiological disorders such as dizziness. Experiencing physiological disorders such as dizziness while driving a vehicle 4 is extremely dangerous. This phenomenon of poor fusion during binocular vision is known as a convergence accommodation conflict. A convergence accommodation conflict occurs when the focal position and binocular parallax (convergence) do not match.

[0041] On the other hand, when an optical mirror is used, the visual fields of the left and right eyes change via the optical mirror, but the point that serves as the reference for the visual field (the central image when viewed from the center of the eyepoint) has a consistent convergence relationship, so no convergence accommodation conflict occurs (i.e., there is no physiological discomfort). Therefore, in the display method for the electronic mirror 3 according to this embodiment, not only the visual field at each eyepoint but also the visual field reference (in this case, the image center) is controlled so that convergence conflict does not occur. Specifically, when the electronic mirror 3 displays an image, it shifts left and right as the eyepoint shifts.

[0042] 6 shows an example in which the distance between the driver 5 and the digital mirror 3 is 600 mm (millimeters). Consider a case in which a virtual image is displayed at a position 1600 mm away from the driver 5 under such circumstances. When the interpupillary distance of the driver 5 is 65 mm, the convergence angle is 2.33° (degrees), as shown in the figure. In this case, by displaying images shifted by approximately 40.6 mm to the left and right eyes of the driver 5 on the digital mirror 3, it is possible to provide images with visibility similar to that of an optical mirror without any convergence accommodation conflicts.

[0043] 7 is a diagram showing an example of a change in rear visibility with a change in the field of view in the electronic mirror display method according to this embodiment. Next, with reference to this figure, we will explain how to handle an image of the rear of the vehicle 4 actually captured by the image capture device 2 and the field of view image with a shift in the eye point. As described above, the electronic mirror system 1 according to this embodiment displays a field of view that changes with a shift in the eye point. This "changing field of view" is actually obtained by changing the area cut out from the image of the rear of the vehicle 4 captured by the image capture device 2, thereby obtaining a field of view image for each eye.

[0044] The figure shows a field of view V1, which is the field of view from the center of the eye point in the image behind vehicle 4 captured by the imaging device 2. Vehicles C1, C2, C3, and C4 are captured in field of view V1. This field of view V1 is a field of view restricted from the eye point by a field diaphragm. Note that the figure also shows a line serving as a field of view reference with a dashed line. This field of view reference is the center of the field of view from the center of the eye point. As the eye point shifts from the center of the eye point, the field of view also changes left and right. At this time, even if the field of view changes due to eye point shift, convergence is managed using this field of view reference.

[0045] FIG. 8 is a diagram for explaining clipping of an image captured by an imaging device according to this embodiment. Clipping of an image actually captured by the imaging device 2 will be described with reference to the same figure. FIG. 8(A) is a diagram showing an example of an image of the rear of a vehicle 4 captured by the imaging device 2. FIG. 8(B) is an image cropped from the image shown in FIG. 8(A) as the field of view of the right eye. FIG. 8(C) is an image cropped from the image shown in FIG. 8(A) as the field of view of the left eye. As shown in the figure, the electronic mirror system 1 clips images corresponding to the field of view from each eye point from the image of the rear of the vehicle 4 captured by the imaging device 2, and generates light field element images from each clipped image so that the light distribution from the electronic mirror 3 matches the eye point (and each field of view image). This processing makes it possible to realize an electronic mirror 3 that is compatible with a far viewpoint and convergence accommodation.

[0046] 9 is a functional configuration diagram showing an example of the functional configuration of the electronic mirror system according to this embodiment. With reference to the diagram, more specific implementation means of the above-described electronic mirror system 1 will be described. The electronic mirror system 1 includes at least an image capture device 2, a sensor 21, an ECU 31, a TFT driver 32, and a liquid crystal display panel 33.

[0047] The imaging device 2 captures an image of the area behind the vehicle 4. The imaging device 2 outputs a video signal obtained by capturing the image to the ECU 31.

[0048] The sensor 21 acquires position information of both eyes of a person viewing the electronic mirror 3 (in this example, the driver of the vehicle 4). The position information of both eyes may be the relative coordinates of the left eye and the right eye relative to the electronic mirror 3. If the distance between the right eye and the left eye is known, the sensor 21 may be configured to acquire position information of only one of the eyes. Note that conventional technology may be used to acquire the position information of both eyes. For example, the electronic mirror 3 may be equipped with an in-camera (not shown) installed in a position capable of capturing an image of the face of the person viewing the electronic mirror 3, and the position information of both eyes of the person viewing the electronic mirror 3 may be acquired by image processing the image captured by the in-camera. In this case, the sensor 21 may be configured to include an in-camera and an image processing unit.

[0049] At night, it may not be possible to obtain a sufficiently bright image of the face of the driver 5. Therefore, when acquiring position information of the eyes of a person viewing the digital mirror 3 based on an image, use at night may be restricted. Therefore, instead of image information acquired by the imaging device 2, position information of the eyes of a person viewing the digital mirror 3 may be acquired by processing a distance image acquired by a ToF (Time of Flight) sensor, LiDAR (Light Detection and Ranging), or the like, which is not affected by ambient brightness.

[0050] The ECU 31 generates an image to be displayed on the electronic mirror 3 based on the image of the subject captured by the imaging device 2. Specifically, the ECU 31 performs calculations based on light field technology to generate an image to be displayed on the electronic mirror 3 so that the image of the subject captured by the imaging device 2 is focused at a focal position behind the electronic mirror 3. The images generated by the ECU 31 and displayed on the electronic mirror 3 include an image for the right eye and an image for the left eye. The image for the right eye is an image that is focused at the focal position when viewed from the position of the right eye of a person viewing the electronic mirror 3. The image for the left eye is an image that is focused at the focal position when viewed from the position of the left eye of a person viewing the electronic mirror 3.

[0051] As described with reference to FIG. 8 , the right-eye image and the left-eye image are generated by clipping (cutting out from) one image. As shown in FIG. 8 (A), the image of the subject captured by the imaging device 2 includes a common area, a right-eye image area, and a left-eye image area. The common area is an area commonly used for generating the right-eye image and the left-eye image. The right-eye image area is an area used for generating the right-eye image but not for generating the left-eye image. The left-eye image area is an area used for generating the left-eye image but not for generating the right-eye image. In other words, the right-eye image and the left-eye image are shifted from each other in the left-right direction, not in the up-down direction. In other words, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the right-eye image, and the number of pixels in the height direction of the left-eye image are all the same, and all images are acquired from the same coordinates in the height direction.

[0052] The shift amount between the right-eye image and the left-eye image is calculated from the eye points of both eyes of the person viewing the electronic mirror 3. In other words, the ECU 31 determines the number of pixels in the width direction of the right-eye image area and the number of pixels in the width direction of the left-eye image area based on position information of both eyes of the person viewing the electronic mirror 3. In this embodiment, when the electronic mirror 3 is used to check the rear of the vehicle 4, it is not easy to change the positional relationship between the electronic mirror 3 and the driver 5. Therefore, the electronic mirror 3 changes the image it displays to match the position of the driver's 5 eyes.

[0053] Returning to FIG. 9, the TFT driver 32 displays an image to the driver 5 by outputting a drive signal based on the right-eye image and the left-eye image generated by the ECU 31 to the liquid crystal display panel 33 .

[0054] In the description given with reference to FIG. 5, the electronic mirror 3 is installed facing the driver 5. However, in an actual vehicle 4, the electronic mirror 3 is often installed to the upper left of the driver 5, and the electronic mirror 3 is often installed at an angle relative to the driver 5. An example of a calculation method for when the electronic mirror 3 is installed at an angle relative to the driver 5 will be described below with reference to FIGS. 10 and 11. Note that in the description given with reference to these figures, the positional relationship between the electronic mirror 3 and the driver 5 may be described with reference to an x-y two-dimensional coordinate system.

[0055] FIG. 10 is a first diagram illustrating an example of calculation that takes into account the tilt of the mirror for the electronic mirror system according to this embodiment. As shown in the figure, the electronic mirror 3 is installed at an angle relative to the driver 5. The tilt of the electronic mirror 3 at this time will be described with reference to FIG. 11. FIG. 11 is a second diagram illustrating an example of calculation that takes into account the tilt of the mirror for the electronic mirror system according to this embodiment. As shown in the figure, the distance in the x-axis direction from the driver 5 to the center of the electronic mirror 3 is defined as y. Furthermore, the distance in the x-axis direction from the center of the electronic mirror 3 to the right edge of the electronic mirror 3 is defined as y'. Similarly, the distance in the x-axis direction from the center of the electronic mirror 3 to the left edge of the electronic mirror 3 is defined as y'. Furthermore, the length (mirror size) of the electronic mirror 3 in the horizontal direction (longitudinal direction) is defined as m. Furthermore, the tilt angle of the electronic mirror 3 with respect to the x-axis direction is defined as β.

[0056] In this case, the calculation can be made as y'=(m / 2) sin β. Furthermore, the distance m' in the x-axis direction at which the real image of the digital mirror 3 is projected can be calculated as m'=mcos β.

[0057] Returning to FIG. 10, if the position of the right eye of the driver 5 is +x from the center and the position of the left eye is −x from the center, then the angles θ′, θ″, ω′, and ω″ can be expressed using the following equations (1) to (4).

[0058]

[0059]

[0060]

[0061]

[0062] Using the angle information obtained in this way, the viewing angle on the left side of the figure can be calculated as δθ = θ'' - θ'. The viewing angle on the right side of the figure can be calculated as δω = ω'' - ω'.

[0063] [Modification of the embodiment] Fig. 12 is a diagram for explaining a modification of the electronic mirror system according to the present embodiment. With reference to the same figure, an electronic mirror system 1A, which is a modification of the electronic mirror system 1, will be explained. The electronic mirror system 1A is applied not to a vehicle 4 but to a smartphone, tablet terminal device, notebook computer, etc. (hereinafter referred to as a smartphone, etc.). In this case, the electronic mirror 3A is a smartphone, etc. The smartphone is equipped with an imaging device 2A as an in-camera that captures an image from the side where the main display unit is located.

[0064] The digital mirror 3A is used by a user 5A. The user 5A captures an image of himself / herself using an imaging device 2A, which is an in-camera of a smartphone or the like (i.e., the user 5A takes a selfie or a self-portrait). The digital mirror system 1A uses the light field technology described above to display the captured image of the user 5A on the display of the smartphone or the like. At this time, the digital mirror 3A generates an image for the right eye and an image for the left eye according to the position of the eyepoint of the user 5A, and by displaying the generated images, a virtual image is projected that is formed further back than the display of the smartphone or the like. By adopting this configuration, the digital mirror system 1A allows the user 5A to capture his / her own image more comfortably.

[0065] Summary of the Embodiment According to the embodiment described above, the electronic mirror system 1 includes an electronic mirror 3 including a plurality of display elements 35 and a microlens array 333 provided in front of the plurality of display elements 35, an imaging device 2 that captures an image of a subject to be displayed on the plurality of display elements 35, a sensor 21 that acquires position information of both eyes of a person viewing the electronic mirror 3, and an ECU 31 that generates images to be displayed on the plurality of display elements 35 based on the image of the subject captured by the imaging device 2. The ECU 31 generates images to be displayed on the display elements 35 so that the image of the subject is formed at a focal position behind the display elements 35. The images to be displayed on the display elements 35 include a right-eye image that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror 3, and a left-eye image that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror 3. The electronic mirror system 1 can provide an electronic mirror 3 with a far-point viewpoint by displaying virtual images using the light field optical technology described above. Therefore, according to this embodiment, it is possible to provide an electronic mirror system 1 and an electronic mirror display method that are easily visible even when installed in a location close to the driver 5 where wide visibility adjustment is difficult.

[0066] Furthermore, according to this embodiment, when the viewpoint is set to a distant viewpoint, the image displayed on the electronic mirror 3 is an image with a viewing angle that corresponds to the position of the eye point of the driver 5, so that the electronic mirror system A can realize an optical mirror-like electronic mirror that is free from convergence adjustment inconsistencies.

[0067] Furthermore, according to the above-described embodiment, in the electronic mirror system 1, the image of the subject captured by the imaging device 2 includes a common area used to generate the right-eye image and the left-eye image, a right-eye image area used to generate the right-eye image but not the left-eye image, and a left-eye image area used to generate the left-eye image but not the right-eye image (see FIG. 8). In other words, according to this embodiment, the electronic mirror system 1 generates the right-eye image and the left-eye image by cutting out two images shifted from each other in the left-right direction from the captured image. Therefore, according to this embodiment, it is possible to easily achieve a far-viewpoint view.

[0068] Furthermore, according to the above-described embodiment, in the electronic mirror system 1, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the right-eye image, and the number of pixels in the height direction of the left-eye image are all the same. Furthermore, the ECU 31 determines the number of pixels in the width direction of the right-eye image area and the left-eye image area based on position information of the eyes of the person viewing the electronic mirror. That is, according to this embodiment, the shift amount is calculated from the eye point of the driver 5. Therefore, according to this embodiment, even if the positions of the electronic mirror 3 and the driver 5 are fixed and the relative positional relationship cannot be changed, a suitable image can be displayed on the electronic mirror 3.

[0069] The electronic mirror system 1 according to the present embodiment is clearly different from a head-up display (HUD) or 3D TV according to the prior art. For example, a HUD creates a natural parallax by displaying a virtual image on an LCD at a distance of approximately 3 to 5 meters. However, in terms of how the screen appears, a HUD creates an unnatural view of the panel in the distance through the mirror frame. Furthermore, a 3D TV provides a stereoscopic image by providing a parallax barrier to create horizontal parallax between left and right images. Display devices such as 3D TVs create left and right parallax for images with the same angle of view, making them appear distant or three-dimensional. However, display devices such as 3D TVs can cause fatigue (convergence accommodation conflict) due to the LCD panel being extremely close to the perspective created by binocular parallax. Furthermore, HUDs and 3D TVs according to the prior art do not incorporate the concept of peripheral vision as in the present embodiment.

[0070] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiments may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0071] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0072] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate.

[0073] According to the present invention, it is possible to provide an electronic mirror system and an electronic mirror display method that are easily visible even when installed in a location close to the driver where wide visibility adjustment is difficult.

[0074] 1...electronic mirror system, 2...imaging device, 3...electronic mirror, 4...vehicle, 5...driver, 31...ECU, 32...TFT driver, 33...liquid crystal display panel, 34...wedge-shaped reflector, 331...TFT panel, 332...backlight, 333...microlens array, 21...sensor, 35...display element, 6...far-point display image, 61...far-point display point, 71...crystalline lens, 72...eyeball, 36...electronic mirror display image, 91...optical inner mirror

Claims

1. An electronic mirror system comprising: an electronic mirror having a display element and a microlens array provided in front of the display element; an imaging device that captures an image of a subject to be displayed on the display element; a sensor that obtains position information of both eyes of a person viewing the electronic mirror; and a control unit that generates an image to be displayed on the display element based on the image of the subject captured by the imaging device, wherein the control unit generates an image to be displayed on the display element so that an image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

2. The electronic mirror system of claim 1, wherein the image of the subject captured by the imaging device includes a common area used to generate the image for the right eye and the image for the left eye, a right-eye image area that is used to generate the image for the right eye and not used to generate the image for the left eye, and a left-eye image area that is used to generate the image for the left eye and not used to generate the image for the right eye.

3. The electronic mirror system of claim 2, wherein the number of pixels in the height direction of the common area, the number of pixels in the height direction of the image for the right eye, and the number of pixels in the height direction of the image for the left eye are all the same, and the control unit determines the number of pixels in the width direction of the image area for the right eye and the number of pixels in the width direction of the image area for the left eye based on position information of the eyes of a person viewing the electronic mirror.

4. An electronic mirror system as described in any one of claims 1 to 3, wherein the imaging device images the direction opposite to the direction of travel of the moving body, and the electronic mirror is fixed in a position where an operator of the moving body can confirm the direction opposite to the direction of travel.

5. An electronic mirror display method for displaying an image of a subject on an electronic mirror having a display element and a microlens array provided in front of the display element, comprising: an imaging step of imaging the subject to be displayed on the display element; an acquisition step of acquiring position information of both eyes of a person viewing the electronic mirror; and a control step of generating an image to be displayed on the display element based on the image of the subject captured in the imaging step, wherein the control step generates an image to be displayed on the display element so that the image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

Images