Electronic mirror system and electronic mirror display method

Abstract

To provide an electronic mirror system and an electronic mirror display method which enable an electronic mirror to be easily viewed even when the electronic mirror is disposed at a place close to a driver having difficulty in wide visibility adjustment.SOLUTION: An electronic mirror system includes: an electronic mirror including a display element and a microlens array provided on a front surface of the display element; an imaging device for capturing an image of a subject to be displayed on the display element; a sensor for acquiring positional information on both eyes of a person viewing the electronic mirror; and a control unit for generating an image to be displayed on the display element on the basis of the image of the subject captured by the imaging device. The control unit generates the image to be displayed on the display element so as to form an image of the subject at a focal position behind the display element. The image to be displayed on the display element includes: a right-eye image formed at the focal position when viewed from a position of the right eye of the person viewing the electronic mirror; and a left-eye image formed at the focal position when viewed from a position of the left eye of the person viewing the electronic mirror.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an electronic mirror system and an electronic mirror display method. [Background technology]

[0002] Conventionally, an in-vehicle electronic mirror is known as a device for a driver to check the rear of a vehicle. Unlike the optical inner mirror that was mainstream until then, the in-vehicle electronic mirror displays an image from a camera for capturing a rear view image installed at the rear of the vehicle on a display device installed at a position where the driver can see it. In the case of the optical inner mirror, there was a problem that the field of view was narrowed due to being blocked by vehicle structures such as C pillars and D pillars, but by using the in-vehicle electronic mirror, it is possible to ensure a sufficient rear field of view. In addition, in the case of the in-vehicle electronic mirror, the device automatically prevents glare and it is possible to ensure a rear field of view without being dazzled by the lights of the rear vehicle even at night. Such an in-vehicle electronic mirror can be said to be a safe driving support device that supports safe driving.

[0003] In-vehicle electronic mirrors are usually installed at a position between 60 [cm (centimeters)] and 80 [cm] from the driver's viewpoint. For middle-aged and elderly drivers (i.e., drivers with farsightedness) who have difficulty in wide-angle diopter adjustment, there is a problem that the in-vehicle electronic mirror installed at such a position cannot be focused on before securing rear visibility. In the case of small cars, the in-vehicle electronic mirror may be installed even closer, and for middle-aged and elderly drivers who have difficulty in wide-angle diopter adjustment, the problem of not being able to focus on the in-vehicle electronic mirror is more prominent. Therefore, companies that provide in-vehicle electronic mirrors are considering making the in-vehicle electronic mirror a distant viewpoint, but at present, no effective method has been proposed. For example, Patent Document 1 and Patent Document 2 can be cited as documents that disclose technologies for improving the visibility of in-vehicle electronic mirrors. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-162431 A [Patent Document 2] JP 2022-178013 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology described in Patent Document 1, a concave mirror is used as a display unit that displays an image from a camera installed at the rear of the vehicle, thereby displaying a virtual image at the front of the vehicle. By adopting such a configuration, it may be possible to improve visibility. However, the technology described in Patent Document 1 has a problem in that the size of the housing becomes significantly large.

[0006] According to the technology described in Patent Document 2, an on-board electronic mirror like an optical mirror is provided by superimposing and displaying an image from a camera installed at the rear of the vehicle and a rear image from inside the vehicle. However, the image displayed by this method is a real image, and there is a problem that it is not possible to display a virtual image from a distant viewpoint.

[0007] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an electronic mirror system and an electronic mirror display method that are easily visible even when the system is installed in a location close to the driver where wide visibility adjustment is difficult. [Means for solving the problem]

[0008] [1] One aspect of the present invention is an electronic mirror system comprising: an electronic mirror having a display element and a microlens array provided in front of the display element; an imaging device that images a subject to be displayed on the display element; a sensor that acquires position information of both eyes of a person viewing the electronic mirror; and a control unit that generates an image to be displayed on the display element based on the image of the subject captured by the imaging device, wherein the control unit generates an image to be displayed on the display element so that the image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror.

[0009] [2] Moreover, one aspect of the present invention is an electronic mirror system as described in [1] above, in which the image of the subject captured by the imaging device includes a common area used for generating the image for the right eye and the image for the left eye, a right eye image area used for generating the image for the right eye but not used for generating the image for the left eye, and a left eye image area used for generating the image for the left eye but not used for generating the image for the right eye.

[0010] [3] Furthermore, one aspect of the present invention is such that, in the electronic mirror system described in [2] above, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the image for the right eye, and the number of pixels in the height direction of the image for the left eye are all the same, and the control unit determines the number of pixels in the width direction of the image area for the right eye and the number of pixels in the width direction of the image area for the left eye based on position information of the eyes of a person viewing the electronic mirror.

[0011] [4] Furthermore, one aspect of the present invention is an electronic mirror system as described in any one of [1] to [3] above, wherein the imaging device images the direction opposite to the direction of travel of a moving body, and the electronic mirror is fixed in a position where an operator of the moving body can confirm the direction opposite to the direction of travel.

[0012] [5] Moreover, one aspect of the present invention is an electronic mirror display method for displaying an image of a subject on an electronic mirror having a display element and a microlens array provided in front of the display element, the electronic mirror display method comprising: an imaging step of imaging the subject to be displayed on the display element; an acquisition step of acquiring positional information of both eyes of a person viewing the electronic mirror; and a control step of generating an image to be displayed on the display element based on the image of the subject captured in the imaging step, wherein the control step generates an image to be displayed on the display element so that the image of the subject is formed at a focal position behind the display element, and the image to be displayed on the display element includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror. Effect of the Invention

[0013] According to the present invention, it is possible to provide an electronic mirror system and an electronic mirror display method that are easily visible even when installed in a location close to the driver where wide visibility adjustment is difficult. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram for explaining an outline of the configuration of an electronic mirroring system according to an embodiment of the present invention; [Diagram 2] 1 is a diagram for explaining an outline of the configuration of a digital mirror according to an embodiment of the present invention; [Diagram 3] FIG. 1 is a first diagram for explaining an outline of a light field optical technology adopted in the digital mirror according to the present embodiment. [Figure 4] FIG. 2 is a second diagram for explaining an outline of the light field optical technology adopted in the digital mirror according to the embodiment. [Diagram 5] 1A and 1B are diagrams comparing the visual fields of an optical mirror and an electronic mirror to explain the left and right visual fields of the electronic mirror according to the embodiment. [Figure 6]1A to 1C are diagrams for explaining changes in eye point and field of view in a display method for an electronic mirror according to the present embodiment. [Figure 7] 10A to 10C are diagrams showing an example of a change in rear visibility accompanying a change in the field of view in the electronic mirror display method according to the present embodiment. [Figure 8] 11A and 11B are diagrams for explaining clipping of an image captured by the imaging device according to the present embodiment. [Figure 9] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of the electronic mirroring system according to the present embodiment. [Figure 10] 1 is a first diagram for explaining an example of a calculation that takes into account the tilt of the mirror in the digital mirror system according to the present embodiment. FIG. [Figure 11] 13 is a second diagram for explaining an example of a calculation that takes into account the tilt of the mirror in the digital mirror system according to the embodiment. FIG. [Figure 12] FIG. 11 is a diagram for explaining a modified example of the electronic mirroring system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] A preferred embodiment of the electronic mirror system and electronic mirror display method according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. In addition, "based on XX" in this application means "based on at least XX", and includes cases based on other elements in addition to XX. In addition, "based on XX" is not limited to cases where XX is directly used, but also includes cases where XX is based on something that has been calculated or processed. "XX" is any element (for example, any information). In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each configuration easier to understand.

[0016] [Embodiment] 1 is a diagram for explaining an outline of the configuration of an electronic mirror system according to an embodiment. First, the configuration of an electronic mirror system 1 will be briefly explained with reference to the diagram. The electronic mirror system 1 includes at least an imaging device 2, an electronic mirror 3, and a vehicle 4. The electronic mirror system 1 transmits an image of the situation behind the vehicle to a driver 5 who drives the vehicle 4 by displaying an image based on a video signal captured by the imaging device 2 on the electronic mirror 3 in real time.

[0017] Here, when the driver 5 is young, the crystalline lens of the eye has sufficient elasticity and is capable of adjusting the diopter over a wide range. Therefore, even if the digital mirror 3 is installed close to the driver 5, the young driver 5 can see the image displayed on the digital mirror 3. However, when the driver 5 reaches a certain age (for example, in his / her 40s or older), the elasticity of the crystalline lens decreases due to hardening of the crystalline lens of the eye, making it difficult to adjust the diopter over a wide range (i.e., he / she develops presbyopia).

[0018] In order to make the image displayed on the electronic mirror 3 easily visible even to such a driver 5, it is conceivable to install the electronic mirror 3 at a position far from the driver 5. On the other hand, there may be a limitation in the location where the electronic mirror 3 is installed in terms of vehicle space. Since the driver 5 drives while looking ahead of the vehicle 4, the electronic mirror 3 needs to be installed in front of the driver 5. However, in the case of a vehicle 4 in which sufficient vehicle space cannot be secured, such as a small car, the electronic mirror 3 needs to be installed near the driver 5, and it may be difficult to secure a sufficient distance between the electronic mirror 3 and the driver 5.

[0019] In driving a vehicle, which requires instantaneous judgment, it is extremely dangerous if the driver is unable to instantly grasp the situation behind the vehicle 4. Therefore, in this embodiment, the electronic mirror system 1 provides an image that the driver 5 can easily view, even if the electronic mirror 3 is installed close to the driver 5.

[0020] In the illustrated example, the vehicle 4 is an automobile moving on land. However, the electronic mirror system 1 according to the present embodiment is not limited to the example of application to the vehicle 4. The electronic mirror system 1 according to the present embodiment can be applied to a wide range of moving objects operated by a driver 5. The moving object operated by the driver 5 may be, for example, an airplane or helicopter moving through the air, a ship moving on the water, or a submarine moving underwater.

[0021] The imaging device 2 captures an image of a subject to be displayed on the electronic mirror 3. Specifically, the imaging device 2 is a camera including an image sensor capable of capturing video. The imaging device 2 is installed at a position where it can capture an image of the opposite direction to the traveling direction of the vehicle 4 (or a moving object). An example of a position where it can capture an image of the opposite direction to the traveling direction is the rear of the vehicle 4, as shown in the figure. The imaging device 2 is fixed and installed at a position where it can capture an image of the opposite direction to the traveling direction of the vehicle 4. By installing the imaging device 2 at such a position, it is possible to solve the problem of the optical inner mirror, in which the field of view is obstructed by vehicle structures such as C-pillars and D-pillars, resulting in a narrow field of view.

[0022] The electronic mirror 3 is fixed in a position where the driver 5 of the vehicle 4 (or the operator of a moving body) can simultaneously check the traveling direction of the vehicle 4 and the direction opposite the traveling direction. The electronic mirror 3 is typically installed above the windshield of the vehicle 4 (where an optical inner mirror has conventionally been provided) facing the driver 5. By installing the electronic mirror 3 in such a position, the driver 5 can simultaneously view the traveling direction and the direction opposite the traveling direction.

[0023] 2 is a diagram for explaining an outline of the configuration of the digital mirror according to this embodiment. The outline of the configuration of the digital mirror 3 will be explained with reference to the same figure. The digital mirror 3 includes at least an ECU (Electronic Control Unit) 31, a TFT (Thin Film Transistor Liquid Crystal) driver 32, a liquid crystal display panel 33, and a wedge-shaped reflecting mirror 34.

[0024] The ECU 31 acquires a video signal of the rear of the vehicle from the imaging device 2. The ECU 31 performs image processing on the acquired video signal to generate a video signal to be displayed to the driver 5. The ECU 31 performs overall control of the electronic mirror system 1. In the following description, the ECU 31 may also be referred to as a control unit.

[0025] The TFT driver 32 performs driving to display the video signal generated by the ECU 31 on the liquid crystal display panel 33. Specifically, the TFT driver 32 controls the display by applying a voltage to each display cell of the liquid crystal display panel 33 based on the video signal generated by the ECU 31. The TFT driver 32 converts the constant-voltage video signal output from the ECU 31 into a voltage level required by the liquid crystal display panel 33, and supplies the voltage to the liquid crystal display panel 33.

[0026] The liquid crystal display panel 33 includes a TFT panel 331, a backlight 332, and a microlens array 333. The TFT panel 331 and the backlight 332 are driven by the TFT driver 32. The TFT panel 331 includes a plurality of display cells (display elements) (not shown), and performs display based on a video signal generated by the ECU 31. The backlight 332 guides or diffuses light obtained by emission of a light emitting diode (LED) (not shown), and generates a surface light source. The generated surface light source illuminates the liquid crystal from the back of the TFT panel 331, thereby displaying a screen to the driver 5.

[0027] The microlens array 333 is disposed close to the image emission side of the TFT panel 331. The microlens array 333 has a plurality of microlenses arranged in a lattice pattern. Each microlens corresponds to one or more display cells of the TFT panel 331. The microlens array 333 controls the light distribution of the image signal displayed on the TFT panel 331, and provides a suitable image to the driver 5. The pitch, aperture, lens focal length, etc. of the microlens array 333 are designed so that the light is selectively distributed so that the focus is at the appropriate position.

[0028] The wedge-shaped reflecting mirror 34 is configured as a part of the mirror housing so as to seal the built-in components of the liquid crystal display panel 33, including the TFT panel 331, the backlight 332, and the microlens array 333. The wedge-shaped reflecting mirror 34 has a structure covered with a wedge-shaped glass mirror having different thicknesses at the top and bottom. The wedge-shaped reflecting mirror 34 is half-mirror coated on the microlens array side, and when the TFT panel 331 is in a non-driven state, the electronic mirror 3 can be used as a normal optical mirror.

[0029] According to this embodiment, by using the liquid crystal display panel 33 that requires the backlight 332, it is possible to selectively use the electronic mirror 3 as a normal optical mirror or as a normal electronic mirror. However, this embodiment is not limited to the example of using liquid crystal, and other display methods can be widely adopted. For example, instead of the liquid crystal display panel 33 that uses a backlight, a self-emitting organic EL display or the like may be used. When a self-emitting organic EL display or the like is used, it is possible to select whether the electronic mirror 3 is used as a normal optical mirror or as a normal electronic mirror by controlling the on / off of the light emission.

[0030] The electronic mirror system 1 according to this embodiment employs the above-mentioned configuration and uses light field optical technology to provide an easily visible display screen even when installed close to the driver 5. The principle of far-viewing using light field optical technology will be described with reference to Figs. 3 and 4.

[0031] FIG. 3 is a first diagram for explaining an outline of the light field optical technology adopted in the electronic mirror according to the present embodiment. The TFT panel 331 includes a plurality of display elements 35. The display elements 35 may be light modulation elements for optically modulating liquid crystal. The plurality of display elements 35 are arranged corresponding to color filters for color display. The display element 35 corresponding to the red color filter is described as a red display element 35R, the display element 35 corresponding to the blue color filter is described as a blue display element 35B, and the display element 35 corresponding to the green color filter is described as a green display element 35G. These display elements 35 may be arranged, for example, in a stripe arrangement or a mosaic arrangement. In the same figure, as an example, a set of a red display element 35R, a blue display element 35B, and a green display element 35G is shown.

[0032] The orientation of light rays is controlled by a microlens array 333 arranged on the exit surface of the TFT panel 331. As a result of controlling the orientation of light rays, an image displayed on the TFT panel 331 is observed as a virtual image. The TFT panel 331 does not display the image from the camera as is, but displays the image so that the image from the camera is observed as a virtual image.

[0033] FIG. 4 is a second diagram for explaining an outline of the light field optical technology adopted in the electronic mirror according to this embodiment. As shown in the figure, the light field element image displayed on the TFT panel 331 reaches the crystalline lens 71 in the eyeball 72 of the driver 5 as a result of selectively controlling the light distribution by the microlens array 333. As shown in the figure, the display image by the electronic mirror 3 is formed as a virtual image at the far-point display point 61. For the sake of explanation, the image forming point of the red display element 35R is described as the far-point red display point 61R, the image forming point of the blue display element 35B is described as the far-point blue display point 61B, and the image forming point of the green display element 35G is described as the far-point green display point 61G. After such light reaches the eyeball 72 of the driver 5, it is focused on the retina. The image focused on the retina is a virtual image that is focused not on the TFT panel 331 of the electronic mirror 3 but on the far-point display point 61, which is a point further away from the back surface of the electronic mirror 3.

[0034] According to this embodiment, the far-point view is achieved by using such light field optical technology. Therefore, the image displayed on the TFT panel 331 is an image aligned with the position of the eyeball 72 of the driver 5. The positions of the eyeballs 72 of the driver 5 are different between the right eye and the left eye. Therefore, according to this embodiment, the image signal for the right eye and the image signal for the left eye are different from each other since the images are formed on the right eye and the left eye, respectively.

[0035] Next, the difference between a normal optical mirror and an electronic mirror will be briefly described with reference to Fig. 5. Fig. 5 is a diagram comparing the visual fields of an optical mirror and an electronic mirror to explain the left and right visual fields of the electronic mirror according to this embodiment. Fig. 5(A) is a diagram showing the difference between the left and right visual fields of an optical mirror. Meanwhile, Fig. 5(B) is a diagram showing the visual field of an electronic mirror.

[0036] As shown in FIG. 5(A), in a normal optical mirror, the vehicle rear view that can be seen through the optical inner mirror 91 looks like the same image with both eyes, but in fact the left and right eyes see different images. In the same figure, the rear image seen by the right eye through the mirror is shown as the rear view right eye field of view FR. Also, the rear image seen by the left eye through the mirror is shown as the rear view left eye field of view FL. In this way, the driver 5 may think that he or she is seeing the same rear view with the left and right, but the respective images are actually different. Specifically, the right eye sees an image shifted to the rear left by the difference G1 between the left and right fields of view, and the left eye sees an image shifted to the rear right by the difference G2 between the left and right fields of view.

[0037] As shown in FIG. 5(B), an electronic mirror 92 according to the prior art displays an image captured by a camera installed at the rear of the vehicle directly on a TFT panel 93. Therefore, the driver 5 sees the same image whether he looks at the electronic mirror 92 with his right eye or with his left eye. The electronic mirror system 1 according to this embodiment is an electronic mirror, but is capable of individually displaying the difference in the field of view of both eyes obtained by an optical mirror. According to this embodiment, by adopting such a configuration, it is possible to achieve the visibility of an optical mirror with an electronic mirror.

[0038] FIG. 6 is a diagram for explaining the change in eye point and field of view in the display method of the electronic mirror according to this embodiment. With reference to the figure, the change in eye point and field of view will be explained, and the concept of convergence will also be explained. In the illustrated example, a linear system without reflection is shown to simplify the explanation. However, when reflection by the electronic mirror 3 is taken into consideration, it is possible to consider it by adapting the regular reflection condition.

[0039] Here, when the eye point (pupil) of the driver 5 shifts, the field of view changes. In the example shown in the figure, the position where the driver 5 shifts to the right is shown as driver 5R, and the position where the driver 5 shifts to the left is shown as driver 5L. At this time, the change in the field of view of the driver 5 is dominated by the aperture of the electronic mirror 3, and is optically called a field stop. In this embodiment, by displaying rear view images (fields of view) corresponding to the left and right eyes at the position of this field stop, the electronic mirror 3 can ensure visibility close to that of an optical mirror, even though it is a so-called electronic mirror.

[0040] Here, if images with different fields of view are simply displayed to each eye without a proper standard using light field technology, and the display image of the electronic mirror 3 is displayed as a virtual image in the distance, the focal position due to the far viewpoint and the far viewpoint position due to the binocular parallax may differ, and fusion may not be successful during binocular vision. If fusion is not successful in this way, the driver 5 may experience physiological disorders such as dizziness. It is extremely dangerous to experience physiological disorders such as dizziness while driving the vehicle 4. The phenomenon in which fusion cannot be achieved successfully during binocular vision is called a convergence accommodation conflict. The convergence accommodation conflict occurs when the focal position and the binocular parallax (convergence) do not match.

[0041] On the other hand, when an optical mirror is used, the visual fields of the left and right eyes change via the optical mirror, but since there is no contradiction in the convergence relationship of the point serving as the reference for the visual field (the central image when viewed from the center of the eye point), no convergence accommodation contradiction occurs (i.e., there is no physiological discomfort). Therefore, in the display method of the electronic mirror 3 according to this embodiment, not only the visual field at each eye point but also the visual field reference (in this case, the image center) is controlled so that there is no convergence contradiction. Specifically, when the electronic mirror 3 displays, the image is shifted left and right as the eye point shifts.

[0042] FIG. 6 shows an example in which the distance between the driver 5 and the electronic mirror 3 is 600 mm (millimeters). In this situation, a case in which a virtual image is displayed at a position 1600 mm away from the driver 5 will be considered. When the interpupillary distance of the driver 5 is 65 mm, the convergence angle is 2.33° as shown in the figure. In this case, the electronic mirror 3 displays images shifted by about 40.6 mm to the left and right eyes of the driver 5, making it possible to provide an image with visibility close to that of an optical mirror without any convergence accommodation conflict.

[0043] FIG. 7 is a diagram showing an example of a change in rear visibility accompanying a change in the field of view in the electronic mirror display method according to this embodiment. Next, with reference to the same figure, the handling of the image of the rear of the vehicle 4 actually captured by the imaging device 2 and the field of view image accompanying a shift in the eye point will be described. As described above, the electronic mirror system 1 according to this embodiment displays the field of view that changes with the shift in the eye point. This "changing field of view" is actually obtained by changing the area to be cut out from the image of the rear of the vehicle 4 captured by the imaging device 2, thereby acquiring the field of view image for each eye.

[0044] The figure shows a field of view V1, which is the field of view from the center of the eye point in the image behind vehicle 4 captured by imaging device 2. Vehicles C1, C2, C3, and C4 are captured in field of view V1. This field of view V1 is restricted from the eye point by a field diaphragm. Note that the figure shows a line serving as the field of view reference with a dashed line. This field of view reference is the center of the field of view from the eye point center. As the eye point shifts from the eye point center, the field of view also changes left and right. At this time, even if the field of view changes due to eye point shift, convergence is managed based on this field of view reference.

[0045] FIG. 8 is a diagram for explaining clipping of an image captured by an imaging device according to this embodiment. Clipping of an image actually captured by an imaging device 2 will be explained with reference to the same figure. FIG. 8(A) is a diagram showing an example of an image of the rear of a vehicle 4 captured by an imaging device 2. FIG. 8(B) is an image cut out from the image shown in FIG. 8(A) as the field of view of the right eye. FIG. 8(C) is an image cut out from the image shown in FIG. 8(A) as the field of view of the left eye. As shown in the figure, according to the electronic mirror system 1, images corresponding to the field of view from each eye point are clipped from the image of the rear of a vehicle 4 captured by an imaging device 2, and light field element images are generated from each clipped image so that the light distribution from the electronic mirror 3 and the eye point (and each field of view image) match. By such processing, it is possible to realize an electronic mirror 3 that is free from a contradiction between a far viewpoint and convergence accommodation.

[0046] 9 is a functional configuration diagram showing an example of the functional configuration of the electronic mirror system according to this embodiment. With reference to the diagram, more specific implementation means of the above-mentioned electronic mirror system 1 will be described. The electronic mirror system 1 includes at least an imaging device 2, a sensor 21, an ECU 31, a TFT driver 32, and a liquid crystal display panel 33.

[0047] The imaging device 2 captures an image of the rear of the vehicle 4. The imaging device 2 outputs a video signal obtained by capturing the image to the ECU 31.

[0048] The sensor 21 acquires position information of both eyes of a person viewing the electronic mirror 3 (in this example, the driver of the vehicle 4). The position information of both eyes may be the relative coordinates of the left eye and the relative coordinates of the right eye with respect to the electronic mirror 3. If the distance between the right eye and the left eye is known, the sensor 21 may be configured to acquire position information of only one of the eyes. Note that a conventional technique may be used to acquire the position information of both eyes. For example, the electronic mirror 3 may be provided with an in-camera (not shown) installed at a position capable of capturing an image of the face of a person viewing the electronic mirror 3, and the position information of both eyes of the person viewing the electronic mirror 3 may be acquired by image processing the image captured by the in-camera. In this case, the sensor 21 may be configured to include an in-camera and an image processing unit.

[0049] At night, it may not be possible to obtain an image of sufficient brightness for the image information of the face of the driver 5. Therefore, when obtaining position information of the eyes of a person viewing the electronic mirror 3 based on an image, use at night may be restricted. Therefore, instead of the image information obtained by the imaging device 2, position information of the eyes of a person viewing the electronic mirror 3 may be obtained by processing a distance image obtained by a ToF (Time of Flight) sensor, LiDAR (Light Detection and Ranging), or the like, which is not affected by the brightness of the surroundings.

[0050] The ECU 31 generates an image to be displayed on the electronic mirror 3 based on the image of the subject captured by the imaging device 2. Specifically, the ECU 31 performs calculations based on light field technology to generate an image to be displayed on the electronic mirror 3 so that the image of the subject captured by the imaging device 2 is formed at a focal position behind the electronic mirror 3. The images generated by the ECU 31 and displayed on the electronic mirror 3 include an image for the right eye and an image for the left eye. The image for the right eye is an image that is formed at the focal position when viewed from the position of the right eye of a person viewing the electronic mirror 3. The image for the left eye is an image that is formed at the focal position when viewed from the position of the left eye of a person viewing the electronic mirror 3.

[0051] As described with reference to FIG. 8, the right-eye image and the left-eye image are generated by clipping one image (cutting out from one image). As shown in FIG. 8(A), a common area, a right-eye image area, and a left-eye image area exist in the image of the subject captured by the imaging device 2. The common area is an area commonly used for generating the right-eye image and the left-eye image. The right-eye image area is an area used for generating the right-eye image and not used for generating the left-eye image. The left-eye image area is an area used for generating the left-eye image and not used for generating the right-eye image. That is, the right-eye image and the left-eye image are shifted from each other in the left-right direction, not in the up-down direction. That is, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the right-eye image, and the number of pixels in the height direction of the left-eye image are all the same, and all images are acquired from the same coordinates in the height direction.

[0052] Further, the shift amount between the right eye image and the left eye image is calculated from the eye points of both eyes of the person viewing the electronic mirror 3. In other words, the ECU 31 determines the number of pixels in the width direction of the right eye image area and the number of pixels in the width direction of the left eye image area based on the position information of both eyes of the person viewing the electronic mirror 3. In this embodiment, when the electronic mirror 3 is used to check the rear of the vehicle 4, it is not easy to change the positional relationship between the electronic mirror 3 and the driver 5. Therefore, the electronic mirror 3 changes the image to be displayed according to the position of the eyes of the driver 5.

[0053] Returning to FIG. 9, the TFT driver 32 displays an image to the driver 5 by outputting a drive signal based on the right-eye image and the left-eye image generated by the ECU 31 to the liquid crystal display panel 33 .

[0054] In the description made with reference to FIG. 5, the electronic mirror 3 is installed facing the driver 5. However, in an actual vehicle 4, the electronic mirror 3 is often installed to the upper left of the driver 5, and is often installed at an angle with respect to the driver 5. An example of a calculation method for a case where the electronic mirror 3 is installed at an angle with respect to the driver 5 will be described below with reference to FIGS. 10 and 11. Note that in the description made with reference to these figures, the positional relationship between the electronic mirror 3 and the driver 5 may be described with reference to an xy two-dimensional coordinate system.

[0055] FIG. 10 is a first diagram for explaining an example of calculation considering the tilt of the mirror for the electronic mirror system according to this embodiment. As shown in the figure, the electronic mirror 3 is installed at an angle with respect to the driver 5. The tilt of the electronic mirror 3 at this time will be explained with reference to FIG. 11. FIG. 11 is a second diagram for explaining an example of calculation considering the tilt of the mirror for the electronic mirror system according to this embodiment. As shown in the figure, the distance in the x-axis direction from the driver 5 to the center of the electronic mirror 3 is y. Also, the distance in the x-axis direction from the center of the electronic mirror 3 to the right end of the electronic mirror 3 is y'. Similarly, the distance in the x-axis direction from the center of the electronic mirror 3 to the left end of the electronic mirror 3 is y'. Also, the length (mirror size) of the electronic mirror 3 in the horizontal direction (longitudinal direction) is m. Also, the tilt angle of the electronic mirror 3 with respect to the x-axis direction is β.

[0056] In this case, it can be calculated as y'=(m / 2)sinβ. Also, the distance m' in the x-axis direction at which the real image of the digital mirror 3 is projected can be calculated as m'=mcosβ.

[0057] Returning to FIG. 10, if the position of the right eye of the driver 5 is +x from the center and the position of the left eye is -x from the center, then angles θ', θ'', ω', and ω'' can be expressed using equations (1) to (4) below.

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062] Using the angle information obtained in this way, the viewing angle on the left side of the figure can be calculated as δθ=θ''-θ', and the viewing angle on the right side of the figure can be calculated as δω=ω''-ω'.

[0063] [Modification of the embodiment] FIG. 12 is a diagram for explaining a modified example of the electronic mirror system according to the present embodiment. With reference to the figure, an electronic mirror system 1A, which is a modified example of the electronic mirror system 1, will be explained. The electronic mirror system 1A is applied not to a vehicle 4 but to a smartphone, a tablet terminal device, a notebook computer, etc. (hereinafter, referred to as a smartphone, etc.). In this case, the electronic mirror 3A is a smartphone, etc. The smartphone is equipped with an imaging device 2A as an in-camera that captures an image from the side where the main display unit is present.

[0064] The electronic mirror 3A is used by a user 5A. The user 5A captures an image of himself / herself using an imaging device 2A, which is an in-camera of a smartphone or the like (i.e., the user 5A takes a selfie or takes a self-portrait). The electronic mirror system 1A displays the captured image of the user 5A on a display of the smartphone or the like using the above-mentioned light field technology. At this time, the electronic mirror 3A generates an image for the right eye and an image for the left eye according to the position of the eyepoint of the user 5A, and displays the generated images to project a virtual image formed on the back side of the display of the smartphone or the like. According to the electronic mirror system 1A, by adopting such a configuration, the user 5A can capture an image of himself / herself more comfortably.

[0065] [Summary of the embodiment] According to the embodiment described above, the electronic mirror system 1 includes an electronic mirror 3 including a plurality of display elements 35 and a microlens array 333 provided in front of the plurality of display elements 35, an imaging device 2 that captures an image of a subject to be displayed on the plurality of display elements 35, a sensor 21 that acquires position information of both eyes of a person viewing the electronic mirror 3, and an ECU 31 that generates an image to be displayed on the plurality of display elements 35 based on the image of the subject captured by the imaging device 2. The ECU 31 generates an image to be displayed on the display element 35 so that the image of the subject is formed at a focal position behind the display element 35. The image to be displayed on the display element 35 includes an image for the right eye that is formed at the focal position when viewed from the position of the right eye of the person viewing the electronic mirror 3, and an image for the left eye that is formed at the focal position when viewed from the position of the left eye of the person viewing the electronic mirror 3. According to the electronic mirror system 1, it is possible to provide an electronic mirror 3 with a far-point viewpoint by displaying a virtual image using the light field optical technology described above. Therefore, according to this embodiment, it is possible to provide an electronic mirror system 1 and an electronic mirror display method that are easily visible even when installed in a location close to the driver 5 where wide visibility adjustment is difficult.

[0066] Furthermore, according to this embodiment, when the viewpoint is far away, the display image displayed on the electronic mirror 3 is a viewing angle image that corresponds to the position of the eye point of the driver 5, so that the electronic mirror system A can realize an optical mirror-like electronic mirror that is free of convergence accommodation contradictions.

[0067] Also, according to the above-described embodiment, in the electronic mirror system 1, the image of the subject captured by the imaging device 2 includes a common area used to generate an image for the right eye and an image for the left eye, a right eye image area used to generate an image for the right eye but not used to generate an image for the left eye, and a left eye image area used to generate an image for the left eye but not used to generate an image for the right eye (see FIG. 8). In other words, according to this embodiment, the electronic mirror system 1 generates an image for the right eye and an image for the left eye by cutting out two images that are shifted from each other in the left-right direction from the captured image. Therefore, according to this embodiment, it is possible to easily achieve a distant viewpoint.

[0068] According to the above-described embodiment, in the electronic mirror system 1, the number of pixels in the height direction of the common area, the number of pixels in the height direction of the right-eye image, and the number of pixels in the height direction of the left-eye image are all the same. Furthermore, the ECU 31 determines the number of pixels in the width direction of the right-eye image area and the number of pixels in the width direction of the left-eye image area based on the position information of both eyes of the person viewing the electronic mirror. That is, according to this embodiment, the shift amount is calculated from the eye point of the driver 5. Therefore, according to this embodiment, even if the position of the electronic mirror 3 and the position of the driver 5 are fixed and the relative positional relationship cannot be changed, a suitable image can be displayed on the electronic mirror 3.

[0069] The electronic mirror system 1 according to the present embodiment is clearly different from a head-up display (HUD) or a 3DTV according to the prior art. For example, in a HUD, a virtual image is displayed on an LCD at a distance of about 3 [m (meters)] to 5 [m], thereby providing a natural parallax. However, in terms of how the screen is viewed, in a HUD, the panel is viewed at a distance through the frame of the mirror, which is unnatural. In addition, a 3DTV provides a stereoscopic image by providing a parallax barrier and providing a lateral parallax between the left and right images. A display device such as a 3DTV provides left and right parallax to an image with the same angle of view, thereby making it appear distant or three-dimensional. However, a display device such as a 3DTV generates a sense of fatigue (convergence accommodation contradiction) due to the position of the LCD panel being extremely close to the perspective of binocular parallax. In addition, the concept of peripheral vision as in this embodiment does not exist in a HUD or a 3DTV according to the prior art.

[0070] In addition, all or part of the functions of each unit of each device in the above-mentioned embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, reading the program recorded on the recording medium into a computer system, and executing it. Note that the "computer system" referred to here includes hardware such as the OS and peripheral devices.

[0071] In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0072] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate. [Explanation of symbols]

[0073] 1...electronic mirror system, 2...imaging device, 3...electronic mirror, 4...vehicle, 5...driver, 31...ECU, 32...TFT driver, 33...liquid crystal display panel, 34...wedge-shaped reflector, 331...TFT panel, 332...backlight, 333...microlens array, 21...sensor, 35...display element, 6...far-point display image, 61...far-point display point, 71...crystalline lens, 72...eyeball, 36...electronic mirror display image, 91...optical inner mirror

Claims

1. An electronic mirror including a display element and a microlens array provided in front of the display element; an imaging device that captures an image of a subject to be displayed on the display element; A sensor for acquiring position information of both eyes of a person looking at the electronic mirror; a control unit that generates an image to be displayed on the display element based on an image of the subject captured by the imaging device, the control unit generates an image to be displayed on the display element such that the image of the subject is formed at a focal position behind the display element; The image displayed on the display element includes a right-eye image that is formed at the focal position when viewed from the right eye position of a person viewing the electronic mirror, and a left-eye image that is formed at the focal position when viewed from the left eye position of a person viewing the electronic mirror. Electronic mirror system.

2. The image of the subject captured by the imaging device includes a common area used for generating the right eye image and the left eye image, a right eye image area that is used for generating the right eye image but not used for generating the left eye image, and a left eye image area that is used for generating the left eye image but not used for generating the right eye image. The electronic mirror system according to claim 1 .

3. the number of pixels in the height direction of the common area, the number of pixels in the height direction of the right-eye image, and the number of pixels in the height direction of the left-eye image are all equal; The control unit determines the number of pixels in the width direction of the right eye image area and the number of pixels in the width direction of the left eye image area based on position information of both eyes of a person viewing the electronic mirror. The electronic mirror system according to claim 2 .

4. The imaging device captures an image in a direction opposite to a traveling direction of the moving object, The electronic mirror is fixed at a position where the operator of the vehicle can check the direction opposite to the traveling direction. The electronic mirror system according to any one of claims 1 to 3.

5. An electronic mirror display method for displaying an image of a subject on an electronic mirror including a display element and a microlens array provided in front of the display element, comprising: an imaging step of imaging a subject to be displayed on the display element; An acquisition step of acquiring position information of both eyes of a person viewing the electronic mirror; a control step of generating an image to be displayed on the display element based on the image of the subject captured in the imaging step, the control step includes generating an image to be displayed on the display element such that the image of the subject is focused at a focal position behind the display element; The image displayed on the display element includes a right-eye image that is formed at the focal position when viewed from the right eye position of a person viewing the electronic mirror, and a left-eye image that is formed at the focal position when viewed from the left eye position of a person viewing the electronic mirror. Electronic mirror display method.

Images