Electronic mirror device and control program

Abstract

To improve the usability of an electronic mirror device.SOLUTION: A first imaging unit (30) takes an image of a region behind a vehicle. An electronic mirror display unit (20) includes: a display (20a) which can display the back image of the vehicle taken by the first imaging unit (30); and a mirror (20b) arranged on the front side of the display (20a) for reflecting light from the front surface and transmitting light from the back surface. A second imaging unit (50) takes an image of the driver in the vehicle. An image recognition unit (11) recognizes behaviors of the driver from the image taken by the second imaging unit (50). A display control unit (12) controls the display (20a) so that both reflected light from the mirror and transmitting light from the display or reflected light from the mirror alone can be seen when an action of trying to see the back part of the vehicle cabin performed by the driver is detected while the back image of the vehicle is being displayed in the display (20a).SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an electronic mirror device used as a digital rearview mirror, and a control program. [Background technology]

[0002] The room mirror (also called rearview mirror) of an automobile is intended to check the situation behind the vehicle, but in reality it is also used to check the situation in the rear of the passenger compartment (rear seats and luggage). In recent years, there has been a growing trend to replace conventional optical room mirrors with electronic digital room mirrors. Although electronic mirrors allow the driver to check the situation behind the vehicle with a clear image, they do not show the rear of the passenger compartment, making it difficult to check the situation in the rear of the passenger compartment.

[0003] One solution to this problem would be to attach an additional small mirror to the windshield, but this would require separate purchase of the small mirror, and it would be difficult to ensure sufficient visibility with the small mirror.

[0004] Many digital mirror devices allow you to turn off the screen and switch to a normal optical mirror with a switch, etc. However, switching between modes with a switch is cumbersome, and operating it while driving can be a factor that reduces safety.

[0005] In response to this, a method has been proposed in which the electronic mirror mode and the optical mirror mode are switched based on the driver's line of sight (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-30507 A [Patent Document 2] JP 2022-148226 A Summary of the Invention [Problem to be solved by the invention]

[0007] The present inventor has developed a method for further improving the user interface in a digital mirror device that is switchable between a digital mirror mode and an optical mirror mode.

[0008] The present embodiment has been made in view of the above circumstances, and has an object to provide a technique for improving the usability of a digital mirror apparatus. [Means for solving the problem]

[0009] In order to solve the above problems, an electronic mirror device of one aspect of this embodiment includes an electronic mirror display unit having a first imaging unit that images the rear of the vehicle, a display capable of displaying the rear image of the vehicle imaged by the first imaging unit, and a mirror arranged in front of the display and reflecting light from the front and transmitting light from the back, a second imaging unit that images the driver inside the vehicle, an image recognition unit that recognizes the behavior of the driver from the image captured by the second imaging unit, and a display control unit that controls the display so that, when a rear image of the vehicle is displayed on the display and the driver's behavior of attempting to look at the rear of the vehicle's cabin is detected, either the reflected light of the mirror or the transmitted light from the display is visible.

[0010] Any combination of the above components, and conversion of the expression of this embodiment between a method, an apparatus, a system, a recording medium, a computer program, etc. are also valid as aspects of this embodiment. Effect of the Invention

[0011] According to this embodiment, the usability of the digital mirror apparatus can be improved. [Brief description of the drawings]

[0012] [Figure 1]1 is a diagram for explaining an example of installation of a digital mirror apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the external configuration of a main body of a digital mirror apparatus. [Diagram 3] 1 is a diagram showing a system configuration of a digital mirror apparatus according to an embodiment. [Figure 4] 4(a) to 4(c) are diagrams showing specific examples of the visual recognition state of the digital mirror display unit in each mode of the digital mirror device. [Diagram 5] 5(a)-(b) are diagrams showing specific examples of when the driver's line of sight moves in the hybrid mode. [Figure 6] FIG. 13 is a diagram showing a state immediately before switching from the optical mirror mode or the hybrid mode to the electronic mirror mode. [Figure 7] 7(a)-(c) are diagrams showing specific examples of the cropping range of the rear image in the electronic mirror mode. [Figure 8] 8(a)-(c) are diagrams showing examples of rear images cut out from the cut-out ranges of FIGS. 7(a)-(c) displayed on an electronic mirror display unit. [Figure 9] 11A and 11B are diagrams illustrating a specific example in which the driver's line of sight moves in an electronic mirror mode. [Figure 10] 10(a)-(b) are diagrams showing changes in the range that is cut out and displayed from the rear image in the electronic mirror mode. [Figure 11] 5 is a flowchart showing an example of the operation of the digital mirror apparatus according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] FIG. 1 is a diagram for explaining an installation example of an electronic mirror device 1 according to an embodiment. The electronic mirror device 1 according to the embodiment includes an electronic mirror device main body 1a and a rear imaging unit 30. In the example shown in FIG. 1, the electronic mirror device main body 1a is placed on the mirror surface of an existing optical rearview mirror and is fixed to the existing rearview mirror via a plurality of bands or a predetermined fixing holder (not shown). The rear imaging unit 30 is fixed, for example, with double-sided tape on the inside of the rear window of the vehicle V1. The rear imaging unit 30 may be installed at any position as long as it can capture an image behind the vehicle V1, or may be stored in a waterproof case and installed outside the vehicle V1. Also, the optical rearview mirror may be omitted, and only the electronic mirror device main body 1a may be installed.

[0014] 2 is a diagram showing an example of the external configuration of the electronic mirror device main body 1a. An electronic mirror display unit 20 and a driver imaging unit 50 are arranged on the front of the electronic mirror device main body 1a. The driver imaging unit 50 is arranged at a predetermined position in the frame area surrounding the electronic mirror display unit 20. In the example shown in FIG. 2, a right-hand drive vehicle is assumed, and the driver imaging unit 50 is arranged at the upper right of the electronic mirror display unit 20, but in the case of a left-hand drive vehicle, it is desirable to arrange it at the upper left of the electronic mirror display unit 20.

[0015] The driver imaging unit 50 may be configured with an external imaging unit. The external imaging unit may be fixed and installed, for example, at the right end of the digital mirror device main body 1a. The external imaging unit may also be installed at the top of the steering column so as to face the driver's face through a gap in the steering wheel. If a driver monitoring system (DMS) for detecting the driver's inattentiveness or drowsiness is already installed in the vehicle V1, the camera of the driver monitoring system may be diverted to the driver imaging unit 50 according to this embodiment.

[0016] 3 is a diagram showing a system configuration of a digital mirror device 1 according to an embodiment. The digital mirror device 1 includes a processing unit 10, a digital mirror display unit 20, a rear imaging unit 30, and a driver imaging unit 50.

[0017] The rear imaging unit 30 and the driver imaging unit 50 each include a lens and a solid-state imaging element. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor can be used as the solid-state imaging element. The solid-state imaging element converts light incident through the lens into an electrical video signal and outputs the signal to the processing unit 10.

[0018] The electronic mirror display unit 20 includes a display 20a and a half mirror 20b. The display 20a is a display such as a liquid crystal display, an organic EL display, or a mini LED display, and is a display for displaying a rear image of the vehicle V1 captured by the rear imaging unit 30.

[0019] When the digital mirror device 1 is configured as a digital mirror device (also called a mirror recorder) integrated with a drive recorder, the display 20a can also display a forward image of the vehicle V1. In this case, an imaging unit for imaging the area ahead of the vehicle V1 is further installed on the rear surface (surface in the traveling direction of the vehicle V1) of the digital mirror device main body 1a.

[0020] The half mirror 20b is disposed in front of the display 20a, and reflects light from the front and transmits light from the back. The half mirror 20b is a mirror whose reflectance is reduced by adjusting the amount of metal used in the reflective film, and the reflectance of a typical half mirror 20b is adjusted to about 50%. In the electronic mirror display unit 20, since the transmitted light can be erased by turning off the display 20a, it is desirable that the transmittance of the half mirror 20b be high. For example, the transmittance is adjusted to about 50%. Instead of the half mirror 20b, a mirror liquid crystal display whose transmittance can be controlled by electronic control may be used.

[0021] The processing unit 10 includes an image recognition unit 11, a display control unit 12, and an initial mode determination unit 13. The processing unit 10 is realized by a combination of hardware resources and software resources, or by hardware resources alone. As hardware resources, a CPU, ROM, RAM, DSPs (Digital Signal Processors), GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and other LSIs can be used. As software resources, programs such as firmware can be used.

[0022] The image recognition unit 11 recognizes the behavior of the driver Pd from the image captured by the driver imaging unit 50. The image recognition unit 11 can use a known detection algorithm for detecting the line of sight or head posture of the driver Pd. For example, the image recognition unit 11 detects the face area of ​​the driver Pd from the image captured by the driver imaging unit 50 using a face detector. It is desirable to prepare a face detector for a specific driver Pd who actually sits in the driver's seat in advance. The face detector for the specific driver Pd can be generated by capturing an image of the driver Pd from the driver imaging unit 50 multiple times while the target driver Pd is sitting in the driver's seat and looking at the electronic mirror display unit 20.

[0023] When a visible light camera is used for the driver imaging unit 50, the image recognition unit 11 identifies the positions of the inner corner of the eye and the iris included in the eye region based on feature points in the face region of the visible light image. The image recognition unit 11 detects the gaze direction or gaze movement based on the relative position of the iris to the inner corner of the eye, with the position of the inner corner of the eye as the reference point and the position of the iris as the moving point. For example, when the iris of the left eye is moving away from the inner corner of the left eye and the iris of the right eye is approaching the inner corner of the right eye, the image recognition unit 11 estimates that the gaze of the driver Pd is moving to the left.

[0024] When an infrared camera is used for the driver imaging unit 50, an infrared light emitting unit that irradiates infrared rays toward the driver Pd is provided near the infrared camera. The image recognition unit 11 detects the face area of ​​the driver Pd from the infrared image captured by the infrared camera using a face detector. The image recognition unit 11 identifies the corneal reflection point included in the eye area and the position of the pupil based on the feature points in the face area. The corneal reflection point is a reflection point on the cornea that reflects the infrared light irradiated from the infrared light emitting unit. The image recognition unit 11 detects the gaze direction or gaze movement based on the relative position of the pupil with respect to the corneal reflection point, with the infrared method basically being more accurate in estimating the gaze position of the driver Pd.

[0025] The image recognition unit 11 can estimate the distance to an object that the driver Pd is gazing at based on the positional relationship (distance, angle) between the installation position of the driver image capture unit 50 and the position of the driver Pd's eyes acquired in advance, the interpupillary distance between the driver Pd's eyes in normal driving, and the interpupillary distance between the driver Pd's eyes in the captured image. The image recognition unit 11 can estimate whether the driver Pd is moving his / her line of sight to a distant or near direction based on the amount of change in the convergence angle of the eyes during the driver Pd's line of sight movement. The image recognition unit 11 can also estimate the roll angle, pitch angle, and yaw angle of the head from the arrangement of each feature point in the face area to estimate the head posture of the driver Pd.

[0026] In this embodiment, when the driver Pd is looking toward the digital mirror device 1, it is necessary to determine whether the driver Pd is looking at the rear of the vehicle cabin or the rear of the vehicle V1. For example, a plurality of facial images of a person looking at the rear of the vehicle cabin are prepared, and the plurality of facial images are machine-learned to generate a detector for detecting a state in which the driver Pd is looking at the rear of the vehicle cabin. Similarly, a plurality of facial images of a person looking at the rear of the vehicle V1 are prepared, and the plurality of facial images are machine-learned to generate a detector for detecting a state in which the driver Pd is looking at the rear of the vehicle V1. The image recognition unit 11 may refer to these detectors to determine whether the driver Pd is looking at the rear of the vehicle cabin or the rear of the vehicle V1 from the image captured by the driver image capture unit 50.

[0027] In this embodiment, it is desirable to be able to determine which area of ​​the electronic mirror display unit 20 the driver Pd is looking at when the driver Pd is looking at the electronic mirror device 1. It is desirable to detect the gaze for a period of about 2 seconds when the gaze is fixed. For example, the display area of ​​the electronic mirror display unit 20 is divided into four or nine parts, and multiple facial images of people looking at each display area are prepared. The multiple facial images are machine-learned for each display area to generate a detector for detecting the state of looking at each display area. The image recognition unit 11 can refer to these detectors and estimate which display area the driver Pd is looking at when looking at the electronic mirror device 1 from the image captured by the driver imaging unit 50.

[0028] In this embodiment, three modes are prepared for using the electronic mirror device 1 for rearward confirmation: an optical mirror mode, an electronic mirror mode, and a hybrid mode. The optical mirror mode is a mode in which the display 20a is turned off or the brightness of the display 20a is reduced to a level that the driver Pd cannot see. In the optical mirror mode, the driver Pd sees the reflected light of the half mirror 20b, and experiences the same sensation as a normal rearview mirror without an image display function.

[0029] The electronic mirror mode is a mode in which the brightness of the display 20a is set high to the extent that the driver Pd cannot see the reflected light of the half mirror 20b. In the electronic mirror mode, the driver Pd feels the same sensation as when looking at a normal display. The hybrid mode is a mode in which the display 20a is controlled so that the reflected light of the half mirror 20b is the main visual state, and the transmitted light from the display 20a is the secondary visual state. That is, in the hybrid mode, the brightness of the display 20a is controlled so that images from both the display 20a and the half mirror 20b can be displayed.

[0030] In the electronic mirror mode, the display control unit 12 causes the display 20a to display the rear image of the vehicle V1 captured by the rear imaging unit 30. In the optical mirror mode, the display control unit 12 turns off the display 20a. In the hybrid mode, the display control unit 12 reduces the rear image of the vehicle V1 and displays a reduced image of the rear image in a partial area of ​​the display 20a. For example, the display control unit 12 reduces the rear image to less than ¼ of the size of the display 20a and displays the reduced image of the rear image in the lower right or lower left corner of the display 20a. It is to be noted that the reduced image may also be displayed in the upper right or upper left corner of the display 20a.

[0031] The display control unit 12 turns off or displays black in areas where reduced images are not displayed. In the case of an organic EL display or a mini LED display, the elements in the non-display area can be controlled to not emit light. In the case of an LCD display equipped with a local dimming function, the backlight in the non-display area can be turned off.

[0032] When the image recognition unit 11 detects in the electronic mirror mode that the driver Pd is attempting to look at the rear of the vehicle V1, the display control unit 12 switches the electronic mirror device 1 to the optical mirror mode or the hybrid mode. When the image recognition unit 11 detects in the optical mirror mode or the hybrid mode that the driver Pd is attempting to look at the rear of the vehicle V1, the display control unit 12 switches the electronic mirror device 1 to the electronic mirror mode.

[0033] The image recognition unit 11 can recognize the behavior of the driver Pd attempting to look at the rear of the cabin of the vehicle V1, for example, as follows: Based on the position of the iris or pupil of the driver Pd, the image recognition unit 11 judges whether the driver Pd is in a line of sight state in which he or she is looking at the upper left where the digital mirror device 1 is located (hereinafter referred to as a mirror viewing state).

[0034] The image recognition unit 11 may estimate that a behavior of trying to look at the rear of the vehicle cabin has occurred when the driver Pd, in a state of looking at the mirror, thrusts his head forward and the head area in the image is enlarged by a predetermined magnification or more. The image recognition unit 11 may also estimate that a behavior of trying to look at the rear of the vehicle cabin has occurred when the driver Pd, in a state of looking at the mirror, lifts his head and the pitch angle of the head area in the image is increased by a predetermined value or more.

[0035] The image recognition unit 11 may also estimate that a behavior of trying to look at the rear of the vehicle cabin has occurred when the driver Pd, in a mirror-viewing state, raises his / her line of sight, causing the iris or pupil in the image to move upward by a predetermined value or more.The image recognition unit 11 may also estimate that a behavior of trying to look at the rear of the vehicle cabin has occurred when the driver Pd, in a mirror-viewing state, changes his / her line of sight to look at something close, causing the interpupillary distance between both eyes in the image to shorten by a predetermined value or more.

[0036] Furthermore, when the image recognition unit 11 can estimate that the driver Pd is looking at the rear of the vehicle compartment when the driver Pd is looking at the gaze position of the half mirror 20b, for example, included in the upper 1 / 3 region, the image recognition unit 11 may estimate that the driver Pd is looking at the rear of the vehicle compartment. When the driver image capturing unit 50 employs an infrared system, the gaze position of the driver Pd on the half mirror 20b can be identified with high accuracy.

[0037] Conversely, the image recognition unit 11 can recognize the behavior of the driver Pd attempting to look behind the vehicle V1, for example, as follows: The image recognition unit 11 may estimate that the behavior of the driver Pd attempting to look behind the vehicle has occurred when the driver Pd moves his / her head back while looking at the mirror, causing the head area in the image to shrink by a predetermined magnification or more. The image recognition unit 11 may also estimate that the behavior of the driver Pd attempting to look behind the vehicle has occurred when the driver Pd moves his / her head back while looking at the mirror, causing the pitch angle of the head area in the image to decrease by a predetermined value or more.

[0038] The image recognition unit 11 may also estimate that a behavior of trying to look at the rear of the vehicle has occurred when the driver Pd lowers his / her line of sight in the mirror and the iris or pupil in the image moves downward by a predetermined value or more. The image recognition unit 11 may also estimate that a behavior of trying to look at the rear of the vehicle has occurred when the driver Pd changes his / her line of sight to look into the distance in the mirror and the interpupillary distance between the eyes in the image shortens by a predetermined value or more. The image recognition unit 11 may also estimate that the driver Pd is looking at the rear of the vehicle when it can be estimated that the gaze position of the half mirror 20b that the driver Pd is looking at is included in the lower 2 / 3 area, for example.

[0039] The image recognition unit 11 basically monitors the gaze position of the half mirror 20b that the driver Pd is looking at in the mirror viewing state. In the hybrid mode, the display control unit 12 displays a reduced image of the rear image on the display 20a so as to avoid the gaze area of ​​the half mirror 20b that includes the driver Pd's viewpoint. The gaze area of ​​the half mirror 20b is defined as an area that includes the gaze position of the half mirror 20b when the half mirror 20b is divided into a plurality of areas (for example, two divided left and right, two divided up and down, four divided up and down and left and right, or nine divided up and down and left and right, etc.).

[0040] In the hybrid mode, for example, the image recognition unit 11 monitors whether the viewpoint of the driver Pd is in the right half area or the left half area of ​​the half mirror 20b when the half mirror 20b is divided into two parts, left and right. When the viewpoint of the driver Pd is in the right half area of ​​the half mirror 20b, the display control unit 12 displays a reduced image of the rear image in the lower left corner or the upper left corner, and when the viewpoint of the driver Pd is in the left half area, the display control unit 12 displays a reduced image of the rear image in the lower right corner or the upper right corner.

[0041] The image recognition unit 11 tracks the gaze area of ​​the half mirror 20b by monitoring the gaze position of the driver Pd on the half mirror 20b. The display control unit 12 adaptively moves the display position of the reduced image of the rear image so as to avoid the gaze area. When the viewpoint of the driver Pd on the half mirror 20b is switched from a state in the right half area to a state in the left half area, the display control unit 12 moves the display position of the reduced image of the rear image from the lower left corner or the upper left corner to the lower right corner or the upper right corner.

[0042] As another form of the hybrid mode, the display control unit 12 may reduce the luminance of the display 20a within a range where the driver Pd can view the rear image of the vehicle V1. Specifically, the display control unit 12 reduces the luminance of the display 20a to a degree where a faint rear image is visible behind the mirror image reflected in the half mirror 20b. The extent to which the luminance is reduced may be determined based on an experiment or a simulation. Since the hybrid mode according to this embodiment is a mode in which the mirror image is the main mode and the display image of the display 20a is the secondary mode, the luminance of the display 20a in the hybrid mode is set so that at least the visibility of the display image of the display 20a is lower than the visibility of the mirror image.

[0043] When switching from the optical mirror mode or the hybrid mode to the electronic mirror mode, the display control unit 12 causes the display 20a to display an image in which the area of ​​the rear image corresponding to the gaze area of ​​the half mirror 20b where the driver Pd is viewing is emphasized. For example, the display control unit 12 causes the display 20a to display a rear image in which the area of ​​the rear image corresponding to the gaze area of ​​the half mirror 20b is enlarged.

[0044] When switching to the electronic mirror mode, for example, if the driver Pd's viewpoint is in the left half area of ​​the half mirror 20b, the display control unit 12 shifts the cut-out range to the left within the imaging range of the solid-state imaging element of the rear imaging unit 30 to cut out an image. The display control unit 12 enlarges the cut-out image and displays it on the display 20a.

[0045] In the hybrid mode, the image recognition unit 11 tracks the gaze area of ​​the half mirror 20b where the driver Pd's viewpoint is located. For example, when the half mirror 20b is divided into two halves, left and right, the image recognition unit 11 monitors whether the driver Pd's viewpoint is in the right half area or the left half area of ​​the half mirror 20b.

[0046] The display control unit 12 moves the display range of the rear image of the vehicle V1 in accordance with the movement of the gaze area of ​​the half mirror 20b. For example, when the viewpoint of the driver Pd is in the left half area of ​​the half mirror 20b, the display control unit 12 shifts the cut-out range within the imaging range of the solid-state imaging element of the rear imaging unit 30 to the left to cut out the image, and when the viewpoint is in the right half area, the display control unit 12 shifts the cut-out range within the imaging range of the solid-state imaging element of the rear imaging unit 30 to the right to cut out the image.

[0047] Furthermore, when switching from the optical mirror mode or the hybrid mode to the electronic mirror mode, the display control unit 12 may cause the display 20a to display an image in which the brightness of the area of ​​the rear image corresponding to the gaze area of ​​the half mirror 20b where the driver Pd is looking is higher than the brightness of the other areas. The display control unit 12 may move the area with a relatively high brightness in accordance with the movement of the gaze area of ​​the half mirror 20b.

[0048] The initial mode determination unit 13 determines the initial mode when starting up the digital mirror device 1. Specifically, when starting up the digital mirror device 1, the initial mode determination unit 13 determines whether to start up in the digital mirror mode or the optical mirror mode based on at least one of the illuminance and the presence or absence of a passenger in the back seat.

[0049] When the illuminance is low, the mirror image reflected on the half mirror 20b is difficult to see in the optical mirror mode. When the illuminance is lower than a set value, the initial mode determination unit 13 starts the digital mirror device 1 in the digital mirror mode. When the illuminance is equal to or higher than a set value, the initial mode determination unit 13 may start the digital mirror device 1 in the optical mirror mode or in the digital mirror mode. For example, the digital mirror device 1 may be started in a default or user-set initial mode, or in the mode in which the digital mirror device 1 was last shut down. In order to reduce power consumption of the digital mirror device 1, the default state may be set to the optical mirror mode.

[0050] If an illuminance sensor is installed in the vehicle V1, the initial mode determination unit 13 can obtain a detection value of illuminance from the illuminance sensor. This allows the illuminance inside the vehicle or around the vehicle to be detected. The initial mode determination unit 13 may also obtain weather information from a car navigation system, VICS (Vehicle Information and Communication System) (registered trademark), etc., and may determine that the illuminance is lower than a set value when it is cloudy, raining, or snowing. The initial mode determination unit 13 may also obtain time information, and may determine that the illuminance is lower than a set value when it is nighttime.

[0051] Furthermore, when there is a passenger in the rear seat, the initial mode determination unit 13 starts up in the optical mirror mode in which the rear seat can be confirmed in principle. When there is no passenger in the rear seat, the initial mode determination unit 13 may start up in the optical mirror mode or the electronic mirror mode. For example, it may start up in the default or user-set initial mode, or it may start up in the mode at the time of the previous end of the electronic mirror device 1. For example, the initial mode determination unit 13 can determine whether there is a passenger in the rear seat based on the detection value of the seating sensor in the rear seat. Note that even if there is a passenger in the rear seat, if the illuminance is lower than the set value, the initial mode determination unit 13 starts up in the electronic mirror mode.

[0052] Figures 4(a)-(c) are diagrams showing specific examples of the viewing state of the electronic mirror display unit 20 in each mode of the electronic mirror device 1. Figure 4(a) shows the viewing state of the electronic mirror display unit 20 in the optical mirror mode, Figure 4(b) shows the viewing state of the electronic mirror display unit 20 in the electronic mirror mode, and Figure 4(c) shows the viewing state of the electronic mirror display unit 20 in the hybrid mode.

[0053] The example shown in Figures 4(a)-(c) shows a situation in which a first occupant P1 is sitting in the right rear seat of a vehicle V1, a second occupant P2 is sitting in the left rear seat, and a rear vehicle V2 is present behind the vehicle V1 in the same lane. In the optical mirror mode shown in Figure 4(a), the first occupant P1 and the second occupant P2 are visible, and the rear vehicle V2 is also visible. However, the lower part of the rear vehicle V2 is not visible due to the headrest of the rear center seat.

[0054] In the electronic mirror mode shown in Fig. 4(b), the rear vehicle V2 can be displayed in its entirety in a larger and clearer view. However, the first occupant P1 and the second occupant P2 in the vehicle interior cannot be seen. In the hybrid mode shown in Fig. 4(c), a reduced image RI of the rear image showing the rear vehicle V2 is displayed in the lower right corner of the electronic mirror display unit 20. This allows the rear vehicle V2 to be seen in its entirety.

[0055] 5(a)-(b) are diagrams showing a specific example of a case where the driver Pd's line of sight moves in the hybrid mode. Fig. 5(a) shows the viewing state of the electronic mirror display unit 20 while the driver Pd's line of sight is moving to the right, and Fig. 5(b) shows the viewing state of the electronic mirror display unit 20 when the driver Pd's line of sight movement has finished.

[0056] When the driver Pd is looking at the first occupant P1, a reduced image RI of the rear image is displayed in the lower right corner of the electronic mirror display unit 20. When the viewpoint of the driver Pd moves from the first occupant P1 to the second occupant P2, the reduced image RI of the rear image is switched to a state in which it is displayed in the lower left corner of the electronic mirror display unit 20. This improves the visibility of the second occupant P2 that the driver Pd wants to see.

[0057] FIG. 6 is a diagram showing a state immediately before switching from the optical mirror mode or hybrid mode to the electronic mirror mode (a superimposed image of the rear of the vehicle in the hybrid mode is not shown). The state shows a state in which the rear vehicle V2 is changing lanes to the right lane, and the driver Pd is following the rear vehicle V2 reflected in the upper part of the half mirror 20b with his eyes. The viewpoint of the driver Pd is present within the left half area of ​​the half mirror 20b. As will be described later, the position of the cropping range C1 of the rear image in the electronic mirror mode changes depending on the position of the viewpoint of the driver Pd.

[0058] 7(a)-(c) are diagrams showing specific examples of the cropping range C1 of the rear image in the electronic mirror mode. Fig. 7(a) shows the default cropping range C1 of the rear image, Fig. 7(b) shows the cropping range C1 of the rear image when the viewpoint of the driver Pd is in the left half area of ​​the half mirror 20b, and Fig. 7(c) shows the display range D1 of the rear image when the viewpoint of the driver Pd is in the left half area of ​​the half mirror 20b.

[0059] As shown in FIG. 7(a), in the default state, an image is cut out from the center of the imaging range 30i of the solid-state imaging element of the rear imaging unit 30, and the image of the cut-out range C1 is displayed on the display 20a. In the example shown in FIG. 7(b), when the driver Pd's viewpoint is in the left half area of ​​the half mirror 20b, the cut-out range C1 is shifted to the left, and the image cut out from the cut-out range C1 is displayed on the display 20a. In the example shown in FIG. 7(c), the image of the cut-out range C1 shifted to the left is enlarged. The display control unit 12 enlarges the image of the cut-out range C1 at a predetermined enlargement ratio by pixel interpolation. The display control unit 12 discards pixel data of the area that protrudes from the display size of the display 20a from the enlarged image. The image of the display range D1 is displayed on the display 20a.

[0060] Figures 8(a)-(c) are diagrams showing examples of display of rear images cut out from the cut-out range C1 of Figures 7(a)-(c) on the electronic mirror display unit 20. Figure 8(a) shows an example of display of a rear image cut out from the default cut-out range C1, and Figure 8(b) shows an example of display of a rear image cut out from the cut-out range C1 of the rear image when the viewpoint of the driver Pd is in the left half area of ​​the half mirror 20b. Figure 8(c) shows an example of display of a rear image cut out from the cut-out range C1 of the rear image when the viewpoint of the driver Pd is in the left half area of ​​the half mirror 20b, in an enlarged form.

[0061] In the example shown in Fig. 8(b), the rear vehicle V2 is positioned more centrally than in the example shown in Fig. 8(a), and the visibility of the rear vehicle V2 is improved. In the example shown in Fig. 8(c), the rear vehicle V2 is enlarged, and the visibility is further improved.

[0062] Fig. 9 is a diagram showing a specific example of a case where the driver Pd's line of sight moves in the electronic mirror mode. Fig. 9 shows a state where the rear vehicle V2 has further changed lanes to the right lane, and the driver Pd is following the rear vehicle V2 reflected in the electronic mirror display unit 20 to the left with his eyes.

[0063] 10(a)-(b) are diagrams showing changes in the range cut out from the rear image and displayed in the electronic mirror mode. As shown in FIG. 10(a), the display control unit 12 moves the cut-out range cut out from the image capture range 30i according to the amount of line-of-sight movement of the driver Pd detected by the image recognition unit 11. When the cut-out range reaches the end of the image capture range 30i, the display control unit 12 may further virtually move the cut-out range, or may stop the movement of the cut-out range when it reaches the end of the image capture range 30i.

[0064] The display control unit 12 enlarges the image of the moved cropped range at a predetermined enlargement rate by pixel interpolation. The display control unit 12 discards pixel data of the area of ​​the enlarged image that protrudes from the display size of the display 20a. As shown in FIG. 10(b), an image of the display range D1' is displayed on the display 20a. In the display example shown in FIG. 10(b), even though the rear vehicle V2 has moved to the right, the rear vehicle V2 is positioned in a more central position on the display 20a, improving the visibility of the rear vehicle V2.

[0065] 11 is a flowchart showing an example of the operation of the digital mirror device 1 according to the embodiment. When the driver Pd turns on the ignition of the vehicle V1 (step S10: Yes), the initial mode determination unit 13 checks the illuminance or the presence or absence of a passenger in the back seat (step S11).

[0066] If the conditions for activating the electronic mirror mode are satisfied (step S12: Yes), the initial mode determination unit 13 instructs the display control unit 12 to start up in the electronic mirror mode, and the display control unit 12 starts up the electronic mirror device 1 in the electronic mirror mode (step S13). If the conditions for activating the electronic mirror mode are not satisfied (step S12: No), the initial mode determination unit 13 instructs the display control unit 12 to start up in the optical mirror mode or hybrid mode, and the display control unit 12 starts up the electronic mirror device 1 in the optical mirror mode or hybrid mode (step S16).

[0067] In the electronic mirror mode, when the driver Pd turns off the ignition of the vehicle V1 (step S14: Yes), the operation of the electronic mirror device 1 stops. When the ignition is on (step S14: No), if the image recognition unit 11 detects the driver Pd's behavior of looking at the rear of the vehicle interior (step S15: Yes), the display control unit 12 switches the electronic mirror device 1 to the optical mirror mode or the hybrid mode (step S16). While the image recognition unit 11 does not detect the driver Pd's behavior of looking at the rear of the vehicle interior (step S15: No), the electronic mirror mode is maintained (step S13).

[0068] In the optical mirror mode or hybrid mode, when the driver Pd turns off the ignition of the vehicle V1 (step S17: Yes), the operation of the electronic mirror device 1 stops. When the ignition is on (step S17: No), if the image recognition unit 11 detects the driver Pd's behavior of looking at the rear of the vehicle (step S18: Yes), the display control unit 12 switches the electronic mirror device 1 to the electronic mirror mode (step S13). While the image recognition unit 11 does not detect the driver Pd's behavior of looking at the rear of the vehicle (step S18: No), the optical mirror mode or hybrid mode is maintained (step S16).

[0069] The control program of the digital mirror apparatus 1 may be installed in advance in the processing unit 10, or may be installed later. In the latter case, the control program may be downloaded from an application program store via a network to the digital mirror apparatus 1 and installed.

[0070] As described above, according to this embodiment, the mode of the digital mirror device 1 is automatically switched according to the line of sight or head posture of the driver Pd, eliminating the need for switch operation or voice operation for mode switching, and improving usability. In addition, by providing a hybrid mode, it is possible to grasp the overall situation of the vehicle behind while checking the situation at the rear of the vehicle cabin. In addition, in the digital mirror mode, the area that the driver Pd is gazing at is displayed in an emphasized manner, so that the driver Pd can more clearly check the object behind the vehicle that he or she is paying attention to (for example, the vehicle behind).

[0071] The present invention has been described above based on the embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0072] For example, when the driver Pd is not in the mirror viewing state, the display control unit 12 may always control the display 20a to the optical mirror mode. In this case, the power consumption and heat generation of the display 20a can be reduced, which contributes to protecting the global environment. [Explanation of symbols]

[0073] 1 Electronic mirror device, 1a Electronic mirror device main body, 10 Processing unit, 11 Image recognition unit, 12 Display control unit, 13 Initial mode determination unit, 20 Electronic mirror display unit, 20a Display, 20b Half mirror, 30 Rear imaging unit, 50 Driver imaging unit, V1 Vehicle, V2 Rear vehicle, Pd Driver, P1 First occupant, P2 Second occupant.

Claims

1. a first imaging unit that images a rear area of ​​the vehicle; an electronic mirror display unit including a display capable of displaying a rear image of the vehicle captured by the first imaging unit, and a mirror disposed on a front side of the display and reflecting light from the front and transmitting light from the rear; a second imaging unit that images a driver inside the vehicle; an image recognition unit that recognizes the behavior of the driver from the image captured by the second imaging unit; a display control unit that controls the display so that both the reflected light from the mirror and the transmitted light from the display or the reflected light from the mirror are visible when a behavior of the driver attempting to look at a rear portion of a cabin of the vehicle is detected while a rear image of the vehicle is displayed on the display; An electronic mirror device comprising:

2. When a behavior of the driver attempting to look at a rear portion of a cabin of the vehicle is detected while a rear image of the vehicle is being displayed on the display, the display control unit reduces the rear image of the vehicle and displays the reduced image of the rear image in a partial area of ​​the display. The digital mirror apparatus according to claim 1 .

3. The image recognition unit tracks a gaze area of ​​the mirror where the driver's viewpoint is located, the display control unit moves the reduced image of the rear image so as to avoid the gaze area. The digital mirror apparatus according to claim 2 .

4. a mode determination unit that determines whether to display a rear image of the vehicle on the display based on at least one of the illuminance inside or around the vehicle and the presence or absence of a passenger in the rear seat when the digital mirror device is started up; The digital mirror apparatus according to claim 1 .

5. the display control unit controls the display to display the rear image of the vehicle when a behavior of the driver attempting to look behind the vehicle is detected in a state in which the rear image of the vehicle is not displayed on the display, The image recognition unit detects a gaze area of ​​the mirror where the driver's viewpoint is located, the display control unit causes the display to display an image in which an area of ​​the rear image of the vehicle corresponding to the gaze area of ​​the mirror is emphasized. The digital mirror apparatus according to claim 1 .

6. A control program for an electronic mirror device including an electronic mirror display unit having a display capable of displaying a rear image of a vehicle captured by a first imaging unit that captures an image of the rear of the vehicle, and a mirror disposed in front of the display and reflecting light from the front and transmitting light from the rear, an image recognition function that recognizes the behavior of the driver from an image captured by a second imaging unit that captures an image of the driver inside the vehicle; a display control function that controls the display so that both the reflected light of the mirror and the transmitted light from the display or the reflected light of the mirror are visible when the driver's behavior of looking at the rear of the vehicle cabin is detected while the rear image of the vehicle is displayed on the display; A control program for the digital mirror device that causes a computer to execute the above.

Images