Image display device

The image display device in vehicles automatically switches between a mirror and a camera image based on gaze point distance and line of sight, addressing the inefficiency of manual switching and improving visibility by adapting to the driver's needs.

JP7577418B2Active Publication Date: 2024-11-05ALPS ALPINE CO LTD

Patent Information

Application Number
JP2021049825
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-24
Publication Date
2024-11-05
Estimated Expiration
2041-03-24

AI Technical Summary

Technical Problem

Existing in-vehicle display devices require manual switching between a liquid crystal display and a half-mirror, complicating operations when switching between rearward image capture and real-time viewing, which is inefficient and inconvenient.

Method used

An image display device that includes a rearview mirror with a display screen, a rear camera, and an interior camera to automatically switch between displaying a mirror image and a camera image based on the driver's gaze point distance and line of sight, using gaze point distance calculation and line-of-sight specification to determine the appropriate display mode.

Benefits of technology

Automatically adjusts the display mode to match the driver's viewing needs, eliminating the need for manual switching and enhancing visibility by optimizing the use of either a real mirror or electronic mirror based on the usage situation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577418000001
    Figure 0007577418000001
  • Figure 0007577418000002
    Figure 0007577418000002
  • Figure 0007577418000003
    Figure 0007577418000003
Patent Text Reader

Abstract

To provide an image display device capable of eliminating troublesomeness when switching mirrors between an electronic mirror and a real mirror for each use status.SOLUTION: An image display device 100 includes: a room mirror 200; a rear camera 52 for imaging the rear of a vehicle; a rear image creation unit 31 for creating a rear image displayed in a display screen using the image taken with the rear camera 52; a display switching unit 32 for switching the states between a first state as a real mirror by which a driver can check the rear of a vehicle reflected by the room mirror 200 in a state where a display of the rear image is stopped, and a second state as an electronic mirror by which the driver can check the rear image displayed on the display screen through the room mirror 200 in a state where the rear image is displayed; and a fixation point distance calculation unit 35 for calculation a distance to a fixation point of the driver. When the distance to the fixation point of the driver viewing the room mirror 200 in the first state is longer than a predetermined value, the display switching unit 32 switches the state to the second state.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image display device that switches between displaying a mirror image of the rear of a vehicle optically reflected by a mirror and displaying an image of the rear of the vehicle captured by a camera. [Background technology]

[0002] Conventionally, there has been known an in-vehicle display device that detects the convergence movement when the driver moves his / her line of sight and switches between a liquid crystal display and a half mirror display (see, for example, Patent Document 1). This in-vehicle display device detects the amount of change in the convergence angle when the driver moves his / her line of sight from the front of the vehicle to the display, and switches the display by turning the liquid crystal display ON or OFF based on this amount of change. This reduces the strain on the driver's eyes caused by sudden focus adjustment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-30507 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a rear image captured by a camera attached to the rear of a vehicle is displayed on a liquid crystal display, the field of view is not obstructed by pillars or the like, so visibility can be improved compared to when viewing the rear of the vehicle through a half mirror, and it is expected that this type of usage will increase in the future. On the other hand, this camera is attached to a position where it can capture an image of the rear of the vehicle, such as the rear of the vehicle or the upper part of the inside of the rear window, and passengers and luggage in the rear seats inside the vehicle are outside the range that can be captured, so when it is desired to visually check passengers and luggage in the rear seats, it is desirable to view the rear through the half mirror. However, in the past, when switching between the use of the liquid crystal display (electronic mirror) and the use of the half mirror (real mirror) for each of these usage situations, it was necessary to manually perform the switching operation each time, which caused a problem of cumbersome operations.

[0005] In addition, the vehicle display device disclosed in the above-mentioned Patent Document 1 switches between using a half mirror and using an LCD display depending on the convergence angle when the line of sight is shifted to the display, and cannot be applied to the above-mentioned switching operation.

[0006] The present invention has been created in consideration of these points, and its purpose is to provide an image display device that can eliminate the complexity of operations required when switching between an electronic mirror and a real mirror depending on the usage situation. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, an image display device of the present invention includes a mirror that reflects the rear of the vehicle and transmits an image on a display screen arranged on the rear side, a first camera that captures an image of the rear of the vehicle, a rear image creation means that creates a rear image to be displayed on the display screen using an image captured by the first camera, a display switching means that switches between a first state in which the driver can check the rear of the vehicle reflected in the mirror with the rear image display stopped, and a second state in which the driver can check the rear image displayed on the display screen through the mirror with the rear image display started, and a gaze point distance calculation means that calculates a distance to the driver's gaze point, and the display switching means switches between the first state and the second state when the distance to the gaze point of the driver looking at the mirror is longer than a predetermined value in the first state. of Switching to status.

[0008] By calculating the distance to the driver's gaze point when looking at the mirror, it is possible to distinguish whether the driver is looking at the distant rear of the vehicle or the close interior of the vehicle, and the system can automatically switch between the first state (real mirror) and the second state (electronic mirror) depending on the usage situation, eliminating the hassle of manual switching operations.

[0009] It is also preferable to further include a second camera that captures the driver's face, and the gaze point distance calculation means calculates the distance to the driver's gaze point based on the facial image captured by the second camera. Alternatively, it is preferable to further include a gaze identification means that identifies the direction of the driver's gaze based on the facial image captured by the above-mentioned second camera, and the display switching means determines whether the driver is looking at the mirror based on the gaze direction identified by the gaze identification means. It is possible to easily obtain the distance to the driver's gaze point and the gaze direction by simply capturing an image of the driver's face.

[0010] It is also preferable to further include a rear seat detection means for detecting the presence or absence of at least one of passengers and luggage in the rear seats in the vehicle interior, and to switch the display switching means when the rear seat detection means detects passengers or luggage. This allows the display to be switched to the second state (electronic mirror) in which the rear image obtained by imaging is displayed when there is no need to check the state inside the vehicle interior, such as when there are no passengers or luggage in the rear seats, thereby improving visibility behind the vehicle.

[0011] In addition, when the line of sight identified by the line of sight identifying means moves away from the mirror, it is desirable that the display switching means maintains the first state if it was in the first state until then, and switches to the first state if it was in the second state until then. When the driver's line of sight moves away from the mirror, it is possible to perform a switching determination based on the distance to the gaze point again, and it becomes possible to check the state of the interior of the vehicle after first checking the rear of the vehicle. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an image display device according to an embodiment. [Diagram 2] 4 is a functional block diagram of a rear image display processing unit that operates when a display switching program is executed by a CPU. [Diagram 3] FIG. 11 is a diagram showing a specific example of calculation of a distance to a gaze point. [Figure 4] 1 is a flowchart showing an operational procedure for switching between a real mirror and an E-mirror for checking the rear of the vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an image display device according to an embodiment of the present invention will be described with reference to the drawings.

[0014] Fig. 1 is a diagram showing the configuration of an image display device according to an embodiment. As shown in Fig. 1, the image display device 100 according to this embodiment includes a display changeover SW (switch) 10, a display processing unit 20, a display device 22, a CPU 30, a memory 40, an input / output interface unit (I / O IF) 50, a rear camera 52, and an in-vehicle camera 54. This image display device 100 creates an image (rear image) captured behind a vehicle, and displays the image in place of a mirror image in a rearview mirror 200.

[0015] The rearview mirror 200 is attached to the upper part of the windshield in the center between the driver's seat and the passenger seat of the vehicle. The rearview mirror 200 is equipped with a display device 22 on the back side, and when an image is displayed on the display screen, the displayed image is visible to the driver through the mirror surface.

[0016] The display switching SW10 is used by the driver of the vehicle to manually switch between a first state in which the rearview mirror 200 reflects a mirror image of the area behind the vehicle, and a second state in which an image on the display screen is displayed through the rearview mirror 200.

[0017] The display processing unit 20 uses an image captured by the rear camera 52 behind the vehicle to create a rear image to be displayed on the display screen of the display device 22. The display device 22 has a display screen using, for example, a liquid crystal display (LCD). This display screen is disposed behind the rearview mirror 200, and when the display screen is dark, the mirror image by the rearview mirror 200 is visible to the driver, and when the display screen is bright (when an image is displayed), the image displayed on the display screen is visible through the mirror surface of the rearview mirror 200 to the driver.

[0018] The CPU 30 executes a predetermined program stored in the memory 40 to control the entire image display device 100. The CPU 30 also creates a rear image for display using an image captured by the rear camera 52, and performs processing to switch between a first state in which the rear view of the vehicle is reflected by the rearview mirror 200 and a second state in which an image on the display screen is displayed through the rearview mirror 200, depending on the operation state of the display switching SW 10 or automatically based on the result of observing the driver.

[0019] The memory 40 stores the operation program of the CPU 30 and is used as a working area for storing various data necessary for the operation of the CPU 30. The operation program executed by the CPU 30 includes a program (display switching program) for automatically switching between the first state and the second state described above.

[0020] The input / output interface unit 50 performs input / output processing of signals (data) between various input / output devices including a rear camera 52 and an in-vehicle camera 54 .

[0021] The rear camera 52 is attached to a predetermined position at the rear of the vehicle (for example, above the rear window) and captures an image of the same range as the mirror image reflected in the rearview mirror 200 or a wider range than this. The rearview camera 52 is used to capture an image of the rear of the vehicle that is the same as or wider than the view behind the vehicle reflected on the mirror surface of the rearview mirror 200.

[0022] The in-vehicle camera 54 is for capturing images of the driver's face including the eyes, passengers in the back seat, and luggage, and captures a predetermined range in the vehicle interior including the driver and the back seat. The image of the driver's face is used to estimate the direction of the driver's line of sight and the distance to the driver's gaze point (focus point). For this reason, the in-vehicle camera 54 is attached to a position where the driver's face can be captured at an angle close to the front, such as the upper part of the windshield. Note that a camera for capturing images of the driver's face and a camera for capturing images of passengers in the back seat and luggage may be provided separately.

[0023] 2 is a functional block diagram of the rear image display processing unit 30A that operates by executing a display switching program by the CPU 30. This rear image display processing unit 30A includes a rear image creation unit 31, a display switching unit 32, a rear seat determination unit 33, a line of sight identification unit 34, and a gaze point distance calculation unit 35.

[0024] The rear image creating unit 31 creates a rear image to be displayed on the display screen of the display device 22 using an image captured by the rear camera 52 .

[0025] The display switching unit 32 switches between a first state (real mirror) in which a mirror image of the rear of the vehicle is reflected by the rearview mirror 200 and a second state (E-mirror (electronic mirror)) in which a rear image on a display screen is displayed through the rearview mirror 200, in response to a manual operation by the driver to turn the display switching SW10 on / off, or automatically based on the distance to the driver's gaze point. A specific example of the automatic switching will be described later.

[0026] The rear seat determination unit 33 determines the presence or absence of at least one of a passenger and luggage in the rear seat based on an image captured by the interior camera 54. Since the image obtained by capturing images of the interior camera 54 includes the driver, the driver's seat, the passenger seat, etc., the presence or absence of a passenger and luggage in the rear seat is determined by performing image recognition processing on the range excluding these. Note that this determination is not limited to being made based on an image captured by the interior camera 54. For example, a sensor may be placed on the seat surface of the rear seat, and the presence or absence of a passenger / luggage may be determined when the weight of the passenger or luggage is detected.

[0027] The gaze identifying unit 34 identifies the direction of the driver's gaze based on the face image of the driver included in the image captured by the in-vehicle camera 54. For example, the center of the pupil of the eye is extracted from the face image of the driver, and the line connecting the pupil center and the center of the eyeball is identified as the gaze direction.

[0028] The gaze point distance calculation unit 35 calculates the distance to the driver's gaze point based on the image captured by the in-vehicle camera 54. For example, the above-mentioned gaze direction is specified for each of the eyes from the face image of the driver, and the intersection point is set as the position of the gaze point (focus point), and the distance to that position (gaze point distance) is calculated. Note that this method of specifying the gaze point is only one example, and methods other than the above may be used. For example, it may be possible to determine whether the driver is looking at a nearby position or a distant position based on the degree of pupil narrowing.

[0029] 3A, 3B, and 3C are diagrams showing specific examples of calculation of distance to the gaze point. In Fig. 3A, 3B, and 3C, P, P1, and P2 indicate the position of the gaze target (gaze point), GL indicates the driver's left eye, and GR indicates the driver's right eye, respectively.

[0030] In Fig. 3(A), the direction of the line of sight a of the driver's left eye can be specified as the direction of a line connecting the center of the pupil of the left eye and the center of the eyeball. Similarly, the direction of the line of sight b of the driver's right eye can be specified as the direction of a line connecting the center of the pupil of the right eye and the center of the eyeball. When looking at the rear of the vehicle through the rearview mirror 200, the actual line of sight is reflected by the rearview mirror 200, but Fig. 3 shows the line of sight corresponding to the virtual image seen through the rearview mirror 200.

[0031] As shown in FIG. 3(A), the convergence angle α of a driver looking at a gaze point P can be calculated as the crossing angle between the line of sight a of the left eye and the line of sight b of the right eye. As shown in FIG. 3(B), when the driver looks at a position behind the vehicle (gazing point P1), the convergence angle α1 is small. On the other hand, as shown in FIG. 3(C), when the driver looks at passengers or luggage in the rear seat inside the vehicle (gazing point P2), the convergence angle α2 is large. In this way, since there is a one-to-one relationship between the convergence angle α and the distance to the gaze point P, the distance to the gaze point P can be calculated by finding the convergence angle α.

[0032] The above-mentioned rear camera 52 corresponds to the first camera, the rear image creation unit 31 corresponds to the rear image creation means, the display switching unit 32 corresponds to the display switching means, the gaze point distance calculation unit 35 corresponds to the gaze point distance calculation means, the in-vehicle camera 54 corresponds to the second camera, the gaze identification unit 34 corresponds to the gaze identification means, and the rear seat determination unit 33 corresponds to the rear seat determination means.

[0033] The image display device 100 of this embodiment has the above-mentioned configuration, and its operation will now be described.

[0034] FIG. 4 is a flowchart showing an operation procedure for switching between the real mirror and the E-mirror for checking the rear of the vehicle.

[0035] Immediately after turning on the accessory switch of the vehicle and starting up the image display device 100, the display switching unit 32 sets the rearview mirror 200 to a first state (real mirror) in which the rearview mirror reflects the image of the rear of the vehicle (step 100). For example, this first state can be realized by stopping the display of the display device 22 and darkening the display screen.

[0036] Next, the rear seat determination unit 33 determines whether or not there is a passenger (rear passenger) and luggage in the rear seat (step 102). The imaging by the vehicle interior camera 54 is performed in parallel with the above-mentioned step 100 and each subsequent step, and the latest captured image is used as appropriate to determine the presence or absence of a passenger / luggage in the rear seat, identify the driver's line of sight, calculate the gaze point, and the like. Similarly, the imaging operation of the rear of the vehicle by the rear camera 52 and the creation operation of the rear image by the rear image creation unit 31 are performed in parallel with the operations of each step, and the contents are updated as appropriate. If there is no passenger or luggage in the rear seat, a negative determination is made in the determination of step 102. In this case, the display switching unit 32 enables the display of the display device 22 and sets the display screen of the display device 22 to a second state (E mirror) in which the rear image created by the rear image creation unit 31 is displayed (step 120). On the other hand, if there is a passenger or luggage in the rear seat, a positive determination is made in the determination of step 102.

[0037] Next, the line-of-sight identifying unit 34 identifies the direction of the driver's line of sight (step 104). The display switching unit 32 also determines whether the driver is looking at the rearview mirror 200 based on the identified line of sight direction (step 106). If the driver is not looking, a negative determination is made. For example, this may be the case when the driver continues to look ahead of the vehicle, in which case the process returns to step 100 and the setting of the real mirror is maintained.

[0038] Also, if the driver is looking at the rearview mirror 200, a positive determination is made in the determination of step 106. Next, the gaze point distance calculation unit 35 calculates the distance to the gaze point of the driver looking at the rearview mirror 200 (step 108). Next, the display switching unit 32 determines whether the calculated distance to the gaze point is longer than a predetermined value (step 110). If the driver is looking at the rear of the vehicle far away through the mirror, the distance to the gaze point is long, and if the driver is looking at passengers or luggage in the rear seats inside the vehicle, the distance to the gaze point is short. The determination of step 110 is for distinguishing between these two situations, and this distinction can be made by using a value of about 5 m as the predetermined value, for example.

[0039] If the distance to the gaze point is not longer than the predetermined value (if the person is looking at something close), a negative determination is made in step 110. In this case, the process returns to step 100 and the real mirror setting is maintained.

[0040] Furthermore, if the distance to the gaze point is longer than a predetermined value (if the driver is looking far away), a positive determination is made in the determination of step 110. In this case, the display switching unit 32 enables the display of the display device 22, and sets the display screen of the display device 22 to a second state (E mirror) in which the rearward image created by the rearward image creation unit 31 is displayed (step 112). This allows the driver to see the rearward image captured by the rear camera 52 through the rearview mirror 200.

[0041] Next, the line of sight identifying unit 34 identifies the direction of the driver's line of sight (step 114). Furthermore, the display switching unit 32 determines whether or not the direction of the driver's line of sight has deviated from the rearview mirror 200 based on the identified line of sight direction (step 116). If the direction has not deviated, a negative determination is made, and the process returns to step 114 to repeat the operation of identifying the direction of the driver's line of sight. Furthermore, if the direction of the driver's line of sight has deviated from the rearview mirror 200, a positive determination is made in the determination of step 116. In this case, the process returns to step 100 to repeat the operations following the setting of the real mirror.

[0042] In this way, in the image display device 100 of this embodiment, by calculating the distance to the gaze point of the driver looking at the rearview mirror 200, it is possible to distinguish whether the driver is looking at the distant rear of the vehicle or the close interior of the vehicle, and it is possible to automatically switch between the real mirror and the E-mirror (electronic mirror) depending on the usage situation, thereby eliminating the cumbersome manual switching operation.

[0043] In addition, by simply capturing an image of the driver's face with the in-vehicle camera 54, the distance to the driver's gaze point and the direction of the driver's line of sight can be easily obtained.

[0044] In addition, when there is no need to check the state of the interior of the vehicle, such as when there are no passengers in the back seats or no luggage, the E-mirror can be used to display a rear image obtained by capturing an image, thereby improving visibility behind the vehicle.

[0045] In addition, when the driver's line of sight moves away from the rearview mirror 200, even if the E-mirror was being used up until then, the E-mirror is temporarily returned to the real mirror, so that a switching decision can be made again based on the distance to the gaze point, and the driver can check the state of the interior of the vehicle after checking the rear of the vehicle with good visibility.

[0046] The present invention is not limited to the above embodiment, and various modifications are possible within the scope of the present invention. For example, in the above embodiment, the presence or absence of passengers or luggage in the rear seats is determined (step 102 in FIG. 4), and if they are not present, the E-mirror is set without determining whether to switch between the real mirror and the E-mirror. However, these steps may be omitted, and the operation of step 100 (setting the real mirror) may be immediately followed by the operation of step 104 (identifying the driver's line of sight). [Industrial Applicability]

[0047] As described above, according to the present invention, by calculating the distance to the driver's gaze point when looking at the mirror, it is possible to distinguish whether the driver is looking at the distant rear of the vehicle or the close interior of the vehicle, and it is possible to automatically switch between the first state (real mirror) and the second state (electronic mirror) depending on the usage situation, thereby eliminating the cumbersome task of manual switching operations. [Explanation of symbols]

[0048] 10 Display switch 20 Display processing section 22 Display device 30 CPU 30A Rear image display processing unit 31 Rear image creation section 32 Display switching section 33 Rear seat judgment section 34 Line of sight identification part 35 Gaze point distance calculation unit 40 Memory 50 Input / Output Interface (I / O IF) 52 Rear camera 54 In-car camera 100 Image display device 200 Rearview mirror

Claims

1. A mirror that reflects the rear of the vehicle and transmits the image of a display screen arranged on the back side; a first camera for capturing an image of a rear area of ​​the vehicle; a rear image creating means for creating a rear image to be displayed on the display screen using an image captured by the first camera; a display switching means for switching between a first state in which the display of the rear image is stopped and the driver can check the rear of the vehicle reflected in the mirror, and a second state in which the display of the rear image is started and the driver can check the rear image displayed on the display screen through the mirror; A gaze point distance calculation means for calculating a distance to a driver's gaze point; and wherein the display switching means switches to the second state when a distance to a gaze point of a driver looking at the mirror is longer than a predetermined value in the first state.

2. Further comprising a second camera for capturing an image of the face of the driver; 2. The image display device according to claim 1, wherein the gaze point distance calculation means calculates the distance to the driver's gaze point based on a face image captured by the second camera.

3. The vehicle further includes a gaze direction determining means for determining a direction of a driver's gaze based on a face image captured by the second camera, 3. The image display device according to claim 2, wherein the display switching means determines whether or not the driver is looking at the mirror based on the direction of the line of sight identified by the line of sight identifying means.

4. a rear seat detection means for detecting the presence or absence of at least one of passengers and luggage in the rear seats of the vehicle interior; 4. The image display device according to claim 1, wherein the display switching means performs switching when a passenger or baggage is detected by the rear seat detection means.

5. The image display device described in claim 3, characterized in that when the direction of the line of sight identified by the line of sight identification means moves away from the mirror, the display switching means maintains the first state if it was in the first state until then, and switches to the first state if it was in the second state until then.

Citation Information

Patent Citations

  • Image display device

    JP2005335410A

  • Ambient status displaying device for vehicle

    JP2006135797A

  • Visibility provider and warning device

    JP2009280196A

  • On-vehicle display device

    JP2016030507A

  • Vehicular display device

    JP2017213935A

Cited By

  • Rear-view mirror switching modes in conjunction with gears and control method thereof

    WO2026141996A1