Visibility detection device

Abstract

To provide a visible area detection device capable of improving the accuracy of estimating a visible area even when using face direction detection data.SOLUTION: A visible area detection device has a display unit 140 that displays predetermined information to a vehicle crew member, and detects a visible area corresponding to a line-of-sight direction of the crew member toward the display unit. The visible area detection device includes: a camera 110 that captures a crew member; a line-of-sight direction calculation unit 122 that calculates the line-of-sight direction from a face image of the crew member captured with the camera; a face direction calculation unit 123 that calculates a face direction of the crew member; a learning update unit 125 that learns a relationship between the line-of-sight direction and the face direction and updates learning data; and a visible area estimation unit 127 that calculates, when the line-of-sight direction can be detected, a visible area corresponding to the line-of-sight direction and calculates, when the line-of-sight direction cannot be detected, on the basis of the learning data, the line-of-sight direction from the face direction, and estimates a visible area corresponding to the calculated line-of-sight direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a visual recognition area detection device. [Background technology]

[0002] In the visual field detection device (visual direction estimation device) described in Patent Document 1, when the visual line detected by the visual line detection unit is equal to or greater than a predetermined angle, the visual direction is estimated based on the facial direction detected by the facial direction detection unit and the driving scene identified by the driving scene identification unit.

[0003] This increases the accuracy of estimating the viewing direction by adding the driving scene condition to the face direction condition. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-154947 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the face direction relative to the gaze direction differs from person to person, further improvement in the accuracy of gaze direction estimation is required.

[0006] In view of the above problem, an object of the present disclosure is to provide a visual recognition area detection device that can improve the estimation accuracy of the visual recognition area even when face direction detection data is used. [Means for solving the problem]

[0007] In order to achieve the above object, the present disclosure employs the following technical means.

[0008] In the present disclosure, a display unit (140) that displays predetermined information to a vehicle occupant is provided, A visual recognition area detection device that detects a visual recognition area corresponding to a line of sight direction of an occupant toward a display unit, a camera (110) for photographing the occupants; a gaze direction calculation unit (122) that calculates a gaze direction from a facial image of an occupant captured by a camera; a face direction calculation unit (123) that calculates the face direction of the occupant from the face image; a learning update unit (125) that learns the relationship between the gaze direction and the face direction and updates the learning data; a visual recognition area estimation unit (127) that calculates a visual recognition area corresponding to the gaze direction when the gaze direction is detectable, and calculates the gaze direction from the face direction based on learning data when the gaze direction is undetectable, and estimates the visual recognition area corresponding to the calculated gaze direction; a brightness adjustment unit (130) that reduces the display brightness of an area of ​​the display unit that does not correspond to the visible area to be lower than the display brightness of an area that corresponds to the visible area; Equipped with.

[0009] According to the present disclosure, when the gaze direction of the occupant can be detected, the visual recognition area estimation unit can accurately calculate the visual recognition area using the gaze direction calculated by the gaze direction calculation unit. On the other hand, even when the gaze direction of the driver cannot be detected, the visual recognition area estimation unit can accurately estimate the visual recognition area by calculating the gaze direction corresponding to the face direction using the face direction calculated by the face direction calculation unit and the relationship between the gaze direction and face direction updated by the learning update unit. In other words, even when face direction detection data is used, the estimation accuracy of the visual recognition area can be improved. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram showing the configuration of a visual recognition area detection device in a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram showing details of a display unit. [Figure 3] FIG. 2 is an explanatory diagram showing the line of sight direction and the visible area. [Figure 4] FIG. 10 is an explanatory diagram illustrating an image of learning data. [Figure 5] FIG. 10 is an explanatory diagram showing the configuration of a visual recognition area detection device in a second embodiment. [Figure 6]FIG. 10 is an explanatory diagram showing the configuration of a visual recognition area detection device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, several embodiments for carrying out the present disclosure will be described with reference to the drawings. In each embodiment, parts corresponding to matters described in the preceding embodiment will be assigned the same reference numerals, and duplicate descriptions may be omitted. In each embodiment, when only a part of the configuration is described, other previously described embodiments may be applied to the other parts of the configuration. In addition to combinations of parts that are specifically specified as being combinable in each embodiment, it is also possible to partially combine embodiments even if not specified, as long as there is no particular problem with the combination.

[0012] (First embodiment) A visual recognition area detection device 100 in the first embodiment will be described with reference to Figures 1 to 4. The visual recognition area detection device 100 is mounted on a vehicle, for example, and is configured to detect a visual recognition area corresponding to the line of sight of an occupant toward a display unit 140.

[0013] The vehicle may be, for example, a private car that is used primarily by the owner, or may be shared by a family, or may be used as a rental car (shared car), etc. In this embodiment, the occupant refers to the driver, but may also include a passenger in the front seat.

[0014] 1, the visual recognition area detection device 100 includes a camera 110, a detection control unit 120, etc. Furthermore, in this embodiment, the visual recognition area detection device 100 includes a brightness adjustment unit 130 that adjusts the brightness of the display unit 140 in accordance with the estimation of the visual recognition area.

[0015] The display unit 140 is a part that displays predetermined information to the driver, and uses, for example, an organic EL (Electro Luminescence) display made up of a layered structure using organic compounds. The organic EL display is capable of forming images by controlling the light emission of each pixel.

[0016] The display unit 140 may be another type of display, such as a TFT (Thin Film Transistor) liquid crystal display that uses a thin film transistor and has a backlight as a light source.

[0017] As shown in FIG. 2, the display unit 140 is a so-called pillar-to-pillar type, which is formed on the dashboard 10 of the vehicle so as to extend in the left-right direction of the vehicle. The display area of ​​the display unit 140 has a plurality of divided display areas so as to display a plurality of different pieces of information. The display unit 140 is formed so that the plurality of displays are arranged horizontally so as to be seamless, and appear as if they were a single horizontally long display unit 140. Note that the display unit 140 may literally be formed from a single horizontally long display.

[0018] The multiple divided display areas (displays) are, for example, from the left side of the vehicle, a left side display section 140a, an entertainment display section 140b, a navigation display section 140c, an air conditioning display section 140d, a meter display section 140e, and a right side display section 140f.

[0019] The left side display unit 140a and the right side display unit 140f respectively display images of the left rear side and the right rear side of the vehicle taken by side cameras provided on the left and right sides of the vehicle. In other words, each side display unit 140a, 140f displays information equivalent to that displayed on the normal left and right door mirrors (such as the status of following vehicles and the rear lane).

[0020] The entertainment display unit 140b plays movies, music, television, and radio, and displays information on leisure, sports, games, etc. The navigation display unit 140c is a display for the navigation system, and displays information on the current position on a map and guidance information to a desired destination. The air conditioning display unit 140d displays the condition setting switches and setting status of the vehicle air conditioner. The meter display unit 140e displays various vehicle information such as the vehicle's traveling speed and engine RPM.

[0021] Each of the display sections 140a to 140f is provided with touch panel type operation buttons (icons) for, for example, setting the conditions of the associated devices, changing the display format, and the like.

[0022] The number of divided display areas (display sections 140a to 140f) is not limited to six, but may be two or more. For example, there may be a navigation display section 140c and an air conditioning display section 140d.

[0023] In visual recognition area detection device 100, camera 110 is an imaging unit (front camera) that mainly captures an image of the upper body including the face of the driver, and is provided, for example, in the rearview mirror area inside the vehicle, or in the upper center in the left-right direction of the vehicle of display unit 140. For example, a CCD camera is used as camera 110, and data on the face image captured by camera 110 is output to detection control unit 120.

[0024] It should be noted that camera 110 is not limited to a dedicated front camera as described above, but may also be a camera set up for a DSM (Driver Status Monitor, registered trademark) that detects the driver's inattentiveness or drowsiness at the wheel and provides driving assistance, or an ADAS (Advanced Driver Assistance System) that has a collision prevention function, lane departure detection function, etc., if the vehicle is equipped with such a system.

[0025] The detection control unit 120 includes an eye point calculation unit 121, a gaze direction calculation unit 122, a face direction calculation unit 123, a driver recognition unit 124, a learning update unit 125, an information storage unit 126, a visual recognition area estimation unit 127, and the like.

[0026] The eye point calculation unit 121 is a component that calculates, for example, the position between the driver's eyes as the eye point from an image of the driver's face. The eye point is a position that substitutes for the position of the driver's eyes. The eye point calculated here is a position based on the camera 110. The eye point calculation unit 121 outputs the calculated eye point data to the visual recognition area estimation unit 127.

[0027] The gaze direction calculation unit 122 is a component that calculates the gaze direction of the driver from an image of the driver's face. The gaze direction is calculated from the position of the pupil (black part of the eye) relative to the position of the eye (white part of the eye) during the driver's eye movement. As shown in Figure 3, the gaze direction is defined as the angle (gaze angle) of the direction in which the pupil faces relative to the display unit 140, which is in the up, down, left and right directions, with the driver's direct front as the reference.

[0028] Here, when the driver is wearing sunglasses, for example, gaze direction calculation unit 122 cannot detect the eyes from the face image and therefore cannot calculate the gaze direction. Therefore, when the driver is not wearing sunglasses and the eyes are detectable, gaze direction calculation unit 122 outputs gaze direction data to learning update unit 125 and visual recognition area estimation unit 127.

[0029] The face direction calculation unit 123 is a component that calculates the face direction (face orientation) of the driver from the face image of the driver. The face direction is calculated based on the positions of the driver's eyes, eyebrows, nose, mouth, ears, etc. The face direction calculation unit 123 outputs the calculated face direction data to the learning update unit 125 and the visual recognition area estimation unit 127.

[0030] The driver recognition unit 124 takes into consideration cases where the vehicle is shared by a family or used as a shared car, etc., and identifies the driver currently driving from a pre-registered facial image and outputs this as driver identification information (Mr. A, Mr. B, Mr. C, etc.) to the learning update unit 125.

[0031] The driver recognition unit 124 is not limited to using a facial image, and may also identify the driver using information such as an ID authentication number provided on a vehicle key or mobile terminal carried by the driver, the weight of the driver, and the seat setting position.

[0032] The learning update unit 125 is a component that learns the relationship between the gaze direction and the face direction and sequentially updates the learned data. For example, while driving, the learning update unit 125 associates (learns) the gaze direction with the face direction at predetermined time intervals using data on the gaze direction calculated by the gaze direction calculation unit 122 and the face direction calculated by the face direction calculation unit 123, as shown in FIG. 4, and updates (stores) the learned data. The learned data is created and updated for each driver (A, B, C, etc.) identified by the driver recognition unit 124. The learning update unit 125 outputs the updated learned data to the visual recognition area estimation unit 127.

[0033] The learning update unit 125 stores, in advance, initial data, such as the relationship between the gaze direction and face direction of an average adult, and this initial data is used for control the first time the driver gets in. The initial data is updated sequentially through the above-mentioned learning.

[0034] The information storage unit 126 is a part into which the position information of the camera 110 within the vehicle and the position information of each divided display area of ​​the display unit 140 (each display unit 140a to 140f) are input in advance, and outputs the various input position information to the visible area estimation unit 127.

[0035] Visual recognition area estimation unit 127 converts the coordinate position of the eye point calculated by eye point calculation unit 121 with camera 110 as the reference into a coordinate position relative to display unit 140. Then, visual recognition area estimation unit 127 calculates distance L from the eye point to display unit 140 in the visual line direction calculated by visual line direction calculation unit 122 (direction of the visual line angle with respect to the front), as shown in FIG.

[0036] Furthermore, when the driver is not wearing sunglasses and the line of sight of the driver can be detected, the visual recognition area estimation unit 127 calculates the visual recognition area corresponding to the line of sight direction calculated by the line of sight direction calculation unit 122.

[0037] Furthermore, when the driver is wearing sunglasses and the driver's gaze direction cannot be detected, the visual recognition area estimation unit 127 calculates the gaze direction from the face direction based on the face direction data from the face direction calculation unit 123 and the learning data from the learning update unit 125, and estimates the visual recognition area corresponding to the calculated gaze direction.

[0038] As shown in FIG. 3, the effective viewing angle relative to the line of sight is usually about ±10 to 20 degrees, and the visible area is the area within the effective viewing angle that extends across display unit 140 at a distance L from the line of sight.

[0039] The brightness adjustment unit 130 is a component that reduces the display brightness of areas that do not correspond to the visible area, among the display areas (divided display areas) of the display unit 140, to be lower than the display brightness of areas that correspond to the visible area. The brightness adjustment unit 130 is set in advance to a brightness level to be reduced.

[0040] When the display unit 140 uses an organic EL display, each pixel (picture element) is self-luminous, so brightness adjustment for each pixel is possible, and the brightness adjustment unit 130 can reduce the display brightness of only the visible area, even within one divided display area (display).

[0041] If the display unit 140 is a liquid crystal display, the light source is a backlight, making it difficult to adjust the brightness for each pixel, so the brightness adjustment unit 130 may reduce the display brightness of the divided display areas (displays) that do not include the visible area. In other words, the brightness may be adjusted for each divided display area (display).

[0042] The visual recognition area detection device 100 of this embodiment is configured as described above, and its operation, functions, and effects will be described below.

[0043] The position information of the camera 110 and the display unit 140 in the information storage unit 126 is output to the visual recognition area estimation unit 127 .

[0044] The learning update unit 125 learns the relationship between the face direction and the gaze direction from the data of the driver's face direction obtained by the face direction calculation unit 123 and the data of the driver's gaze direction obtained by the gaze direction calculation unit 122, and stores and updates the learned data. This learned data is stored and updated for each different driver by the driver recognition unit 124 (FIG. 4).

[0045] The coordinate position of the eye point calculated by the eye point calculation unit 121 with respect to the camera 110 as the reference is converted into a coordinate position relative to the display unit 140 by the visual recognition area estimation unit 127, and the distance L to the display unit 140 in the visual line direction is calculated by the visual line direction calculation unit 122.

[0046] Furthermore, when the driver is not wearing sunglasses, the visual recognition area estimation unit 127 calculates the visual recognition area (FIG. 3) corresponding to the data of the driver's line of sight direction obtained by the line of sight direction calculation unit 122.

[0047] Furthermore, when the driver is wearing sunglasses and the driver's gaze direction cannot be detected, visual recognition area estimation unit 127 calculates the gaze direction from the face direction based on the face direction data from face direction calculation unit 123 and the learning data stored and updated in learning update unit 125 (FIG. 4), and estimates the visual recognition area (FIG. 3) corresponding to the calculated gaze direction.

[0048] Data on the visual recognition area calculated or estimated by visual recognition area estimation unit 127 is output to brightness adjustment unit 130. Brightness adjustment unit 130 reduces the brightness of areas other than the visual recognition area of ​​display unit 140 to be lower than the display brightness of the visual recognition area. In other words, the brightness of the area that the driver is viewing is maintained, and the brightness of the area that the driver is not viewing is reduced.

[0049] As described above, according to this embodiment, when the driver's gaze direction is detectable, visual recognition area estimation unit 127 can accurately calculate the visual recognition area using the gaze direction calculated by gaze direction calculation unit 122. On the other hand, even when the driver's gaze direction is undetectable, visual recognition area estimation unit 127 can accurately estimate the visual recognition area by calculating the gaze direction corresponding to the face direction using the face direction calculated by face direction calculation unit 123 and the relationship between the gaze direction and face direction updated by learning update unit 125. In other words, even when face direction detection data is used, the estimation accuracy of the visual recognition area can be improved.

[0050] Furthermore, the brightness adjustment unit 130 reduces the display brightness of the area that does not correspond to the visible area to be lower than the display brightness of the area that corresponds to the visible area, thereby enabling power saving of the display unit 140 that displays multiple different pieces of display information.

[0051] When the display unit 140 is an organic EL display, it is possible to adjust the brightness of pixels corresponding to the visible area within the display. However, if there are areas where the brightness is maintained and areas where the brightness is reduced within a display that displays one piece of display information, the driver may feel uncomfortable. Therefore, it is better to adjust the brightness (reduce the brightness) for the display that does not include the visible area.

[0052] (Second embodiment) A visual recognition area detecting device 100A of the second embodiment is shown in Fig. 5. The visual recognition area detecting device 100A of the second embodiment is obtained by adding an operation detecting unit 128 to the visual recognition area detecting device 100 of the first embodiment.

[0053] The operation detection unit 128 is a part that detects whether or not a predetermined operation has been performed by the driver, and outputs a detected operation signal to the learning update unit 125.

[0054] The predetermined operation is, for example, a touch operation on an operation button of each display on the display unit 140, or an operation on a vehicle operation unit related to the display on the display unit 140 (for example, a turn signal, etc.).

[0055] A touch operation on an operation button can be determined from the position of the driver's arm and a hand image accompanying the movement in the image of camera 110. An operation on a vehicle operation unit can also be determined from an operation signal accompanying the operation (for example, an ON signal for a turn signal).

[0056] When an operation button is touched, the driver looks at the associated display of the display unit 140. When a turn signal is operated, the driver looks at the display (side display units 140a, 140f) to check for safety around the vehicle turning direction. Therefore, when such an operation signal is detected, the learning update unit 125 learns the relationship between the face direction and the line of sight direction and updates the learning data.

[0057] This allows the display unit 140 (each display) to set a timing for learning when the driver's facial direction becomes clear from a hand image or an operation signal, thereby improving the reliability of the learning data.

[0058] (Third embodiment) A visual recognition area detecting device 100B of the third embodiment is shown in Fig. 6. The visual recognition area detecting device 100B of the third embodiment is obtained by adding a visual recognition instruction unit 129 to the visual recognition area detecting device 100 of the first embodiment.

[0059] The visual instruction unit 129 issues a visual instruction to the driver on the display unit 140, for example, by voice (or by display on the display unit 140). The visual instruction is an instruction to intentionally instruct the driver to look at a predetermined display unit 140 (display), and examples of such an instruction include "Look at the center of the navigation display unit 140c" or "Look at the side display units 140a and 140f." The visual instruction may be generated, for example, when the driver gets into the vehicle for the first time (first time).

[0060] At the time of issuing a viewing instruction, the viewing instruction unit 129 outputs a viewing instruction signal to the learning update unit 125. When such a viewing instruction signal is output, the learning update unit 125 learns the relationship between the face direction and the gaze direction and updates the learning data.

[0061] This allows for more reliable learning data to be obtained from pre-stored initial data (for example, the relationship between the gaze direction and face direction of an average adult). In other words, highly reliable estimation of the visual recognition area becomes possible from the first time the vehicle is driven.

[0062] (Other embodiments) The disclosure in this specification and drawings, etc. is not limited to the exemplified embodiments. The disclosure encompasses the exemplified embodiments and modifications thereto by those skilled in the art. For example, the disclosure is not limited to the combinations of parts and / or elements shown in the embodiments. The disclosure can be implemented in various combinations. The disclosure can have additional parts that can be added to the embodiments. The disclosure encompasses the omission of parts and / or elements from the embodiments. The disclosure encompasses the substitution or combination of parts and / or elements between one embodiment and another embodiment. The disclosed technical scope is not limited to the description of the embodiments. Some disclosed technical scopes are defined by the claims, and should be interpreted as including all modifications within the meaning and scope equivalent to the claims.

[0063] In the above embodiments, the visual recognition area of ​​the driver as an occupant is calculated and estimated, but the visual recognition area of ​​the passenger in the front passenger seat may be calculated and estimated and reflected in the brightness control of the display unit 140. In this case, it is preferable to prioritize the condition for maintaining the display brightness for the driver's visual recognition area. [Explanation of symbols]

[0064] 100, 100A, 100B Visibility area detection device 110 Camera 122 Gaze direction calculation unit 123 Face direction calculation unit 125 Learning Update Department 127 Visibility Area Estimation Unit 129 Visual Indicator 130 Brightness adjustment section 140 Display section 140a Left side display section (multiple divided display areas, displays) 140b Entertainment display section (multiple divided display areas, displays) 140c Navigation display (multiple split display areas, display) 140d Air conditioning display (multiple split display areas, display) 140e Meter display (multiple divided display areas, display) 140f Right side display (multiple split display areas, displays)

Claims

1. a display unit (140) for displaying predetermined information to a vehicle occupant; A visual recognition area detection device that detects a visual recognition area corresponding to a line of sight direction of the occupant toward the display unit, a camera (110) for photographing the occupant; a gaze direction calculation unit (122) that calculates the gaze direction from the facial image of the occupant captured by the camera; a face direction calculation unit (123) that calculates the face direction of the occupant from the face image; a learning update unit (125) that learns the relationship between the gaze direction and the face direction and updates learning data; a visual recognition area estimation unit (127) that, when the gaze direction is detectable, calculates the visual recognition area corresponding to the gaze direction, and, when the gaze direction is undetectable, calculates the gaze direction from the face direction based on the learning data and estimates the visual recognition area corresponding to the calculated gaze direction; a brightness adjustment unit (130) that reduces the display brightness of an area of ​​the display area of ​​the display unit that does not correspond to the visible area to be lower than the display brightness of an area that corresponds to the visible area.

2. The visual recognition area detecting device according to claim 1 , wherein the learning update unit performs the learning and updates the learning data when a predetermined operation is performed by the occupant.

3. An instruction unit (129) for instructing the occupant to view the display unit, The visual recognition area detecting device according to claim 1 , wherein the learning update unit performs the learning and updates the learning data when the visual recognition instruction is received.

4. The display area has a plurality of divided display areas (140a to 140f) for displaying a plurality of different pieces of information, 4. The visual recognition area detection device according to claim 1, wherein the brightness adjustment section reduces the display brightness of the divided display area that does not include the visual recognition area.

Images