Line-of-sight detection device, line-of-sight detection method, and computer program product

Abstract

The invention provides a line-of-sight detection device, a line-of-sight detection method, and a computer program product capable of quickly determining a change in a line-of-sight position using a specific eye movement. An operation assistance unit (2) that functions as a line-of-sight detection device is provided with: an inter-pupil distance detection unit (13) that detects the inter-pupil distance (Dp) of a driver (D) that is a subject; and a line-of-sight detection unit (14) that detects the line-of-sight position of the driver (D), the line-of-sight detection unit (14) having a line-of-sight change determination unit (14a) that determines the line-of-sight change of the driver (D) when detecting that a change of a predetermined amount or more of the interpupillary distance (Dp) has occurred twice eye movements within a predetermined time period, and the line-of-sight change determination unit (14a) determines the line-of-sight change of the driver (D). It is determined that the line-of-sight position of the driver (D) changes with a change in the line-of-sight.

Description

Technical Field

[0001] This invention relates to a line-of-sight detection device, a line-of-sight detection method, and a computer program product. Background Technology

[0002] In recent years, research and development have been conducted to assist in obtaining data related to the interpupillary distance of users.

[0003] For example, Patent Document 1 discloses a technique for determining the direction of gaze and the state of eye movement based on the positions of a first pupil coordinate, a second pupil coordinate, a first inner canthus coordinate, and a second inner canthus coordinate. Furthermore, it describes a state of eye movement where one of the following is a threshold value (i.e., eye position deviation): a state of eye movement involving vertical gaze movement, horizontal gaze movement, convergence, or divergence, and a change in pupil position where one of these changes is above a threshold value while the other is below a threshold value.

[0004] [Existing Technical Documents]

[0005] [Patent Literature]

[0006] Patent Document 1: International Publication No. 2020 / 152732 Summary of the Invention

[0007] [The problem the invention aims to solve]

[0008] However, previous technologies only detected gaze position based on the left and right pupil positions, without taking into account the specific eye movements of living organisms such as humans. Therefore, depending on the processing power of the processor used to detect gaze position, there is a possibility that changes in gaze position cannot be determined quickly. Therefore, the objective of this application is to enable the rapid determination of changes in gaze position using specific eye movements.

[0009] This application aims to address the aforementioned issues and improve safety. Furthermore, it contributes to the development of a sustainable transportation system that further enhances traffic safety.

[0010] [Methods used to solve problems]

[0011] As one aspect of this disclosure, a gaze detection device is provided, comprising: an interpupillary distance detection unit that detects the interpupillary distance of the subject's left and right eyes based on an image including the subject's eyes; and a gaze detection unit that detects the subject's gaze position based on acquired information including the interpupillary distance, the gaze detection unit having a gaze change determination unit that determines, based on the acquired information, that if it detects multiple eye movements within a predetermined time period that result in a change of more than a predetermined amount in the interpupillary distance, the gaze change determination unit determines that the subject's gaze position has changed along with a change in visual distance.

[0012] As another aspect of this disclosure, a gaze detection method is provided, which is performed by a gaze detection device. The gaze detection method includes: an interpupillary distance detection step, which detects the interpupillary distance of the subject's left and right eyes based on an image containing the subject's eyes; and a gaze detection step, which detects the subject's gaze position based on acquired information including the interpupillary distance. The gaze detection step includes a gaze change determination process, in which, if eye movements that produce a change greater than a predetermined amount in the interpupillary distance are detected multiple times within a predetermined time period based on the acquired information, it is determined that the subject's gaze position has changed along with the change in visual distance.

[0013] As another aspect of this disclosure, a computer program product is provided, comprising a program that enables a computer-controlled gaze detection device to function as an interpupillary distance detection unit and a gaze detection unit. The interpupillary distance detection unit detects the interpupillary distance of the subject's left and right eyes based on an image including the subject's eyes. The gaze detection unit detects the subject's gaze position based on acquired information including the interpupillary distance. The gaze detection unit includes a gaze change determination process, in which, if eye movements that have caused a change of more than a predetermined amount in the interpupillary distance are detected multiple times within a predetermined time based on the acquired information, it is determined that the subject's gaze position has changed along with the change in visual distance.

[0014] [Invention Effects]

[0015] According to the present invention, changes in the position of the gaze can be quickly determined by utilizing specific eye movements. Attached Figure Description

[0016] Figure 1 This is a diagram showing the structure of the operation assistance system according to the first embodiment.

[0017] Figure 2 This is a diagram showing the structure near the driver's seat of a vehicle.

[0018] Figure 3 It is a diagram showing the relationship between viewing distance and interpupillary distance.

[0019] Figure 4 It is a diagram showing the change in interpupillary distance when the gaze position switches between far and near.

[0020] Figure 5 This is a diagram used to illustrate the notification processing of the notification control department.

[0021] Figure 6 This is a flowchart for determining and processing changes in line of sight.

[0022] Figure 7 This is a diagram showing the pattern 1, which illustrates the change in interpupillary distance.

[0023] Figure 8 This is a diagram showing the pattern 2, which illustrates the change in interpupillary distance.

[0024] Figure 9 This is a diagram showing the information image displayed through notification processing by the notification control unit.

[0025] Figure 10 This is a diagram used to illustrate the notification processing of the notification control department.

[0026] Figure 11 This is a diagram showing the structure of the operation assistance system according to the second embodiment.

[0027] Figure 12 This is a flowchart used to illustrate the notification processing of the notification control department. Detailed Implementation

[0028] [1. First Implementation Method]

[0029] [1.1 Structure of the Operation Assistance System]

[0030] Reference Figure 1 , Figure 2 The structure of the operation assistance system 1 of the first embodiment will be described.

[0031] like Figure 1 As shown, the operation assistance system 1 includes an operation assistance unit 2 having a processor 10 and a memory 20. The operation assistance unit 2 is mounted on the vehicle 100 and can also be referred to as an on-board device. The operation assistance unit 2 is an example of the gaze detection device of this disclosure.

[0032] The operation assistance unit 2 is connected to the communication unit 30, camera 31, radar 32, speed sensor 33, position sensor 34, driver monitoring camera 35, HUD (Head-Up Display) 36, display 37 and speaker 38 of the vehicle 100.

[0033] Vehicle 100 is a four-wheeled vehicle and is an example of a vehicle disclosed herein.

[0034] The communication unit 30 communicates with external communication systems such as the traffic information server 210 and the service provider server 211 via the communication network 200. The camera 31 captures images of the surrounding environment outside the vehicle 100, including the area in front of the vehicle, and outputs the captured images to the operation assistance unit 2. The radar 32 detects the positions of objects (vehicles in front, oncoming vehicles, etc.) around the vehicle body, including the area in front of the vehicle 100, and outputs the position detection data to the operation assistance unit 2. The speed sensor 33 detects the vehicle 100's speed and outputs the speed detection data to the operation assistance unit 2. The position sensor 34 uses known positioning techniques such as GNSS to detect the vehicle 100's current position and outputs the current position data to the operation assistance unit 2.

[0035] like Figure 2 As shown, a driver monitoring camera 35 is installed on the upper part of the windshield 101 of the vehicle 100, etc., to capture the driver D of the vehicle 100 and output the facial image of the driver D to the operation assistance unit 2. The driver D can also be referred to as a user or occupant, and is an example of the subject of this disclosure. The HUD 36 projects an information image 36a onto the windshield 101 for the driver D. The information image 36a is an image that is recognized by the driver D by overlapping with visually confirmed objects (road, oncoming vehicles, scenery, etc.) in front of and far from the vehicle 100.

[0036] exist Figure 2 In the example shown as information image 36a, the driving speed of vehicle 100 is displayed, but it is not limited to this; various notification information can be displayed. The notification information includes notifications related to driving operations, warnings, and notifications unrelated to driving operations.

[0037] In addition to the driving speed mentioned above, the notification information related to driving operation also includes traffic information obtained from the traffic information server 210 and route guidance information obtained from the navigation device equipped in the vehicle 100.

[0038] Warning messages include alerts to objects around the vehicle and error messages notifying the vehicle of abnormalities.

[0039] Notifications unrelated to driving operations include settings for the air conditioning and other systems used to improve in-vehicle comfort, as well as the operating status of the vehicle's audio system.

[0040] Information image 36a is an example of the information display section of this disclosure.

[0041] Display 37 is installed on the dashboard 102 of vehicle 100, displaying various information in a manner that can be visually confirmed by the driver D and passenger P. Display 37 is a touch panel, and the driver D inputs various instructions to the operation assistance unit 2 by touching the display 37. That is, display 37 also functions as an operating unit. For example, display 37 can display control screens for controlling the vehicle 100's air conditioning, audio system (e.g., radio, volume), and setting screens for adjusting the air conditioning, audio system, etc.

[0042] Speaker 38 plays various sounds into vehicle 100 under the control of operation assistance unit 2. Sounds such as those related to driving operations and audio equipment (e.g., radio) are output from speaker 38, which can be heard by driver D and passenger P.

[0043] Figure 2 The diagram also shows a steering wheel 103 positioned in front of the driver D and side mirrors 104 positioned to the left and right in front of the driver D. Additionally, instrument clusters for displaying the status of the vehicle 100 are sometimes provided on the instrument panel 102.

[0044] The processor 10 of the operation assistance unit 2 functions as a computer controlling the operation assistance unit 2. The processor 10 reads and executes the program 21 recorded in the memory 20, such as... Figure 1 As shown, it functions as an information acquisition unit 11, a face orientation detection unit 12, an interpupillary distance detection unit 13, a gaze detection unit 14, an ambient state detection unit 15, a danger level detection unit 16, and a notification control unit 17.

[0045] The information acquisition unit 11 includes an image acquisition unit 11a. The image acquisition unit 11a continuously acquires facial images of the driver D captured by the driver monitoring camera 35. The images acquired by the image acquisition unit 11a only need to include the driver D's eyes and face.

[0046] Furthermore, the information acquisition unit 11 is able to acquire information obtained by the communication unit 30, camera 31, radar 32, speed sensor 33, and position sensor 34. Figure 1 This illustrates the case where these data are recorded in memory 20 as acquired data 22.

[0047] The face orientation detection unit 12 continuously detects the orientation of the driver D's face (hereinafter referred to as "face orientation") from the face image of the driver D acquired by the image acquisition unit 11a. The processing used for detecting face orientation is not particularly limited; for example, by utilizing a known image recognition algorithm, face orientation can be detected with high accuracy. The face orientation information detected by the face orientation detection unit 12 is recorded in the memory 20 in a time sequence.

[0048] The interpupillary distance detection unit 13 continuously detects the distance between the left and right pupils of the driver D (hereinafter referred to as "interpupillary distance") from the image of the driver D acquired by the image acquisition unit 11a. As a process for detecting the interpupillary distance, the following image recognition processing is applied: first, the left and right eyes are detected from the image of the driver D; based on the brightness of the detected left and right eyes, the pupil portions (commonly known as "black pupils") of the left and right eyes are detected; and the distance between the center positions of the detected pupil portions of the left and right eyes is taken as the interpupillary distance.

[0049] According to this image recognition processing, interpupillary distance can be easily detected using existing image recognition techniques. Furthermore, the processing for detecting interpupillary distance is not limited to the image recognition processing described above. For example, the following image recognition processing can also be applied: detecting portions including both the pupil portion and the iris portion, and measuring the distance between the center positions of the detected left and right portions as the interpupillary distance. Moreover, the iris portion of this disclosure includes either the pupil portion or both the pupil portion and the iris portion.

[0050] The interpupillary distance detected by the interpupillary distance detection unit 13 is recorded in the memory 20 in a time sequence. Figure 1 The diagram shows the face orientation data and interpupillary distance data recorded in memory 20 as a face orientation / interpupillary distance database (hereinafter referred to as "DB") 23. Furthermore, face orientation and interpupillary distance are recorded so that their respective acquisition times can be determined.

[0051] The gaze detection unit 14 processes the driver D's gaze position based on the driver D's face orientation detected by the face orientation detection unit 12 and the driver D's interpupillary distance detected by the interpupillary distance detection unit 13. The gaze position can be described as the location of the gaze or the visually confirmed location.

[0052] Here, Figure 3 This is a diagram showing the relationship between visual distance and interpupillary distance. Visual distance is the distance at which the left and right eyes visually confirm an object; it can also be called focal length or the distance to the fixation point. Figure 3 In the attached figure, reference numeral 50 indicates the left and right eyeballs of the driver D, and reference numeral 53 indicates the left and right irises (commonly known as "black pupils", which include the pupils).

[0053] like Figure 3 As shown, when driver D visually confirms a distant visual object (in this embodiment, a road, a vehicle in front, an oncoming vehicle, scenery, etc.) through the windshield 101, the viewing distance becomes longer due to the distance of the line of sight, and the interpupillary distance of driver D becomes the value L1.

[0054] In contrast, such as Figure 3 As shown, when driver D visually confirms a nearby object (information image 36a in this embodiment) located on or around the windshield 101, the viewing distance is shorter due to the close proximity of the line of sight, and the interpupillary distance of driver D becomes value L2 (shorter than value L1). When the depth of such a gaze object changes, the pupils of both eyes move inward or outward, a movement known as convergence / divergence.

[0055] In the case of vehicle 100, such as Figure 3 As shown, the driver D visually confirms a distant object through the windshield 101 and visually confirms a nearby object (information image 36a), thus causing the visual confirmation position to switch between distant and near while driving.

[0056] Figure 4 This is a diagram showing the change in interpupillary distance when the driver D's line of sight switches between far and near. Figure 4 The horizontal axis t represents the passage of time, and the vertical axis Dp represents the interpupillary distance (hereinafter referred to as "interpupillary distance Dp"). Figure 4 The phrase "visually confirming a distant object" indicates that driver D is visually confirming the condition of a distant object through the windshield 101. Additionally, Figure 4 The phrase "visually confirming nearby" indicates that the driver D visually confirms the condition of the nearby visual object (information image 36a).

[0057] In this description, a distant visual object is, for example, a visual object several meters to about 100 meters in front of you, while a nearby visual object is, for example, a visual object about 60 centimeters to 1 meter in front of you. Furthermore, the range of "distant" and "near" is simply the range that will produce the next specific eye movement, and is not limited to the aforementioned distances.

[0058] The inventors, by observing the change in interpupillary distance Dp as the driver D's gaze shifts between near and far, noticed specific eye movements such as: Figure 4 As shown, the interpupillary distance Dp changes drastically multiple times at the moment of switching.

[0059] More specifically, such as Figure 4 As shown in region α, it was determined that when the gaze position changes from far to near, an eye movement occurs that produces a sudden decrease followed by a sudden increase in the interpupillary distance Dp. Additionally, as... Figure 4 As shown in region β, it was found that even when the gaze position changed from near to far, there was a sudden decrease followed by a sudden increase in interpupillary distance Dp. These eye movements can at least be presumed to be specific human eye responses.

[0060] By detecting the movement of that specific eye, it is possible to determine whether the driver D's gaze position has changed between near and far. Moreover, this movement can be identified as a state where a change of more than a certain amount in the interpupillary distance Dp occurs multiple times (more than twice) within a specified time period. Therefore, the computational load will not become enormous, and even without using a processor with high computing power, it is possible to quickly detect whether the above-mentioned movement has occurred.

[0061] Therefore, in the operation assistance system 1 of this embodiment, the gaze detection unit 14 includes a gaze change determination unit 14a, which determines whether the driver D's gaze position has changed along with a change in viewing distance by detecting whether the specific eye movement described above has occurred. The specific processing performed by the gaze change determination unit 14a will be explained later.

[0062] The surrounding state detection unit 15 searches for objects in front of the vehicle 100 based on the front image of the vehicle 100 captured by the camera 31 and the position detection data of objects in front of the vehicle 100 detected by the radar 32.

[0063] When the surrounding condition detection unit 15 detects an object, the danger level detection unit 16 calculates the predicted time until the vehicle 100 comes into contact with the object, i.e., the TTC (Time To Collision). Objects with a calculated TTC of less than a specified time are identified as objects with a danger level of more than a specified danger level and are thus identified as warning objects.

[0064] The level of danger is not limited to detection based on TTC; it can also be detected based on information other than TTC. The danger level detection unit 16 is an example of the "danger level detection unit that detects the level of danger based on the relationship between the vehicle and the surrounding environment" disclosed herein.

[0065] The notification control unit 17 performs notification processing to the driver D based on at least one of the determination results from the line-of-sight change determination unit 14a and the detection results from the danger level detection unit 16. The notification processing includes displaying notification information related to driving operations using a HUD 36, displaying warning information, and playing sounds equivalent to notification information and warning information using a speaker 38.

[0066] Figure 5 This diagram illustrates an example of notification processing in the notification control unit 17. (See diagram for example.) Figure 5 As shown, in order to notify the driver D, the notification control unit 17 projects the outer frame 93 in the direction from the driver D toward the warning object 250 (in this case, the vehicle in front) via the HUD 36. The outer frame 93 can prompt attention to the warning object 250. In addition, the notification control unit 17 can also output a sound prompting attention ("Please note the vehicle ahead", etc.) from the speaker 38.

[0067] [1.2 Handling of line-of-sight changes]

[0068] according to Figure 6 The flowchart shown illustrates the line-of-sight change determination process performed by the line-of-sight change determination unit 14a. The line-of-sight change determination unit 14a continues to perform this process when the vehicle 100 is in an operating state (power-on state). Figure 6 The flowchart shown illustrates the processing.

[0069] First, the gaze change determination unit 14a obtains the interpupillary distance Dp of a continuous predetermined number of frames (a predetermined amount of time) detected by the interpupillary distance detection unit 13 (step S1).

[0070] Here, Figure 7 as well as Figure 8 The diagrams show examples of eye movements where the interpupillary distance Dp decreases and then increases sharply due to changes in the driver D's line of sight between near and far. Figure 7 This is a diagram example of how the line of sight changes when the viewpoint moves from far away to near (Change Pattern 1). Figure 8 This is an example diagram showing the change in line-of-sight as the viewpoint shifts from near to far (Change Pattern 2). In this description, the change is based on the result obtained in step S1. Figure 7 , Figure 8 The following explanation will be based on the interpupillary distance Dp of the five frames f1 to f5.

[0071] Next, the gaze change determination unit 14a determines whether the interpupillary distance Dp has decreased (step S2). If the interpupillary distance Dp has decreased, the gaze change determination unit 14a proceeds to step S3; otherwise, it returns to step S1. If it returns to step S1, it obtains the interpupillary distance Dp for a predetermined number of consecutive frames after shifting the frame number by a predetermined number, and then executes the processing after step S2 again.

[0072] In step S3, the gaze change determination unit 14a determines the decrease in interpupillary distance Dp, De1 (refer to...). Figure 7 , Figure 8 Whether it exceeds the first threshold T1 used for sudden decrease determination.

[0073] The first threshold T1 is sufficient to determine... Figure 4 The threshold for the sudden decrease in interpupillary distance Dp generated in the illustrated regions α and β is, more specifically, set to be able to determine Figure 7 The threshold for the sudden decrease in interpupillary distance Dp in the variation pattern 1 shown.

[0074] If the reduction amount De1 exceeds the first threshold T1 (step S3; yes), the gaze change determination unit 14a proceeds to the processing in step S4 if the interpupillary distance Dp increases after the decrease.

[0075] In step S4, the gaze change determination unit 14a determines the increase in interpupillary distance Dp after the decrease, De2 (refer to...). Figure 7 Whether it exceeds the second threshold T2 used for sudden increase determination.

[0076] The second threshold T2 is able to determine that... Figure 4 The threshold for the sudden increase after a sudden decrease in the interpupillary distance Dp in regions α and β, as shown, is more specifically set to be able to determine... Figure 7 The threshold for the sudden increase after a sudden decrease in the interpupillary distance Dp in the variation pattern 1 shown.

[0077] If the increase De2 exceeds the second threshold T2 used for sudden increase determination (step S4; Yes), it can be determined that an increase equivalent to... Figure 4 The phenomenon shown is that the interpupillary distance Dp in regions α and β suddenly decreases and then suddenly increases. Therefore, it can be determined that the driver D's line of sight position has changed between near and far; in other words, it can be determined that the driver D's line of sight position has changed along with the change in visual distance. In this case, the process of step S5 is executed.

[0078] In contrast, if the increase De2 does not exceed the second threshold T2 used for sudden increase determination (step S4; no), the line of sight change determination unit 14a returns to the processing of step S1.

[0079] In step S5, the gaze change determination unit 14a determines whether the interpupillary distance Dp is shorter than before the change (before the sudden decrease). If the interpupillary distance Dp is shorter than before the change (step S5; yes), it is known that the driver D's gaze position has changed to a closer position (equivalent to change pattern 1), therefore the gaze change determination unit 14a determines that the driver D's gaze position has changed from far to near (step S6). If the interpupillary distance Dp is not shorter than before the change (step S5; no), the gaze change determination unit 14a returns to the processing of step S1.

[0080] On the other hand, in step S3, if the reduction amount De1 does not exceed the first threshold T1 (step S3; no), the gaze change determination unit 14a determines the reduction amount De1 of the interpupillary distance Dp (refer to...). Figure 7 , Figure 8 Whether it exceeds the first threshold T1' used for sudden decrease determination (step S7).

[0081] The first threshold T1' is able to determine that... Figure 4 The threshold for the sudden decrease in interpupillary distance Dp generated in the illustrated regions α and β is, more specifically, set to be able to determine Figure 8 The threshold for the sudden decrease in interpupillary distance Dp in the variation pattern 2 shown.

[0082] According to the inventors' research, the decrease in interpupillary distance Dp of variation pattern 2, De1, tends to be less than the decrease in interpupillary distance Dp, De1, of variation pattern 1. Therefore, it is preferable to set the first threshold T1' to a value smaller than the first threshold T1.

[0083] If the reduction amount De1 exceeds the first threshold T1' (step S7; yes), the gaze change determination unit 14a proceeds to the processing in step S8 if the interpupillary distance Dp increases after the decrease.

[0084] In contrast, if the reduction amount De1 does not exceed the first threshold T1' (step S7; no), the line-of-sight change determination unit 14a returns to the processing of step S1.

[0085] In step S8, the gaze change determination unit 14a determines the increase in interpupillary distance Dp after the decrease, De2 (refer to...). Figure 8 Whether it exceeds the second threshold T2' used for sudden increase determination.

[0086] The second threshold T2' is able to determine that... Figure 4 The threshold for the sudden increase after a sudden decrease in the interpupillary distance Dp in regions α and β, as shown, is more specifically set to be able to determine... Figure 8The threshold for the sudden increase after a sudden decrease in the interpupillary distance Dp in the variation pattern 2 shown.

[0087] According to the inventors' research, there is a tendency for the increase in interpupillary distance Dp, De2, of variation pattern 2 to be greater than the increase in interpupillary distance Dp, De2, of variation pattern 1. Therefore, it is preferable to set the second threshold T2' to a value larger than the second threshold T2.

[0088] If the increase De2 exceeds the second threshold T2' used for sudden increase determination (step S8; Yes), it can be determined that an increase equivalent to... Figure 4 The phenomenon shown is that the interpupillary distance Dp in regions α and β suddenly decreases and then suddenly increases. In this case, the process in step S9 is performed.

[0089] In contrast, if the increase De2 does not exceed the second threshold T2' used for sudden increase determination (step S8; no), the line of sight change determination unit 14a returns to the processing of step S1.

[0090] In step S9, the gaze change determination unit 14a determines whether the interpupillary distance Dp is longer than before the change (before the sudden decrease). If the interpupillary distance Dp is longer than before the change (step S9; yes), it is known that the driver D's gaze position has changed to a farther position (equivalent to change pattern 2), therefore the gaze change determination unit 14a determines that the driver D's gaze position has changed from near to far (step S10). If the interpupillary distance Dp is not longer than before the change (step S9; no), the gaze change determination unit 14a returns to the processing of step S1.

[0091] However, the line-of-sight change determination unit 14a can also use the above determination result based on the interpupillary distance Dp to determine the line-of-sight position of the driver D in a more specific location.

[0092] For example, the line-of-sight change determination unit 14a may determine that the driver D's line of sight has changed from far away to near when it is determined that the driver D is looking at the windshield 101 because the driver D is likely to change from looking at and confirming a distant object through the windshield 101 to looking at the windshield 101 (e.g., information image 36a).

[0093] In addition, the gaze change determination unit 14a can also use the detection results of the face orientation detection unit 12 in addition to the above-mentioned determination results based on the interpupillary distance Dp, so as to determine the gaze position of the driver D in a more specific position.

[0094] For example, if the gaze change determination unit 14a determines that the driver D's gaze position changes from far to near and the face orientation detection unit 12 detects that the driver D's face is facing either left or right, it is determined that the driver D is looking at one side of the side mirror 104. Conversely, if the gaze change determination unit 14a determines that the driver D's gaze position changes from far to near and the face orientation detection unit 12 detects that the driver D's face is facing the center in the vehicle width direction, it is determined that the driver D is looking at the display 37, which functions as a central display unit and is located in the center in the vehicle width direction. Therefore, the driver D's gaze position can be determined at a more specific location.

[0095] [1.3 Notification Processing]

[0096] Reference Figure 9 , Figure 10 The notification processing performed by the notification control department 17 is explained. For example... Figure 9 As shown, when the line-of-sight change determination unit 14a determines that the driver D's line-of-sight position has changed from far to near (step S6), it notifies the control unit 17 to proceed with... Figure 9 The information image 36a shown is processed as notification information related to driving operation and projected onto the windshield 101.

[0097] exist Figure 9 In the information image 36a, image G1 represents the object in front extracted by the surrounding state detection unit 15, and image G2 represents the marker identified by the surrounding state detection unit 15 from the image of camera 31. Additionally, image G3 represents the vehicle 100's speed, and image G4 represents route guidance information obtained from the navigation device equipped on the vehicle 100.

[0098] For example, a portion of images G1 to G4 (e.g., image G3 composed of driving speed) may always be displayed, while the remaining images may be displayed when the gaze change determination unit 14a determines that the driver D's gaze position has changed from far to near (including when it is determined that the driver D is looking at the windshield 101 (e.g., information image 36a)). Furthermore, the content of information image 36a, i.e., notification information related to driving operations, may also be appropriately changed.

[0099] like Figure 10 As shown, if the line-of-sight change determination unit 14a determines that the driver D's line of sight has changed from a distance to a closer location, and the danger level detection unit 16 identifies a warning object 250 with a danger level exceeding a prescribed danger level, it is also possible to display [the warning object]. Figure 9 The illustrated warning messages differ from the notification messages related to driving operations. Figure 10 The outer frame shown is 93).

[0100] If, after displaying the warning message (outer frame 93), the gaze change determination unit 14a determines that the driver D's gaze position has changed from near to far (step S10), the display of the warning message (outer frame 93) can be stopped. Conversely, if, after displaying the warning message (outer frame 93), the gaze change determination unit 14a determines that the driver D's gaze position has not changed (step S11), the warning intensity can be increased. As an example of increasing the warning intensity, the intensity can be increased by displaying an information image 36a, or by emitting a warning sound from the speaker 38.

[0101] Furthermore, the gaze change determination unit 14a can determine that the driver D's gaze position has not changed, for example, if the interpupillary distance Dp does not change. The warning message is not limited to the outer frame 93 and can be modified appropriately.

[0102] Thus, the operation assistance unit 2 includes: an interpupillary distance detection unit 13, which detects the interpupillary distance Dp of the driver D's left and right eyes based on an image including the driver D's eyes as the subject; and a gaze detection unit 14, which detects the driver D's gaze position based on acquired information including the interpupillary distance Dp. The gaze detection unit 14 includes a gaze change determination unit 14a, which determines that the driver D's gaze position has changed along with the change in visual distance if, based on the acquired information, two eye movements have occurred within a specified time period due to a change in the interpupillary distance Dp of a predetermined amount.

[0103] Therefore, by utilizing at least specific human eye movements, changes in driver D's line of sight can be quickly determined. This allows for real-time monitoring of driver D's line of sight and immediate response to changes in driver attention and driving conditions, thus contributing to improved safety. Consequently, this can further enhance traffic safety and contribute to the development of sustainable transportation systems.

[0104] Furthermore, in this embodiment, an example is shown where the gaze detection unit 14 detects two eye movements occurring within a specified time that result in a change of more than a predetermined amount in the interpupillary distance Dp. However, it can also be configured to detect more than two eye movements that result in the aforementioned change. In this case, the change in the driver D's gaze position can also be quickly determined using the specific eye movements described above.

[0105] In this embodiment, the processing of the interpupillary distance detection unit 13 is an example of the interpupillary distance detection step of this disclosure. Similarly, the processing of the gaze detection unit 14 is an example of the gaze detection step of this disclosure. Furthermore, the processing of the gaze change determination unit 14a is an example of the gaze change determination step of this disclosure. Additionally, the processing of the ambient situation detection unit 15 is an example of the ambient situation detection step of this disclosure, the processing of the danger level detection unit 16 is an example of the danger level detection step, and the processing of the notification control unit 17 is an example of the notification step of this disclosure.

[0106] [2. Second Implementation]

[0107] Reference Figure 11 The structure of the operation assistance system 1 in the second embodiment will be described.

[0108] The second embodiment differs from the first embodiment in that the operation assistance system 1 includes a microphone 39, a sound acquisition unit 11b, and a voice recognition unit 18, and the notification control unit 17 uses these structures to perform notification processing related to tasks performed by the driver D, etc. Other than this, the structure is the same as the first embodiment; therefore, the differences will be described. Furthermore, in Figure 11 In this document, structures identical to those in the first embodiment are indicated by the same reference numerals.

[0109] Microphone 39 is connected to the operation assistance unit 2, collects sound inside the vehicle 100, and outputs the sound to the operation assistance unit 2. Microphone 39 is positioned around the driver D and can collect the sound of the driver D and passenger P.

[0110] The sound acquisition unit 11b is part of the information acquisition unit 11 and acquires the sound collected via the microphone 39.

[0111] The voice recognition unit 18 recognizes the voice acquired by the voice acquisition unit 11b and generates text data that transcribes the voice data based on the recognition result. Furthermore, the voice recognition unit 18 has the function of detecting predefined tasks and instructions based on the acquired text data. More specifically, the voice recognition unit 18 can determine the corresponding task from the acquired text data by referring to voice instructions and keywords used to determine the task to be performed by the driver D.

[0112] This task includes operations performed by the driver D via display 37, such as turning the air conditioner on and off, adjusting the air conditioner, and operating the audio system (radio, volume, etc.).

[0113] Thus, for example, if the conversation between driver D and passenger P includes sounds associated with display 37 such as "turn on the air conditioning" or "play the radio", the task to be performed via display 37 can be detected in advance based on the conversation.

[0114] [2.1 Notification Processing]

[0115] In addition to the notification processing using HUD 36 and speaker 38 described in the first embodiment, the notification control unit 17 also uses display 37 to perform other notification processing that helps improve the convenience of driver D.

[0116] according to Figure 12 The flowchart shown illustrates the notification processing performed by the notification control unit 17 and related processing. Furthermore, Figure 12 The flowchart shown is executed continuously when the vehicle 100 is in a working state (power-on state) or in a working state (power-on state) and in motion.

[0117] like Figure 12 As shown, the voice recognition unit 18 performs voice recognition processing to identify the voice of the driver D and others inside the vehicle (step S1a). Based on the voice recognition result, the voice recognition unit 18 detects the task performed by the driver D (step S2a).

[0118] Next, the control unit 17 is notified to determine whether the task detected by the voice recognition unit 18 is a task performed via the display 37 (touch operation) (step S3a). If the task is performed via the display 37 (touch operation) (step S3a: Yes), the control unit 17 is notified to execute the processing of step S4a; if the task is not performed via the display 37 (step S3a: No), the process returns to step S1a.

[0119] In step S4a, the control unit 17 is notified to determine whether the line-of-sight change determination unit 14a has determined that the driver D's line of sight has changed from a distance to a closer distance. Here, if the line-of-sight change determination unit 14a does not determine that the driver D's line of sight has changed from a distance to a closer distance (step S4a: No), the control unit 17 is notified to repeat the determination process of step S4a until a preset standby time has elapsed (step S5a: Yes).

[0120] That is, in order for driver D to perform the task detected in step S2a, driver D needs to bring his line of sight close (the position of display 37), so he stands still until the opportunity for driver D to bring his line of sight close arrives.

[0121] If the line-of-sight change determination unit 14a determines that the driver D's line of sight has changed from far to near (step S4a: yes), then the control unit 17 is notified to display information related to the task on the display 37 (step S6a).

[0122] For example, when the task involves air conditioning operation, the system displays the air conditioning operation screen on display 37 or emphasizes the air conditioning operation buttons as task-related information. Similarly, when the task involves radio, volume, or other similar operations, the system displays the radio, volume, or other similar operation screens on display 37 or emphasizes the radio, volume, or other similar operation buttons as task-related information. This improves the convenience for driver D.

[0123] [3. Other Implementation Methods]

[0124] The above-described embodiments are merely one embodiment of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

[0125] For example, in the above embodiment, when the line-of-sight change determination unit 14a determines that the driver D's line of sight has changed from far to near, it is determined that the driver D is looking at the windshield 101 (e.g., information image 36a). However, it is also possible to determine that the driver D is looking at the instrument panel 102. This is because, for example, if the information image 36a is not projected onto the windshield 101, the driver D is more likely to be looking at the instrument cluster or display 37 (central display) provided on the instrument panel 102.

[0126] In addition, regarding the notification information displayed on the information image 36a (including notification information related to driving operation, warning information, and notification information unrelated to driving operation), in the absence of HUD 36, it can also be displayed on the information display unit of the instrument panel 102, such as the display 37 or measuring instruments.

[0127] Furthermore, while the invention has been described as being applied to an operation assistance unit 2 mounted on a vehicle 100 and functioning as a line-of-sight detection device, it can also be applied to a line-of-sight detection device mounted on any moving body operated by a driver D. The moving body could be an airplane, motorcycle, or ship, etc.

[0128] in addition, Figure 1 , Figure 11The structures shown can be widely applied to software-implemented structures, hardware-implemented structures, and structures combining software and hardware. Furthermore, part or all of the structure of the gaze detection device of the present invention can be configured as a structure possessed by an external system of the vehicle 100, such as a server on the communication network 200. In this case, the device is configured to send information obtained by the vehicle 100 to the external system, whereby the external system performs at least a portion of the processing performed by the gaze detection device, and sends output data based on the processing results to the vehicle 100, thereby implementing various notifications and warnings within the vehicle 100.

[0129] In addition, the method of dividing the processing units and the order of processing in the flowchart are not limited to the example shown in the figure, and can be changed appropriately.

[0130] Furthermore, while the case where the program 21 implementing the gaze detection method of the present invention is recorded on the gaze detection device has been described, the program 21 can also be obtained from an external device via communication. Alternatively, the program 21 can be recorded on a computer-readable recording medium. Magnetic or optical recording media, semiconductor memory devices, etc., can be used as the recording medium.

[0131] [4. Structure supported by the above embodiments]

[0132] The above implementation is a specific example of the following structure.

[0133] (Structure 1) A gaze detection device, comprising: an interpupillary distance detection unit that detects the interpupillary distance of the left and right eyes of the subject based on an image including the eyes of the subject; and a gaze detection unit that detects the gaze position of the subject based on acquired information including the interpupillary distance, the gaze detection unit having a gaze change determination unit that determines that the gaze position of the subject has changed along with the change in viewing distance when it detects multiple eye movements that have caused a change of more than a predetermined amount in the interpupillary distance within a predetermined time period based on the acquired information.

[0134] Based on this structure, it is possible to quickly determine changes in the gaze position of an object by utilizing at least specific eye movements, which can contribute to improved security.

[0135] (Structure 2) According to the gaze detection device of Structure 1, when the gaze change determination unit detects that the change in the interpupillary distance has decreased by more than a predetermined threshold, and the change in the interpupillary distance has increased by more than a predetermined threshold and the interpupillary distance has become shorter than before the change, it determines that the gaze position of the subject has changed from far to near.

[0136] Based on this structure, it is possible to quickly determine that the viewer's line of sight has changed from far away to near.

[0137] (Structure 3) According to the gaze detection device of Structure 1 or 2, if the gaze change determination unit detects that the change in the interpupillary distance decreases by more than a predetermined threshold, and the change in the interpupillary distance increases by more than a predetermined threshold and the interpupillary distance becomes longer than before the change, it determines that the gaze position of the subject has changed from near to far.

[0138] Based on this structure, it is possible to quickly determine that the viewer's line of sight has changed from near to far.

[0139] (Structure 4) A gaze detection device according to any one of Structures 1 to 3, wherein the gaze detection device is mounted on a vehicle and the object is an occupant of the vehicle.

[0140] Based on this structure, changes in the line of sight of vehicle occupants can be quickly determined.

[0141] (Structure 5) According to the gaze detection device of Structure 3, wherein the gaze detection device is mounted on a vehicle, the subject is the driver of the vehicle, and the gaze change determination unit determines that the driver is looking at at least one of the windshield and the dashboard of the vehicle when it determines that the driver's gaze position has changed from far to near.

[0142] Based on this structure, it is possible to quickly determine whether the driver's line of sight has changed from the windshield to the dashboard.

[0143] (Structure 6) According to the gaze detection device of Structure 2, wherein the gaze detection device is mounted on a vehicle, the subject is the driver of the vehicle, and the gaze change determination unit determines that the driver's gaze position has changed from far to near and detects a change in the driver's face orientation when it determines that the driver is looking at the central display or side mirror located in the center of the vehicle width direction.

[0144] This structure allows for the rapid determination of whether the driver's gaze has shifted to the central display or the side mirrors. Furthermore, it can identify whether the driver is looking at the central display or the side mirrors based on the direction of their face.

[0145] (Structure 7) According to the line-of-sight detection device of Structure 5, the line-of-sight detection device includes a notification control unit, which displays notification information related to driving operation on the information display unit provided on the windshield or the instrument panel when the line-of-sight change determination unit determines that the line-of-sight position has changed from far to near.

[0146] According to this structure, by displaying notification information related to driving operations in conjunction with the driver's line of sight, the driver can confirm the necessary information at the appropriate time.

[0147] (Structure 8) According to the line-of-sight detection device of Structure 7, the line-of-sight detection device includes: an ambient state detection unit that detects the ambient state of the vehicle; and a danger level detection unit that detects the danger level based on the relationship between the vehicle and the ambient state, and displays a warning message different from the notification information related to the driving operation on the information display unit when the line-of-sight change determination unit determines that the line-of-sight position has changed from far to near and the danger level is above a predetermined danger level.

[0148] According to this structure, if a highly dangerous situation occurs outside the vehicle while the driver is observing nearby, a warning message can be quickly sent to the driver, making it easier to avoid danger.

[0149] (Structure 9) According to the gaze detection device of Structure 8, when the gaze change determination unit detects that the change in the interpupillary distance decreases by a predetermined threshold or more, and the change in the interpupillary distance increases by a predetermined threshold or more and the interpupillary distance becomes longer than before the change, it determines that the gaze position of the subject has changed from near to far. After displaying the warning information, the notification control unit stops displaying the warning information if the gaze change determination unit determines that the gaze position has changed from near to far. If it determines that the gaze position has not changed, it increases the level of warning based on the warning information.

[0150] Based on this structure, it is possible to coordinate with the driver's line of sight after the warning information is displayed, and to appropriately suspend or increase the level of the warning, thus easily avoiding danger.

[0151] (Structure 10) A gaze detection device according to any one of Structures 7 to 9, wherein the gaze detection device has a sound recognition unit for recognizing sounds inside the vehicle, the notification control unit has a function for detecting a task performed by the driver based on the sound recognition result, and when the task is detected and the gaze change determination unit determines that the gaze position has changed from far to near, information associated with the task is displayed on the information display unit.

[0152] According to this structure, when the driver is performing a task, task-related information can be displayed at the appropriate location and time, improving convenience.

[0153] (Structure 11) A gaze detection device according to any one of Structures 1 to 10, wherein the interpupillary distance detection unit detects the left and right eyes from the image, detects the black part of the left and right eyes based on the brightness of the detected left and right eyes, and detects the distance between the center positions of the detected black parts of the left and right eyes as the interpupillary distance.

[0154] Based on this structure, interpupillary distance can be easily detected using existing image recognition technologies.

[0155] (Structure 12) According to the gaze detection device of Structure 3, the gaze change determination unit determines whether the gaze position of the subject has changed from near to far by utilizing the fact that the change in interpupillary distance when the gaze position changes from near to far is less than the change in interpupillary distance when the gaze position changes from far to near.

[0156] Based on this structure, by utilizing the fact that the change in interpupillary distance when the gaze position changes from near to far (variation pattern 2) is less than the change in interpupillary distance when the subject's gaze position changes from far to near (variation pattern 1), it is possible to determine with high accuracy whether the subject's gaze position has changed from near to far. Thus, the accuracy of determining the distance of the gaze position is improved.

[0157] (Structure 13) A gaze detection method, performed by a gaze detection device, comprising: an interpupillary distance detection step, wherein the interpupillary distance of the left and right eyes of the subject is detected based on an image containing the eyes of the subject; and a gaze detection step, wherein the gaze position of the subject is detected based on acquired information including the interpupillary distance, the gaze detection step comprising a gaze change determination process, wherein in the gaze change determination process, if eye movements that produce a change of more than a predetermined amount of the interpupillary distance are detected multiple times within a predetermined time based on the acquired information, it is determined that the gaze position of the subject has changed along with the change in visual distance.

[0158] According to this method, changes in the gaze position of an individual can be quickly determined by utilizing at least specific eye movements, which can contribute to improved security.

[0159] (Structure 14) A program that enables a computer-controlled gaze detection device to function as an interpupillary distance detection unit and a gaze detection unit, wherein the interpupillary distance detection unit detects the interpupillary distance of the subject's left and right eyes based on an image containing the subject's eyes, and the gaze detection unit detects the subject's gaze position based on acquired information including the interpupillary distance, wherein the gaze detection unit includes a gaze change determination process, wherein in the gaze change determination process, if eye movements that have caused a change of more than a predetermined amount of the interpupillary distance are detected multiple times within a predetermined time based on the acquired information, it is determined that the subject's gaze position has changed along with the change in visual distance.

[0160] According to this procedure, changes in the gaze position of an individual can be quickly determined by utilizing specific eye movements, which can contribute to improved security.

[0161] Explanation of reference numerals in the attached figures

[0162] 1…Operator assistance system, 2…Operator assistance unit (eye contact detection device), 10…Processor, 11…Information acquisition unit, 12…Face orientation detection unit, 13…Interpupillary distance detection unit, 14…Eye contact detection unit, 14a…Eye contact change determination unit, 15…Surroundings detection unit, 16…Danger level detection unit, 17…Notification control unit, 18…Voice recognition unit, 20…Memory, 21…Program, 30…Communication unit, 31…Camera, 32…Radar, 33…Speed ​​sensor, 34…Position sensor, 35…Driver monitoring camera, 36…HUD, 36a…Information image (information display unit), 37…Display (information display unit, central display unit), 38…Speaker, 39…Microphone, 100…Vehicle, 101…Windshield, 102…Instrument panel, 104…Side mirror, D…Driver (object), P…Passenger, Dp…Interpupillary distance.

Claims

1. A gaze detection device, comprising: an interpupillary distance detection section that detects an interpupillary distance of a subject's left and right eyes based on an image including the subject's eyes; and a gaze detection section that detects a gaze position of the subject based on acquired information including the interpupillary distance, the gaze detection section having a gaze change determination section that determines that the gaze position of the subject has changed along with a change in a viewing distance in a case where an eye movement that generates a change in the interpupillary distance by a prescribed amount or more occurs a plurality of times within a prescribed time based on the acquired information.

2. The gaze detection device according to claim 1, wherein the gaze change determination section determines that the gaze position of the subject has changed from a far distance to a near distance in a case where the amount of change in the interpupillary distance increases by a prescribed threshold or more after the amount of change in the interpupillary distance decreases by the prescribed threshold or more and the interpupillary distance becomes shorter than before the change.

3. The gaze detection device according to claim 1, wherein the gaze change determination section determines that the gaze position of the subject has changed from a near distance to a far distance in a case where the amount of change in the interpupillary distance increases by a prescribed threshold or more after the amount of change in the interpupillary distance decreases by the prescribed threshold or more and the interpupillary distance becomes longer than before the change.

4. The gaze detection device according to claim 1, wherein the gaze detection device is mounted on a vehicle, the subject is an occupant of the vehicle.

5. The gaze detection device according to claim 2, wherein the gaze detection device is mounted on a vehicle, the subject is a driver of the vehicle, the gaze change determination section determines that the driver is visually recognizing at least either one of a windshield and an instrument panel of the vehicle in a case where it is determined that the gaze position of the driver has changed from a far distance to a near distance.

6. The gaze detection device according to claim 2, wherein the gaze detection device is mounted on a vehicle, the subject is a driver of the vehicle, the gaze change determination section determines that a center display or a side mirror provided at a center in a vehicle width direction of the vehicle is being visually recognized in a case where it is determined that the gaze position of the driver of the vehicle has changed from a far distance to a near distance and a change in the driver's face orientation is detected.

7. The gaze detection device according to claim 5, wherein the gaze detection device comprises a notification control section that displays notification information related to a driving operation at an information display section provided at the windshield or the instrument panel in a case where it is determined by the gaze change determination section that the gaze position has changed from a far distance to a near distance.

8. The gaze detection device according to claim 7, wherein the gaze detection device comprises: a surrounding state detection section that detects a state of a surrounding of the vehicle; and a danger degree detection section that detects a danger degree according to a relationship between the vehicle and the state of the surrounding, the notification control section displays notification information related to a danger degree in a case where it is determined by the danger degree detection section that the danger degree is high. In a case where it is determined by the gaze change determination portion that the gaze position has changed from a distance to a proximity and the degree of danger is a prescribed degree of danger or more, warning information different from the notification information related to the driving operation is displayed on the information display portion.

9. The gaze detection device according to claim 8, wherein the gaze change determination portion determines that the gaze position of the subject has changed from a proximity to a distance in a case where, after detecting that the amount of change in the interpupillary distance has decreased by a prescribed threshold or more, the amount of change in the interpupillary distance has increased by a prescribed threshold or more and the interpupillary distance has become longer than before the change, the notification control portion, after displaying the warning information, stops the display of the warning information in a case where it is determined by the gaze change determination portion that the gaze position has changed from a proximity to a distance, and performs processing to increase the degree of warning based on the warning information in a case where it is determined that the gaze position has not changed.

10. The gaze detection device according to claim 7, wherein the gaze detection device has a sound recognition portion that recognizes a sound within the vehicle, the notification control portion has a function of detecting a task performed by the driver based on a recognition result of the sound, and displays information associated with the task on the information display portion in a case where the task is detected and it is determined by the gaze change determination portion that the gaze position has changed from a distance to a proximity.

11. The gaze detection device according to claim 1, wherein the interpupillary distance detection portion detects the left and right eyes from the image, detects the dark iris portions of the detected left and right eyes based on the luminance of the detected left and right eyes, and detects the distance between the center positions of the detected dark iris portions of the left and right eyes as the interpupillary distance.

12. The gaze detection device according to claim 3, wherein the gaze change determination portion determines whether or not the gaze position of the subject has changed from a proximity to a distance using the fact that the amount of change in the interpupillary distance when the gaze position has changed from a proximity to a distance is smaller than the amount of change in the interpupillary distance when the gaze position has changed from a distance to a proximity.

13. A gaze detection method performed by a gaze detection device, the gaze detection method comprising: an interpupillary distance detection step of detecting an interpupillary distance of left and right eyes of a subject based on an image containing the eyes of the subject; and a gaze detection step of detecting a gaze position of the subject based on acquired information containing the interpupillary distance, the gaze detection step including a gaze change determination process in which, in a case where it is detected based on the acquired information that an eye movement in which a change in the interpupillary distance of a prescribed amount or more occurs a plurality of times within a prescribed time has occurred, it is determined that the gaze position of the subject has changed in association with a change in the distance.

14. A computer program product comprising a program that causes a line-of-sight detection device controlled by a computer to function as an interpupillary distance detection section that detects an interpupillary distance of left and right eyes of a subject based on an image including the eyes of the subject, and a line-of-sight detection section that detects a line-of-sight position of the subject based on acquired information including the interpupillary distance, wherein the line-of-sight detection section includes a line-of-sight change determination process in which, in a case where an eye movement in which a change in the interpupillary distance of a prescribed amount or more occurs a plurality of times within a prescribed time is detected based on the acquired information, it is determined that the line-of-sight position of the subject has changed in association with a change in a viewing distance. ​

Images