Eye-tracking device, eye-tracking method, and program

Abstract

To enable rapid identification of changes in gaze position by utilizing specific eye movements. [Solution] The operation support unit 2, which functions as a gaze detection device, includes an interpupillary distance detection unit 13 that detects the interpupillary distance Dp of the target, the driver D, and a gaze detection unit 14 that detects the gaze position of the driver D. The gaze detection unit 14 has a gaze change determination unit 14a that, when it detects that eye movements have occurred that cause a change in the interpupillary distance Dp of a predetermined amount or more twice within a predetermined time, determines that the gaze position of the driver D has changed along with a change in viewing distance.

Description

Technical Field

[0001] The present invention relates to a gaze detection device, a gaze detection method, and a program.

Background Art

[0002] In recent years, research and development have been conducted to obtain data related to the interpupillary distance of users and provide support. For example, Patent Document 1 discloses a technique for determining a gaze direction and an eye movement state based on the positions of a first pupil coordinate, a second pupil coordinate, a first inner canthus coordinate, and a second inner canthus coordinate. Further, the eye movement state includes both-eye movement states such as upward and downward gaze movement, left and right gaze movement, convergence movement, or divergence movement, and an oblique position state (that is, eye misalignment) in which one of the fluctuations in the pupil position is greater than or equal to a threshold value and the other is less than the threshold value is described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, the conventional technology simply detects the gaze position based on the left and right pupil positions, and does not delve into gaze detection considering specific eye movements in a living body such as a human. Therefore, depending on the arithmetic processing ability of the processor used for detecting the gaze position, there is a possibility that a situation where a change in the gaze position cannot be quickly identified may occur. Therefore, it is an object of the present application to make it possible to quickly identify a change in the gaze position by using a specific eye movement. The present application is for solving the above problems and aims to improve safety. And, by extension, it contributes to the development of a sustainable transportation system by further improving traffic safety. [Means for solving the problem]

[0005] One aspect of the present disclosure is a gaze detection device comprising: a pupillary distance detection unit that detects the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes; and a gaze detection unit that detects the subject's gaze position based on acquired information including the pupillary distance, wherein the gaze detection unit has a gaze change determination unit that, based on the acquired information, detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time, and determines that the subject's gaze position has changed along with a change in viewing distance.

[0006] Another aspect of the present disclosure is a gaze detection method performed by a gaze detection device, comprising: an interpupillary distance detection step of detecting the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes; and a gaze detection step of detecting the subject's gaze position based on acquired information including the interpupillary distance, wherein the gaze detection step includes a gaze direction change determination process that determines that the subject's gaze position has changed with a change in viewing distance when it detects, based on the acquired information, that eye movements have occurred in which changes in the interpupillary distance exceeding a predetermined amount occur multiple times within a predetermined time.

[0007] Another aspect of the present disclosure is a program that causes a computer-controlled gaze detection device to function as a pupillary distance detection unit that detects the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes, and a gaze detection unit that detects the subject's gaze direction based on acquired information including the pupillary distance, wherein the gaze detection unit includes a gaze direction change determination process that determines that the subject's gaze position has changed accompanied by a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time. [Effects of the Invention]

[0008] According to the present invention, changes in gaze position can be quickly identified by utilizing specific eye movements. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows the configuration of the operation support system according to the first embodiment. [Figure 2] Figure 2 shows the configuration of the vehicle near the driver's seat. [Figure 3] Figure 3 shows the relationship between viewing distance and interpupillary distance. [Figure 4] Figure 4 shows the change in interpupillary distance when the line of sight switches between far and near. [Figure 5] Figure 5 is a diagram illustrating the notification processing of the notification control unit. [Figure 6] Figure 6 is a flowchart of the gaze change detection process. [Figure 7] Figure 7 shows pattern 1 of the change in interpupillary distance. [Figure 8] Figure 8 shows the second pattern of change in interpupillary distance. [Figure 9] Figure 9 shows the information image displayed by the notification processing of the notification control unit. [Figure 10] Figure 10 is a diagram illustrating the notification processing of the notification control unit. [Figure 11] Figure 11 shows the configuration of the operation support system according to the second embodiment. [Figure 12] Figure 12 is a flowchart illustrating the notification processing of the notification control unit. [Modes for carrying out the invention]

[0010] [1. First Embodiment] [1.1 Configuration of the Operation Support System] Referring to Figures 1 and 2, the configuration of the operation support system 1 according to the first embodiment will be described. As shown in FIG. 1, the operation support system 1 includes an operation support unit 2 having a processor 10 and a memory 20. The operation support unit 2 is mounted on the vehicle 100 and can also be referred to as an in-vehicle device. The operation support unit 2 is an example of the gaze detection device of the present disclosure.

[0011] The operation support unit 2 is connected to a communication unit 30, a camera 31, a radar 32, a speed sensor 33, a position sensor 34, a driver monitor camera 35, a HUD (Head-Up Display) 36, a display 37, and a speaker 38 provided in the vehicle 100. The vehicle 100 is a four-wheeled vehicle and is an example of the vehicle of the present disclosure.

[0012] The communication unit 30 communicates with external communication systems such as a traffic information server 210 and a service providing server 211 via a communication network 200. The camera 31 captures the surrounding outside of the vehicle 100 including the front of the vehicle 100 and outputs the captured surrounding image to the operation support unit 2. The radar 31 detects the positions of objects (preceding vehicle, oncoming vehicle, etc.) existing around the vehicle body including the front of the vehicle 100 and outputs the position detection data to the operation support unit 2. The speed sensor 32 detects the traveling speed of the vehicle 100 and outputs the speed detection data to the operation support unit 2. The position sensor 33 detects the current position of the vehicle 100 using a known positioning technique utilizing GNSS or the like and outputs the current position data to the operation support unit 2.

[0013] As shown in FIG. 2, the driver monitor camera 35 is installed at the upper part of the windshield 101 of the vehicle 100 or the like, captures the driver D of the vehicle 100, and outputs the face image of the driver D to the operation support unit 2. The driver D can also be referred to as a user and a passenger and is an example of the target person of the present disclosure. The HUD 36 projects an information image 36a onto the windshield 101 for the driver D. The information image 36a is an image that is superimposed on a visual recognition target (road, preceding vehicle, oncoming vehicle, scenery, etc.) in front of and far from the vehicle 100 and is recognized by the driver U.

[0014] In FIG. 2, an example is shown in which the traveling speed of the vehicle 100 is displayed as the information image 36a. However, the present invention is not limited to this, and various notification information can be displayed. The notification information includes notification information regarding driving operations, warning information, and notification information not related to driving operations.

[0015] The notification information regarding driving operations includes, in addition to the traveling speed described above, traffic information obtained from the traffic information server 210 and route guidance information obtained from the navigation device provided in the vehicle 100. The warning information includes a warning such as a caution display for an object around the vehicle and an error message notifying an abnormality of the vehicle 100. The notification information not related to driving operations includes setting information such as air conditioning for improving the comfort inside the vehicle and the operation state of the audio system of the vehicle 100. The information image 36a is an example of the information display unit of the present disclosure.

[0016] The display 37 is installed on the dashboard 102 of the vehicle 100 and displays various information so that the driver D and the passenger P can view it. The display 37 is of a touch panel type, and various instructions are input to the operation support unit 2 by the driver D touching the display 37. That is, the display 37 also functions as an operation unit. For example, on the display 37, a control screen for controlling the air conditioning and audio (e.g., radio and volume) of the vehicle 100 and a setting screen for setting the air conditioning and audio can be displayed.

[0017] The speaker 38 emits various sounds into the vehicle 100 under the control of the operation support unit 2. From the speaker 38, for example, sounds related to driving operations and sounds of audio (e.g., radio) are output, and the driver D and the passenger P can listen to them. FIG. 2 also shows a steering wheel 103 provided in front of the driver D and side mirrors 104 provided on the left and right in front of the driver D. Note that the dashboard 102 may be provided with instruments for displaying the state of the vehicle 100.

[0018] The processor 10 of the operation support unit 2 functions as a computer that controls the operation support unit 2. By reading and executing the program 21 recorded in the memory 20, the processor 10 functions as an information acquisition unit 11, a face orientation detection unit 12, an interpupillary distance detection unit 13, a gaze detection unit 14, an ambient state detection unit 15, a danger level detection unit 16, and a notification control unit 17, as shown in Figure 1.

[0019] The information acquisition unit 11 includes an image acquisition unit 11a. The image acquisition unit 11a continuously acquires facial images of driver D captured by the driver monitor camera 35. The images acquired by the image acquisition unit 11a only need to include the eyes and face of driver D.

[0020] Furthermore, the information acquisition unit 11 can acquire information obtained by the communication unit 30, camera 31, radar 32, speed sensor 33, and position sensor 34. Figure 1 shows the case when this data is recorded in the memory 20 as acquired data 22.

[0021] The face orientation detection unit 12 continuously detects the orientation of driver D's face (hereinafter referred to as "face orientation") from the face image of driver D acquired by the image acquisition unit 11a. The processing for detecting face orientation is not particularly limited, but for example, by using a known image recognition algorithm, face orientation can be detected with high accuracy. The face orientation information detected by the face orientation detection unit 12 is recorded in the memory 20 in chronological order.

[0022] The interpupillary distance detection unit 13 continuously detects the distance between the left and right pupils of driver D (hereinafter referred to as "interpupillary distance") from the image of driver D acquired by the image acquisition unit 11a. The process for detecting the interpupillary distance involves first detecting the left and right eyes from the image of driver D, detecting the pupils (commonly known as "blacks") of the left and right eyes based on the brightness of the detected left and right eyes, and then applying image recognition processing to detect the distance between the center positions of the detected left and right pupils as the interpupillary distance.

[0023] This image recognition process allows for easy detection of interpupillary distance using existing image recognition techniques. However, the process for detecting interpupillary distance is not limited to the image recognition process described above. For example, an image recognition process may be applied that detects a portion including both the pupil and the iris, and then determines the distance between the centers of the detected left and right portions as the interpupillary distance. In this disclosure, the "black portion of the eye" includes both the pupil and the portion including both the pupil and the iris.

[0024] The interpupillary distance detected by the interpupillary distance detection unit 13 is recorded in memory 20 in chronological order. Figure 1 shows the case where face orientation data and interpupillary distance data are recorded in memory 20 as a face orientation / interpupillary distance database (hereinafter, "DB") 22. Note that face orientation and interpupillary distance are recorded in a way that allows for the identification of their respective acquisition timings.

[0025] The gaze detection unit 14 performs a process to detect the driver D's gaze position based on the driver D's face orientation detected by the face orientation detection unit 12 and the driver D's interpupillary distance detected by the interpupillary distance detection unit 13. The gaze position can be described as the location the gaze is directed towards, or the viewing position.

[0026] Here, Figure 3 shows the relationship between viewing distance and interpupillary distance. Viewing distance is the distance at which the left and right eyes perceive the object, and can also be called focal length or distance to the point of fixation. In Figure 3, reference numeral 50 indicates the left and right eyeballs of driver D, and reference numeral 53 indicates the left and right irises (commonly known as the "black of the eye," which includes the pupil). As shown in Figure 3, when driver D is viewing a distant object (in this embodiment, a road, a vehicle in front, an oncoming vehicle, scenery, etc.) through the windshield 101, the viewing distance becomes long because the line of sight is far away, and driver D's interpupillary distance becomes value L1.

[0027] In contrast, as shown in Figure 3, when driver D is viewing a nearby object located on or around the windshield 101 (information image 36a in this embodiment), the viewing distance becomes shorter because the line of sight is close, and driver D's interpupillary distance becomes value L2 (shorter than value L1). When the depth of such a gazed object changes, the pupils of both eyes move inward or outward, and this movement is called convergence and divergence. In the case of vehicle 100, as shown in Figure 3, the driver D will be in a situation where he views distant objects through the windshield 101 and a nearby object (information image 36a), resulting in a situation where the viewing position switches between distant and nearby objects while driving.

[0028] Figure 4 shows the change in interpupillary distance when driver D's line of sight switches between far and near. In Figure 4, the horizontal axis t represents the passage of time, and the vertical axis Dp represents interpupillary distance (hereinafter, "interpupillary distance Dp"). In Figure 4, "Viewing a distant object" indicates the situation where driver D is viewing a distant object through the windshield 101. Also, in Figure 4, "Viewing a near object" indicates the situation where driver D is viewing a near object (information image 36a). In this explanation, distant objects are those visible at a distance of, for example, a few meters to about 100 meters, and near objects are those visible at a distance of, for example, about 60 centimeters to about 1 meter. Note that the range of distant and near objects is not limited to the above distances, but rather any range in which the following specific eye movements occur.

[0029] The inventors focused on the change in interpupillary distance Dp when driver D's gaze position switches between far and near, and noticed that a specific eye movement occurs in which multiple rapid changes in interpupillary distance Dp occur at the timing of the switch, as shown in Figure 4.

[0030] More specifically, as shown in region α in Figure 4, we found that when the line of sight changes from far to near, eye movements occur in which the interpupillary distance Dp decreases sharply before increasing sharply. Furthermore, as shown in region β in Figure 4, we found that eye movements also occur in which the interpupillary distance Dp decreases sharply before increasing sharply when the line of sight changes from near to far. These eye movements can be presumed to be specific eye responses, at least in humans.

[0031] By detecting this specific eye movement, it becomes possible to identify that driver D's gaze position has changed between distant and near objects. Moreover, since this movement can be captured as a state in which the interpupillary distance Dp changes by a predetermined amount or more multiple times (two or more times) within a predetermined time, the amount of computation required is not enormous, and it is possible to quickly detect whether or not the above movement has occurred without using a processor with high computational processing power.

[0032] Therefore, in the operation support system 1 of this embodiment, the gaze detection unit 14 is equipped with a gaze change determination unit 14a that detects whether or not the above-mentioned specific eye movement has occurred, thereby determining whether or not the driver D's gaze position has changed with a change in viewing distance. The specific processing performed by the gaze change determination unit 14a will be described later.

[0033] The surrounding condition detection unit 15 searches for objects in front of the vehicle 100 based on the forward image of the vehicle 100 captured by the camera 31 and the position detection data of objects in front of the vehicle 100 detected by the radar 32. When the surrounding condition detection unit 15 extracts an object, the hazard level detection unit 16 calculates the TTC (Time To Collision), which is the predicted time until the vehicle 100 comes into contact with the object. If the calculated TTC is less than or equal to a predetermined time, the unit recognizes the object as having a hazard level of a predetermined level or higher and is subject to a warning.

[0034] The degree of risk is not limited to detection based on TTC, but may also be detected based on information other than TTC. The degree of risk detection unit 16 is an example of a "degree of risk detection unit that detects the degree of risk from the relationship between the vehicle and the surrounding conditions" as described in this disclosure.

[0035] The notification control unit 17 performs notification processing to the driver D based on at least one of the determination results of the gaze change determination unit 14a and the detection results of the danger level detection unit 16. The notification processing includes processing to display notification information related to driving operations using the HUD 36, processing to display warning information, and processing to emit audio corresponding to the notification information and warning information using the speaker 38.

[0036] Figure 5 shows an example of notification processing by the notification control unit 17. As shown in Figure 5, the notification control unit 17 uses the HUD 36 to project an outer frame 93 in the direction from driver D toward the vehicle to be warned about 250 (in this case, the vehicle in front) as a notification process to driver D. The outer frame 93 can draw attention to the vehicle to be warned about 250. The notification control unit 17 may also output a warning voice message ("Please pay attention to the vehicle in front") from the speaker 38.

[0037] [1.2 Eye-line change detection process] The gaze change determination process performed by the gaze change determination unit 14a will be explained according to the flowchart shown in Figure 6. The gaze change determination unit 14a continuously executes the process shown in the flowchart in Figure 6 when the vehicle 100 is in an operating state (power on state).

[0038] First, the gaze change determination unit 14a acquires the interpupillary distance Dp for a predetermined number of consecutive frames (a predetermined time) detected by the interpupillary distance detection unit 13 (step S1). Here, Figures 7 and 8 show examples of change patterns when the eye movement causes a sharp decrease followed by a sharp increase in interpupillary distance Dp due to a change in driver D's gaze position between far and near. Figure 7 is an example of a change pattern when the gaze position changes from far to near (change pattern 1), and Figure 8 is an example of a change pattern when the gaze position changes from near to far (change pattern 2). In this explanation, we will use the case where, in step S1, the interpupillary distance Dp for 5 frames from frames f1 to f5 in Figures 7 and 8 is obtained as an example.

[0039] Next, the gaze change determination unit 14a determines whether or not there is a decrease in the interpupillary distance Dp (step S2). If there is a decrease in the interpupillary distance Dp, the gaze change determination unit 14a proceeds to step S3; otherwise, it returns to step S1. If it returns to step S1, the interpupillary distance Dp for a predetermined number of consecutive frames after shifting the frame number by a predetermined number is obtained, and the processes from step S2 onwards are executed again.

[0040] In step S3, the gaze change determination unit 14a determines whether the decrease in interpupillary distance Dp De1 (see Figures 7 and 8) exceeds the first threshold T1 for determining a rapid decrease. The first threshold T1 is a threshold capable of detecting a sharp decrease in interpupillary distance Dp occurring in regions α and β as illustrated in Figure 4. More specifically, it is set to a threshold capable of detecting a sharp decrease in interpupillary distance Dp in change pattern 1 shown in Figure 7.

[0041] If the decrease amount De1 exceeds the first threshold T1 (step S3; YES), the gaze change determination unit 14a proceeds to step S4 if there is an increase in the interpupillary distance Dp after the decrease. In step S4, the gaze change determination unit 14a determines whether the increase in interpupillary distance Dp after the decrease De2 (see Figure 7) exceeds the second threshold T2 for determining a rapid increase. The second threshold T2 is a threshold capable of detecting the sharp decrease followed by a sharp increase in interpupillary distance Dp that occurs in regions α and β shown in Figure 4. More specifically, it is set to a threshold capable of detecting the sharp decrease followed by a sharp increase in interpupillary distance Dp in change pattern 1 shown in Figure 7.

[0042] If the increase in De2 exceeds the second threshold T2 for determining a sharp increase (step S4; YES), it can be determined that a situation corresponding to the phenomenon in which the interpupillary distance Dp in regions α and β shown in Figure 4 decreases sharply and then increases sharply has occurred. Therefore, it can be determined that the driver D's line of sight has changed between far and near, or in other words, that the driver D's line of sight has changed accompanied by a change in viewing distance. In this case, the process in step S5 is executed. In contrast, if the increase amount De2 does not exceed the second threshold T2 for determining a rapid increase (step S5; NO), the gaze change determination unit 14a returns to the process of step S1.

[0043] In step S5, the gaze change determination unit 14a determines whether the interpupillary distance Dp has become shorter than before the change (before the sudden decrease). If the interpupillary distance Dp has become shorter than before the change (step S5; YES), it is determined that the driver D's gaze position has changed to a closer position (corresponding to change pattern 1), and the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to near (step S6). If the interpupillary distance Dp has not become shorter than before the change (step S5; NO), the gaze change determination unit 14a returns to the process in step S1.

[0044] On the other hand, in step S3, if the decrease amount De1 does not exceed the first threshold T1 (step S3; NO), the gaze change determination unit 14a determines whether the decrease amount De1 of the interpupillary distance Dp (see Figures 7 and 8) exceeds the first threshold T1' for determining a rapid decrease (step S7). The first threshold T1' is a threshold capable of detecting a sharp decrease in interpupillary distance Dp occurring in regions α and β as illustrated in Figure 4. More specifically, it is set to a threshold capable of detecting a sharp decrease in interpupillary distance Dp in change pattern 2 shown in Figure 8. According to the inventors' studies, the decrease in interpupillary distance Dp De1 in change pattern 2 tended to be smaller than the decrease in interpupillary distance Dp De1 in change pattern 1. Therefore, it is preferable to set the first threshold T1' to a value smaller than the first threshold T1.

[0045] If the decrease amount De1 exceeds the first threshold T1' (step S7; YES), the gaze change determination unit 14a proceeds to step S8 if there is an increase in the interpupillary distance Dp after the decrease. In contrast, if the decrease amount De1 does not exceed the first threshold T1' (step S7; NO), the gaze change determination unit 14a returns to the process of step S1.

[0046] In step S8, the gaze change determination unit 14a determines whether the increase in interpupillary distance Dp after the decrease De2 (see Figure 8) exceeds the second threshold T2' for determining a rapid increase. The second threshold T2' is a threshold capable of detecting the sharp decrease followed by a sharp increase in interpupillary distance Dp that occurs in regions α and β shown in Figure 4. More specifically, it is set as the threshold capable of detecting the sharp decrease followed by a sharp increase in interpupillary distance Dp in change pattern 2 shown in Figure 8. According to the inventors' studies, the increase in interpupillary distance Dp De2 in change pattern 2 tended to be smaller than the increase in interpupillary distance Dp De2 in change pattern 1. Therefore, it is preferable to set the second threshold T2' to a value smaller than the second threshold T2.

[0047] If the increase in De2 exceeds the second threshold T2' for determining a rapid increase (step S8; YES), it can be determined that a situation corresponding to the phenomenon in which the interpupillary distance Dp in regions α and β shown in Figure 4 decreases sharply and then increases sharply has occurred. In this case, the process in step S9 is executed. In contrast, if the increase amount De2 does not exceed the second threshold T2' for determining a rapid increase (step S8; NO), the gaze change determination unit 14a returns to the process of step S1.

[0048] In step S9, the gaze change determination unit 14a determines whether the interpupillary distance Dp has increased compared to before the change (before the sudden decrease). If the interpupillary distance Dp has increased compared to before the change (step S9; YES), it is determined that the driver D's gaze position has changed to a more distant position (corresponding to change pattern 2), and the gaze change determination unit 14a determines that the driver D's gaze position has changed from near to far (step S10). If the interpupillary distance Dp has not increased compared to before the change (step S9; NO), the gaze change determination unit 14a returns to the process in step S1.

[0049] Incidentally, the gaze change determination unit 14a may use the above determination result based on the interpupillary distance D to determine the driver D's gaze position at a more specific location. For example, if the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to near, it is highly likely that the driver D has changed from looking at a distant object through the windshield 101 to looking at the windshield 101 (for example, the information image 36a), and therefore the unit may determine that the driver D is looking at the windshield 101.

[0050] Furthermore, the gaze change determination unit 14a may determine the driver's gaze position at a more specific location by utilizing the detection result of the face orientation detection unit 12 in addition to the determination result based on the interpupillary distance D. For example, if the gaze change determination unit 14a determines that driver D's gaze position has changed from far away to near, and the face direction detection unit 12 indicates that driver D's face is turned to either the left or the right, then it is determined that driver D is looking at the side mirror 104. Conversely, if the gaze change determination unit 14a determines that driver D's gaze position has changed from far away to near, and the face direction detection unit 12 indicates that driver D's face is turned towards the center in the vehicle width direction, then it is determined that driver D is looking at the display 37, which functions as a central display unit located in the center in the vehicle width direction. This allows for a more specific identification of driver D's gaze position.

[0051] [1.3 Notification Processing] Referring to Figures 9 and 10, the notification process performed by the notification control unit 17 will be described. As shown in Figure 9, when the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to near (step S7), the notification control unit 17 performs the process of projecting the information image 36a shown in Figure 9 onto the windshield 101 as notification information related to driving operations.

[0052] In Figure 9, image G1 in information image 36a shows an object ahead extracted by the surrounding state detection unit 15, and image G2 shows a sign recognized by the surrounding state detection unit 15 from the image of camera 31. Image G3 shows the driving speed of vehicle 100, and image G4 shows route guidance information obtained from the navigation device installed in vehicle 100.

[0053] For example, some of the images G1 to G4 (for example, image G3, which shows the driving speed) may be displayed at all times, while the remaining images may be displayed only when the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to near (including when it determines that the driver D is looking at the windshield 101 (for example, information image 36a)). The content of information image 36a, that is, the notification information regarding driving operations, may be changed as appropriate.

[0054] As shown in Figure 10, if the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to near, and the danger level detection unit 16 recognizes a warning target 250 whose danger level is above a predetermined danger level, then warning information different from the driving operation notification information exemplified in Figure 9 (the outer frame 93 shown in Figure 10) may be displayed.

[0055] After displaying the warning information (outer frame 93), if the gaze change determination unit 14a determines that the driver D's gaze position has changed from near to far, the display of the warning information (outer frame 93) may be stopped. Conversely, if, after displaying the warning information (outer frame 93), the gaze change determination unit 14a determines that the driver D's gaze position has not changed (step S10), processing to increase the level of the warning may be performed. For example, the level of the warning may be increased by displaying it on the information image 36a, or by emitting a warning sound from the speaker 38.

[0056] The gaze change determination unit 14a only needs to determine that the driver D's gaze position has not changed if, for example, the interpupillary distance Dp does not change. The warning information is not limited to the outer frame 93 and may be changed as appropriate.

[0057] Thus, the operation support unit 2 includes a pupillary distance detection unit 13 that detects the interpupillary distance Dp of the left and right eyes of the driver D based on an image including the eyes of the driver D, and a gaze detection unit 14 that detects the gaze position of the driver D based on acquired information including the pupillary distance Dp. The gaze detection unit 14 includes a gaze change determination unit 14a that, based on the acquired information, detects that eye movements have occurred in which the interpupillary distance Dp changes by a predetermined amount or more twice within a predetermined time, and determines that the gaze position of the driver D has changed accompanied by a change in viewing distance. This makes it possible to quickly identify changes in driver D's gaze position by utilizing specific eye movements in humans. This allows for real-time monitoring of driver D's gaze position and immediate responses to changes in driver D's attention and driving conditions, thereby contributing to improved safety. Ultimately, this can further enhance traffic safety and contribute to the development of a sustainable transportation system.

[0058] In this embodiment, the gaze detection unit 14 was shown as an example of detecting eye movements in which the interpupillary distance Dp changes by a predetermined amount or more twice within a predetermined time. However, it may also be configured to detect eye movements in which the above change occurs two or more times. In this case as well, the specific eye movements described above can be used to quickly identify changes in the driver D's gaze position.

[0059] In this embodiment, the processing of the interpupillary distance detection unit 13 is an example of the interpupillary distance detection step of this disclosure. The processing of the gaze detection unit 14 is an example of the gaze detection step of this disclosure. The processing of the gaze change determination unit 14a is an example of the gaze direction change determination step of this disclosure. The processing of the ambient state detection unit 15 is an example of the ambient state detection step of this disclosure, the processing of the danger level detection unit 16 is an example of the danger level detection step, and the processing of the notification control unit 17 is an example of the notification step of this disclosure.

[0060] [2. Second Embodiment] Referring to Figure 11, the configuration of the operation support system 1 according to the second embodiment will be described. The second embodiment differs from the first embodiment in that the operation support system 1 includes a microphone 39, a voice acquisition unit 11b, and a voice recognition unit 18, and the notification control unit 17 uses these components to perform notification processing related to tasks performed by the driver D, etc. Other configurations are the same as in the first embodiment, so only the differences will be explained. Also, in Figure 11, components identical to those in the first embodiment are denoted by the same reference numerals.

[0061] Microphone 39 is connected to the operation support unit 2, collects sound from inside the vehicle 100, and outputs that sound to the operation support unit 2. Microphone 39 is positioned around driver D and can collect the voices of driver D and passenger P. The voice acquisition unit 11b constitutes part of the information acquisition unit 11 and acquires voice collected via the microphone 39.

[0062] The speech recognition unit 18 recognizes the speech acquired by the speech acquisition unit 11b and generates text data by converting the speech data into text based on the recognition result. Furthermore, the speech recognition unit 18 has the function of detecting predefined tasks and commands based on the acquired text data. More specifically, the speech recognition unit 18 can refer to speech commands and keywords that make it possible to identify the tasks to be performed by driver D and identify the corresponding tasks from the acquired text data.

[0063] This task includes operations performed by driver D via the display 37, such as turning the air conditioning on and off, adjusting the air conditioning, and operating the audio system (radio, volume, etc.). This makes it possible to detect in advance, based on the conversation between driver D and passenger P, if the conversation includes voices related to the display 37, such as "Turn on the air conditioning" or "Play the radio."

[0064] [2.1 Notification Processing] In addition to the notification processing using the HUD 36 and speaker 38 described in the first embodiment, the notification control unit 17 uses the display 37 to perform other notification processing that contributes to improving the convenience of the driver D. The notification process performed by the notification control unit 17 and related processes will be explained according to the flowchart shown in Figure 12. Note that the flowchart shown in Figure 12 is executed continuously when the vehicle 100 is in operation (power on), or when it is in operation (power on) and driving.

[0065] As shown in Figure 12, the speech recognition unit 18 performs speech recognition processing to recognize the voice of the driver D or other person inside the vehicle (step S1a). Based on the speech recognition result, the speech recognition unit 18 detects the task that driver D will perform (step S2a). Next, the notification control unit 17 determines whether the task detected by the voice recognition unit 18 is a task to be performed via the display 37 (touch operation) (step S3a). If the task is to be performed via the display 37 (touch operation) (step S3a: YES), the notification control unit 17 executes the process in step 4a, and if it is not a task to be performed via the display 37 (step S3a: NO), it returns to the process in step S1.

[0066] In step S4a, the notification control unit 17 determines whether the gaze change determination unit 14a has determined that the driver D's gaze position has changed from far away to close. If the gaze change determination unit 14a has not determined that the driver D's gaze position has changed from far away to close (step S4a: NO), the notification control unit 17 repeats the determination process in step S4a until a preset waiting time has elapsed (step S5a: YES). In other words, for driver D to perform the task extended in step S2a, driver D needs to bring their gaze closer (to the position of the display 37), so the system waits until the timing is right for driver D to bring their gaze closer.

[0067] When the gaze change determination unit 14a determines that the driver D's gaze position has changed from far away to close (step S4a: YES), the notification control unit 17 displays information related to that task on the display 37 (step S6a). For example, if the task is related to air conditioning operation, the system will either display the air conditioning operation screen on the display 37 or highlight the air conditioning operation buttons as information related to the task. Similarly, if the task is related to radio or volume control, the system will either display the radio or volume control screen on the display 37 or highlight the radio or volume control buttons as information related to the task. This improves convenience for driver D.

[0068] [3. Other Embodiments] The above embodiments are merely one way of implementing the present invention, and can be modified and applied at will without departing from the spirit of the invention.

[0069] For example, in the above embodiment, the gaze change determination unit 14a determined that the driver D was looking at the windshield 101 (for example, the information image 36a) when it determined that the driver D's gaze position had changed from far away to near. However, it is also possible to determine that the driver D is looking at the dashboard 102. For example, if the information image 36a is not projected onto the windshield 2, there is a high possibility that the driver D is looking at the instruments or the display 37 (central display unit) on the dashboard 102.

[0070] Furthermore, notification information displayed on the information image 36a (including notification information related to driving operations, warning information, and notification information not related to driving operations) may be displayed on an information display unit provided on the dashboard 102, such as a display 37 or instruments, if the HUD 36 is not provided.

[0071] Furthermore, although the present invention has been described in the case where it is applied to an operation support unit 2 mounted on a vehicle 100 and functioning as a gaze detection device, the present invention may also be applied to a gaze detection device mounted on any moving body that is operated by a driver D. The moving body may be an aircraft, a motorcycle, or a ship, etc.

[0072] Furthermore, the configurations of each part shown in Figures 1, 11, etc., can be broadly applied to configurations implemented by software, configurations implemented by hardware, and configurations implemented by a combination of software and hardware. In addition, some or all of the configuration of the gaze detection device of the present invention may be provided on an external system of the vehicle 100, such as a server on the communication network 200. In this case, the information obtained by the vehicle 100 is transmitted to the external system, at least a part of the processing performed by the gaze detection device is executed on the external system, and output data based on the processing results is transmitted to the vehicle 100 to realize various notifications and warnings in the vehicle 100.

[0073] Furthermore, the way in which the processing units in the flowchart are divided and the processing order are not limited to the illustrated example and may be changed as appropriate. Furthermore, although the case in which the program 21 for realizing the gaze detection method of the present invention is recorded in a gaze detection device has been described, the program 21 may also be obtained from an external device via communication. Alternatively, the program 21 may be recorded on a recording medium that is readable by a computer. Magnetic, optical, or semiconductor memory devices can be used as the recording medium.

[0074] [4. Configurations supported by the above embodiments] The above embodiment is a specific example of the following configuration.

[0075] (Configuration 1) A gaze detection device comprising: an interpupillary distance detection unit that detects the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes; and a gaze detection unit that detects the subject's gaze position based on acquired information including the interpupillary distance, wherein the gaze detection unit has a gaze change determination unit that, based on the acquired information, detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time, and determines that the subject's gaze position has changed along with a change in viewing distance. This configuration allows for the rapid identification of changes in a subject's gaze position by utilizing specific eye movements, at least in humans, thereby contributing to improved safety.

[0076] (Configuration 2) The gaze detection device according to Configuration 1, wherein the gaze change determination unit determines that the subject's gaze position has changed from far away to near when it detects that the amount of change in the interpupillary distance decreases by a predetermined threshold or more, and then the amount of change in the interpupillary distance increases by a predetermined threshold or more, and the interpupillary distance is shorter than before the change. This configuration allows for the rapid detection of a change in the subject's gaze position from a distant to a close-up.

[0077] (Configuration 3) The gaze detection device according to Configuration 1 or 2, wherein the gaze change determination unit determines that the subject's gaze position has changed from near to far when it detects that the amount of change in the interpupillary distance decreases by a predetermined threshold or more, and then the amount of change in the interpupillary distance increases by a predetermined threshold or more, and the interpupillary distance is longer than before the change. This configuration allows for the rapid detection of a change in the subject's gaze position from near to far.

[0078] (Configuration 4) The gaze detection device according to any one of Configurations 1 to 3, wherein the gaze detection device is mounted on a vehicle and the target person is an occupant of the vehicle. This configuration allows for the rapid detection of changes in the vehicle occupants' line of sight.

[0079] (Configuration 5) The gaze detection device according to Configuration 3, wherein the gaze detection device is mounted on a vehicle, the target person is the driver of the vehicle, and the gaze change determination unit determines that the driver is looking at at least one of the vehicle's windshield or dashboard when it determines that the driver's gaze position has changed from far away to near. This configuration allows for the rapid detection of changes in the driver's line of sight, such as shifting to the windshield or dashboard.

[0080] (Configuration 6) The gaze detection device according to Configuration 2, wherein the gaze detection device is mounted on a vehicle, the target person is the driver of the vehicle, and the gaze change determination unit determines that the driver of the vehicle has changed their gaze position from far away to near, and when it detects a change in the direction of the driver's face, it determines that the driver is looking at the central display unit located in the center of the vehicle width direction, or the side mirror. This configuration allows for quick detection of changes in the driver's line of sight between the central display and the side mirrors. Furthermore, it is possible to determine whether the driver is looking at the central display or the side mirrors based on the direction of their face.

[0081] (Configuration 7) The gaze detection device according to Configuration 5, further comprising a notification control unit that displays notification information related to driving operations on an information display unit provided on the windshield or dashboard when the gaze change determination unit determines that the gaze position has changed from far away to near. This configuration allows drivers to see necessary information at the appropriate time by displaying driving operation notifications in line with their line of sight.

[0082] (Configuration 8) A gaze detection device according to Configuration 7, comprising: an ambient state detection unit for detecting the state of the area around the vehicle; and a risk level detection unit for detecting the degree of risk from the relationship between the vehicle and the surrounding state, wherein the gaze change determination unit determines that the gaze position has changed from far away to near, and if the degree of risk is above a predetermined risk level, the information display unit displays warning information different from the notification information regarding the driving operation. This configuration allows for the rapid notification of warning information to the driver if a highly dangerous situation occurs outside the vehicle while the driver is looking nearby, making it easier to avoid danger.

[0083] (Configuration 9) The gaze detection device according to Configuration 8, wherein the gaze change determination unit determines that the subject's gaze position has changed from near to far when it detects that the amount of change in the interpupillary distance has decreased by a predetermined threshold or more, and that the amount of change in the interpupillary distance has increased by a predetermined threshold or more, and that the interpupillary distance has become longer than before the change, and the notification control unit displays the warning information, and if the gaze change determination unit determines that the gaze position has changed from near to far, it stops displaying the warning information, and if it determines that the gaze position has not changed, it increases the degree of the warning based on the warning information. This configuration allows the system to appropriately adjust whether to stop or increase the intensity of the warning based on the driver's gaze after the warning information is displayed, making it easier to avoid danger.

[0084] (Configuration 10) A gaze detection device according to any one of Configurations 7 to 9, comprising a voice recognition unit that recognizes voices inside the vehicle, wherein the notification control unit has a function to detect a task to be performed by the driver based on the voice recognition result, and when the task is detected and the gaze change determination unit determines that the gaze position has changed from far away to near, the information display unit displays information related to the task. This configuration allows drivers to see task-related information at the appropriate location and timing while performing a task, improving convenience.

[0085] (Configuration 11) The gaze detection device according to any one of Configurations 1 to 10, wherein the interpupillary distance detection unit detects the left and right eyes from the image, detects the pupils of the left and right eyes based on the brightness of the detected left and right eyes, and detects the distance between the center positions of the detected left and right pupils as the interpupillary distance. This configuration allows for easy detection of interpupillary distance using existing image recognition technology.

[0086] (Configuration 12) The gaze detection device according to Configuration 3, wherein the gaze change determination unit determines whether the subject's gaze position has changed from near to far, by utilizing the fact that the amount of change in interpupillary distance when the gaze position changes from far to near is smaller than the amount of change in interpupillary distance when the gaze position changes from near to far. This configuration utilizes the fact that the change in interpupillary distance when the subject's gaze position changes from near to far (change pattern 2) is smaller than the change in interpupillary distance when the subject's gaze position changes from far to near (change pattern 1). This makes it possible to accurately determine whether the subject's gaze position has changed from near to far. As a result, the accuracy of distance determination of the gaze position is improved.

[0087] (Configuration 13) A gaze detection method performed by a gaze detection device, comprising: an interpupillary distance detection step of detecting the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes; and a gaze detection step of detecting the subject's gaze position based on acquired information including the interpupillary distance, wherein the gaze detection step includes a gaze direction change determination process that determines that the subject's gaze position has changed with a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time, based on the acquired information. This method allows for the rapid identification of changes in a subject's gaze position by utilizing specific eye movements, at least in humans, thereby contributing to improved safety.

[0088] (Configuration 14) A program that causes a computer-controlled gaze detection device to function as a pupillary distance detection unit that detects the interpupillary distance between the left and right eyes of a subject based on an image including the subject's eyes, and a gaze detection unit that detects the subject's gaze direction based on acquired information including the pupillary distance, wherein the gaze detection unit includes a gaze direction change determination process that determines that the subject's gaze position has changed along with a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time based on the acquired information. According to this program, by utilizing specific eye movements in humans, it is possible to quickly identify changes in a subject's gaze position, thereby contributing to improved safety. [Explanation of symbols]

[0089] 1...Operation support system, 2...Operation support unit (gaze detection device), 10...Processor, 11...Information acquisition unit, 12...Face orientation detection unit, 13...Interpupillary distance detection unit, 14...Gaze detection unit, 14a...Gaze change determination unit, 15...Surrounding condition detection unit, 16...Danger level detection unit, 17...Notification control unit, 18...Voice recognition unit, 20...Memory, 21...Program, 30...Communication unit, 31...Camera, 32...Radar, 33...Speed ​​sensor, 34...Position sensor, 35...Driver monitor camera, 36...HUD, 36a...Information image (information display unit), 37...Display (information display unit, central display unit), 38...Speaker, 39...Microphone, 100...Vehicle, 101...Windshield, 102...Dashboard, 104...Side mirror, D...Driver (target person), P...Passenger, Dp...Interpupillary distance.

Claims

1. A pupillary distance detection unit detects the interpupillary distance between the left and right pupils of the subject based on an image including the subject's eyes, The system includes a gaze detection unit that detects the gaze position of the subject based on acquired information including the interpupillary distance, The gaze detection unit has a gaze change determination unit that, based on the acquired information, determines that the subject's gaze position has changed along with a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time. Eye-tracking device.

2. The gaze change determination unit determines that the subject's gaze position has changed from far away to near when it detects that the amount of change in the interpupillary distance decreases by a predetermined threshold or more, and then increases by a predetermined threshold or more, and that the interpupillary distance is shorter than before the change. The gaze detection device according to claim 1.

3. The gaze change determination unit determines that the subject's gaze position has changed from near to far when it detects that the amount of change in the interpupillary distance decreases by a predetermined threshold or more, and then increases by a predetermined threshold or more, and that the interpupillary distance is longer than before the change. The gaze detection device according to claim 1.

4. The US eye-tracking device is mounted on a vehicle. The person concerned is a occupant of the vehicle. The gaze detection device according to claim 1.

5. The US eye-tracking device is mounted on a vehicle. The person in question is the driver of the vehicle, If the gaze change determination unit determines that the driver's gaze position has changed from far away to near, it determines that the driver is looking at at least one of the vehicle's windshield or dashboard. The gaze detection device according to claim 2.

6. The US eye-tracking device is mounted on a vehicle. The person in question is the driver of the vehicle, The gaze change determination unit determines that the driver's line of sight has changed from far away to near, and also detects a change in the driver's facial orientation, and determines that the driver is looking at the central display unit located in the center of the vehicle's width direction, or at the side mirrors. The gaze detection device according to claim 2.

7. If the gaze change determination unit determines that the gaze position has changed from far away to near, the system includes a notification control unit that displays notification information related to driving operations on the information display unit provided on the windshield or dashboard. The gaze detection device according to claim 5.

8. The surrounding condition detection unit detects the conditions around the vehicle, The vehicle comprises a risk level detection unit that detects the degree of risk based on the relationship between the vehicle and the surrounding conditions, If the gaze change determination unit determines that the gaze position has changed from far away to near, and the degree of danger is above a predetermined level, the information display unit will display warning information different from the notification information regarding the driving operation. The gaze detection device according to claim 7.

9. The gaze change determination unit determines that the subject's gaze position has changed from near to far when it detects that the amount of change in the interpupillary distance decreases by a predetermined threshold or more, and then increases by a predetermined threshold or more, and that the interpupillary distance is longer than before the change. After displaying the warning information, the notification control unit stops displaying the warning information if the gaze change determination unit determines that the gaze position has changed from near to far, and increases the intensity of the warning provided by the warning information if it determines that the gaze position has not changed. The gaze detection device according to claim 8.

10. The vehicle has a voice recognition unit that recognizes voices inside the vehicle, The notification control unit has a function to detect a task to be performed by the driver based on the voice recognition result, and when it detects the task and the gaze change determination unit determines that the gaze position has changed from far away to near, it displays information related to the task on the information display unit. The gaze detection device according to claim 7.

11. The interpupillary distance detection unit detects the left and right eyes from the image, detects the pupils of the left and right eyes based on the brightness of the detected pupils, and detects the distance between the center positions of the detected pupils as the interpupillary distance. The gaze detection device according to claim 1.

12. The gaze change determination unit determines whether the subject's gaze position has changed from near to far by utilizing the fact that the change in interpupillary distance when the gaze position changes from far to near is smaller than the change in interpupillary distance when the gaze position changes from near to far. The gaze detection device according to claim 3.

13. A gaze detection method performed by a gaze detection device, A pupillary distance detection step that detects the interpupillary distance between the left and right eyes of the subject based on an image including the subject's eyes, The process includes a gaze detection step that detects the gaze position of the subject based on acquired information including the interpupillary distance, The gaze detection step includes a gaze direction change determination process that, based on the acquired information, determines that the subject's gaze position has changed along with a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time. Eye-tracking detection method.

14. A computer-controlled gaze detection device, A pupillary distance detection unit detects the interpupillary distance between the left and right pupils of the subject based on an image including the subject's eyes, A program that functions as a gaze detection unit that detects the direction of the subject's gaze based on acquired information including the interpupillary distance, The gaze detection unit includes a gaze direction change determination process that, based on the acquired information, determines that the subject's gaze position has changed along with a change in viewing distance when it detects that eye movements have occurred in which the interpupillary distance changes by a predetermined amount or more multiple times within a predetermined time. program.

Images