Monitoring device, monitoring-purpose computer program and monitoring method

Abstract

To provide a monitoring device that can accurately determine a gaze position of a driver.SOLUTION: A monitoring device includes: a first estimation unit that estimates a face direction and a visual line direction of a driver based on an image representing the vicinity of a driver seat; a second estimation unit that estimates a gazing area within a vehicle cabin where the driver's face is facing based on the face direction of the driver estimated by the first estimation unit; a determination unit that determines a gaze position of the driver using a predetermined determining criterion based on the visual line direction estimated by the first estimation unit; and a changing unit that changes the determining criterion so that the determination unit is more likely to determine that the gaze position is included in the gazing area more than in an adjacent region adjacent to the gazing area. The determination unit determines the gaze position of the driver using the determining criterion changed by the changing unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a monitoring device, a monitoring computer program, and a monitoring method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, monitoring devices are used to monitor a driver who drives a vehicle. The monitoring devices monitor the driver, for example, to monitor whether the driver is looking ahead of the vehicle (see, for example, Patent Document 1).

[0003] Methods for the monitoring device to monitor the driver include, for example, estimating the direction of the driver's face and estimating the driver's gaze position inside the vehicle cabin.

[0004] Since the driver does not necessarily look in the direction in which the face is facing, it is considered that monitoring the driver based on the gaze position enables accurate monitoring of whether the driver is looking ahead. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-151606 A Summary of the Invention [Problem to be solved by the invention]

[0006] The driver's gaze position is estimated based on an image showing the driver's eyes, but it can be difficult to accurately estimate the driver's gaze position because the estimation of the gaze position is affected by the state of the image in the eye area, and the gaze position can fluctuate slightly depending on the movement of the eye.

[0007] Therefore, an object of the present disclosure is to provide a monitoring device that can accurately determine the driver's gaze position. [Means for solving the problem]

[0008] According to one embodiment, a monitoring device is provided, which includes a first estimation unit that estimates the driver's face direction and line of sight direction based on an image showing the vicinity of the driver's seat, a second estimation unit that estimates a gaze area in the vehicle cabin to which the driver's face is directed based on the driver's face direction estimated by the first estimation unit, a determination unit that determines the driver's gaze position using a predetermined determination criterion based on the gaze direction estimated by the first estimation unit, and a change unit that changes the determination criterion so that the determination unit is more likely to determine that the gaze position is included in the gaze area than in an adjacent area adjacent to the gaze area, and the determination unit determines the driver's gaze position using the determination criterion changed by the change unit. Effect of the Invention

[0009] The monitoring device according to the present disclosure determines the driver's gaze position based on the determination criteria that are changed based on the direction of the driver's face, and therefore can accurately estimate the driver's gaze position. [Brief description of the drawings]

[0010] [Figure 1] 1A and 1B are diagrams illustrating an outline of the operation of the monitoring device of the present embodiment, where 1A is a diagram showing a vehicle, and 1B is a diagram showing the interior of the vehicle. [Diagram 2] 1 is a hardware configuration diagram of a vehicle in which a monitoring device according to an embodiment of the present invention is implemented. [Diagram 3] 5 is an example of an operational flowchart relating to a monitoring process of the monitoring device of the present embodiment. [Figure 4] 3A and 3B are diagrams illustrating examples of an inattentive region and a non-inattentive region of the monitoring device of the present embodiment. [Diagram 5] 6 is an example of an operational flowchart relating to a determination process of the monitoring device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] 1(A) and 1(B) explain an outline of the operation of the monitoring device of this embodiment, where Fig. 1(A) is a diagram showing a vehicle, and Fig. 1(B) is a diagram showing the interior of the vehicle.

[0012] As shown in Fig. 1(A), a vehicle 10 has an automatic control device 11 and a monitoring device 12. The automatic control device 11 has an automatic driving mode (e.g., driving modes of levels 3 to 5) in which the automatic control device 11 mainly drives the vehicle 10, and a manual driving mode (e.g., driving modes of levels 0 to 2) in which the driver 40 mainly drives the vehicle 10. The vehicle 10 may be an automatic driving vehicle. The monitoring device 12 monitors whether the driver 40 is looking away from the road while driving.

[0013] In the automatic driving mode, the driver 40 is not primarily responsible for driving, and is therefore able to remain relaxed. However, even in the automatic driving mode, if the automatic control device 11 determines that the vehicle 10 cannot be driven safely under automatic control, it notifies the driver 40 of a control transfer request via a user interface (UI) 3, which requests that the primary responsibility for driving the vehicle 10 be transferred from the automatic control device 11 to the driver 40.

[0014] The driver 40 must approve the control transfer request before starting to drive the vehicle 10 in the manual driving mode. Therefore, the driver 40 is required to pay attention to the environment around the vehicle 10 even in the automatic driving mode.

[0015] In the manual driving mode, the driver 40 drives the vehicle 10 while paying attention to the environment around the vehicle 10. As shown in Fig. 1(B), the automatic control device 11 controls the operation of the vehicle 10 based on the operation of the driver 40. The driver 40 sits in the driver's seat 31 in the vehicle compartment 30 so as to be able to operate the steering wheel 32, the brake pedal 33, and the accelerator pedal 34.

[0016] The monitoring device 12 estimates the facial orientation and gaze direction of the driver 40 based on a monitoring image acquired by the monitoring camera 2. The monitoring image shows the vicinity of the driver's seat 31. The monitoring device 12 estimates the gaze area in the vehicle interior 30 to which the face of the driver 40 is directed based on the facial orientation of the driver 40. The monitoring device 12 also uses a predetermined determination criterion to determine the gaze position in the vehicle interior 30 at which the eyes of the driver 40 are gazing, based on the gaze direction of the driver 40.

[0017] The monitoring device 12 changes the criteria used to determine the gaze position of the driver 40 so that the gaze position is more likely to be determined to be included in the gaze area than in an adjacent area adjacent to the gaze area. The monitoring device 12 uses the changed criteria to determine the gaze position in the vehicle interior 30 based on the line of sight of the driver 40.

[0018] In this embodiment, the areas within the vehicle interior 30 are classified into non-inattentive areas, in which if the driver's 40 gaze position is included, it is determined that the driver 40 is not looking aside, and inattentive areas, in which if the driver's 40 gaze position is included, it is determined that the driver 40 is looking aside.

[0019] The monitoring device 12 judges whether the gaze position is in the inattentive area or in the non-inattentive area. When the gaze position of the driver is in the inattentive area, the monitoring device 12 notifies the driver 40 of a warning via the UI 3.

[0020] Here, the gaze position may be influenced by the state of the monitoring image showing the eye area. Also, the gaze position may vary minutely depending on the movement of the line of sight. On the other hand, the face direction is estimated using the entire face of the driver 40, so it is less influenced by the monitoring image than the gaze position. Also, the face direction does not change as quickly as the movement of the line of sight.

[0021] Therefore, the monitoring device 12 changes the determination criterion so that the gaze area toward which the face of the driver 40 is facing is more likely to be estimated as being gazed upon by the driver 40. The monitoring device 12 uses this determination criterion to determine the gaze position of the driver 40, so that the gaze position of the driver 40 is not erroneously determined.

[0022] As described above, the monitoring device 12 of the present embodiment determines the gaze position of the driver 40 based on the determination criterion changed based on the face direction of the driver 40, and therefore can accurately estimate the gaze position of the driver 40.

[0023] 2 is a hardware configuration diagram of a vehicle 10 in which a monitoring system 1 including a monitoring device 12 of this embodiment is implemented. The vehicle 10 has a monitoring camera 2, a user interface (UI) 3, an automatic control device 11, a monitoring device 12, etc. The monitoring system 1 includes at least the monitoring camera 2 and the monitoring device 12.

[0024] The surveillance camera 2, the UI 3, the automatic control device 11, and the surveillance device 12 are communicatively connected via an in-vehicle network 13 that complies with a standard such as a controller area network.

[0025] The surveillance camera 2 is disposed within the vehicle interior 30 so as to be able to obtain a surveillance image including the face of a driver 40 who drives the vehicle 10. The surveillance camera 2 is disposed, for example, above the steering column.

[0026] The surveillance camera 2 is an example of an image acquisition unit. The surveillance camera 2 acquires a surveillance image showing the vicinity of the driver's seat 31 at a surveillance image acquisition time having a predetermined period, for example. The face of the driver 40 seated in the driver's seat 31 is shown in the surveillance image.

[0027] The surveillance camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector.

[0028] Every time the surveillance camera 2 captures a surveillance image, it outputs the surveillance image and the image capture time when the surveillance image was captured to the surveillance device 12 via the in-vehicle network 13. The surveillance image is used by the surveillance device 12 to determine the facial orientation and gaze position of the driver 40.

[0029] The UI3 is an example of a notification unit. The UI3 is controlled by the automatic control device 11 or the monitoring device 12 to notify the driver 40 of information related to the vehicle 10, such as a warning or a control transition request to watch the traveling direction of the vehicle 10. The UI3 has a display device 3a such as a liquid crystal display or a touch panel to display the warning or the control transition request. The UI3 may also have an audio output device (not shown) for notifying the driver 40 of information. The UI3 also has, for example, a touch panel or an operation button as an input device for inputting operation information from the driver 40 to the vehicle 10. The UI3 outputs the input operation information to the automatic control device 11 or the like via the in-vehicle network 13.

[0030] The automatic control device 11 controls the operation of the vehicle 10. The automatic control device 11 has an automatic driving mode in which the vehicle 10 is driven automatically, and a manual driving mode in which the operation of the vehicle 10 is controlled based on the operation of the driver 40. In the automatic driving mode, the automatic control device 11 mainly drives the vehicle 10. In the automatic driving mode, the automatic control device 11 controls operations such as steering, driving, and braking based on detection information and the like from sensors (not shown) mounted on the vehicle 10.

[0031] In addition, in the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10, such as steering, driving, and braking, based on the operation of the driver 40. In the manual driving mode, the driver 40 takes the lead in driving the vehicle 10. In the manual driving mode, the automatic control device 11 controls the operation of the vehicle 10 based on the operation of at least one of the steering wheel 32, the brake pedal 33, and the accelerator pedal 34 by the driver 40.

[0032] The automatic control device 11 mainly drives the vehicle 10 means that the automatic control device 11 controls the steering, driving, and braking operations without the driver's involvement. The driver 40 mainly drives the vehicle 10 means that the driver is involved in controlling at least one of the steering, driving, and braking operations.

[0033] The monitoring device 12 executes estimation processing, determination processing, change processing, and control processing. To this end, the monitoring device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 12 to the in-vehicle network 13.

[0034] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in the information processing executed by the processor 23 and various data.

[0035] All or part of the functions of the monitoring device 12 are functional modules realized by, for example, a computer program running on the processor 23. The processor 23 has an estimation unit 231, a determination unit 232, a change unit 233, and a control unit 234. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0036] The automatic control device 11 and the monitoring device 12 are, for example, electronic control devices (ECUs). In Fig. 3, the automatic control device 11 and the monitoring device 12 are described as separate devices (for example, ECUs), but all or part of these devices may be configured as a single device.

[0037] Fig. 3 is an example of an operation flowchart relating to the monitoring process of the monitoring device 12 of this embodiment. When the automatic control device 11 is in the manual operation mode, the monitoring device 12 executes the monitoring process at a monitoring time having a predetermined cycle in accordance with the operation flowchart shown in Fig. 3. The cycle of the monitoring time can be set to, for example, 10 to 60 seconds. Note that the monitoring device 12 may execute the monitoring process even when the automatic control device 11 is in the automatic operation mode.

[0038] First, the monitoring device 12 determines the gaze position of the driver 40 in the vehicle interior 30 based on the monitored image (step S101). The details of this determination process will be described later.

[0039] Next, the determination unit 232 determines whether the gaze position of the driver 40 in the vehicle interior 30 is in the inattentive region or the non-inattentive region (step S102).

[0040] 4 is a diagram showing examples of inattentive viewing areas and non-inattentive viewing areas of the monitoring device of this embodiment. The interior of the vehicle compartment 30 is classified into non-inattentive viewing areas G1 to G6 and an inattentive viewing area NG. The non-inattentive viewing areas G1 to G6 and the inattentive viewing area NG are set on a three-dimensional surface. The three-dimensional surface is represented, for example, by a vehicle coordinate system having an origin at a predetermined position within the vehicle. The inattentive viewing area NG includes areas adjacent to the non-inattentive viewing areas G1 to G6. The inattentive viewing area NG surrounds the periphery of the non-inattentive viewing areas G1 to G6.

[0041] The non-inattentive looking area G1 is set in a predetermined range of the windshield. The non-inattentive looking area G2 is set at the position of the meter panel. The non-inattentive looking area G3 is set at the position of the display device 3a. The non-inattentive looking area G4 is set at the position of the rearview mirror. The non-inattentive looking areas G5 and G6 are set at the positions of the side mirrors.

[0042] The gaze position of the driver 40 is represented as an intersection of the line of sight of the driver 40 and the non-inattentive regions G1 to G6 or the inattentive region NG.

[0043] When the gaze position of the driver 40 is within the inattentive zone NG, the driver 40 is presumed to be looking aside. On the other hand, when the gaze position of the driver 40 within the vehicle interior 30 is within any of the non-inattentive zones G1 to G6, the driver 40 is presumed not to be looking aside.

[0044] If the gaze position of the driver 40 in the vehicle interior 30 is within the inattentive viewing area NG (step S102-Yes), the control unit 234 issues a warning to the driver 40 via the UI3 requesting him to gaze in the traveling direction of the vehicle 10 (step S103), and ends the series of processes. On the other hand, if the gaze position of the driver 40 in the vehicle interior 30 is within any of the non-inattentive viewing areas G1 to G6 (step S102-No), the series of processes ends.

[0045] Fig. 5 is an example of an operational flowchart relating to the judgment process of the monitoring device 12 of this embodiment. The monitoring device 12 executes the judgment process at a judgment time having a predetermined cycle in accordance with the operational flowchart shown in Fig. 5. The cycle of the judgment time can be, for example, 1 to 10 seconds.

[0046] First, the estimation unit 231 estimates the face direction and gaze direction of the driver 40 based on the monitoring image (step S201). The angle representing the face direction is represented by a horizontal or vertical angle between the traveling direction of the vehicle 10 and the direction in which the driver 40 faces. The estimation unit 231 inputs the acquired monitoring image to a classifier trained to detect predetermined parts of the face, such as the corners of the eyes, the inner corners of the eyes, and the corners of the mouth, to specify the positions of the predetermined parts included in the monitoring image. Then, the estimation unit 231 compares the positions of the predetermined parts detected from the monitoring image with a standard three-dimensional model of a face. Then, the estimation unit 231 detects the face direction angle in the three-dimensional model when the position of each part is most compatible with the position of the part detected from the monitoring image as the face direction angle in the monitoring image.

[0047] The classifier can be, for example, a convolutional deep neural network (DNN) having multiple layers connected in series from the input side to the output side. The DNN, which has previously learned face images including predetermined facial features as training data, operates as a classifier that identifies the positions of the predetermined facial features.

[0048] Next, the estimation unit 231 determines the position of the pupil center and the position of the eyeball center based on the positions of the outer corner and the inner corner of the eye, and estimates the direction connecting the positions of the eyeball center and the pupil center as the line of sight of the driver 40 (step S202).

[0049] Next, the estimation unit 231 estimates the gaze area of ​​the driver 40 (step S203). The estimation unit 231 determines whether the line of sight of the driver 40 intersects with any of the non-inattentive areas G1 to G6 or with the inattentive area NG. The estimation unit 231 estimates the area that intersects with the line of sight direction as the gaze area.

[0050] Next, the change unit 233 changes the determination criterion so that the determination unit 232 is more likely to determine that the gaze position is included in the gaze region than in an adjacent region adjacent to the gaze region (step S204).

[0051] That is, the modification unit 233 modifies the determination criteria for the position to which the gaze direction is directed so that the gaze position to which the driver 40 is directed is more likely to be determined to be included in the gaze area than in an adjacent area.

[0052] For example, when the gaze area is the non-inattentive looking area G3, the change unit 233 enlarges the non-inattentive looking area G3 which is the gaze area and reduces the portion of the inattentive looking area NG adjacent to the non-inattentive looking area G3, as shown in Fig. 4. The adjacent inattentive looking area NG is reduced by the amount that the non-inattentive looking area G3 is enlarged.

[0053] Furthermore, the change unit 233 may move the gaze area in a predetermined direction according to the gaze area. For example, when the gaze area is the non-inattentive looking area G4, the change unit 233 may move the non-inattentive looking area G4 downward as shown in Fig. 4. This is because the driver 40 tends to look downward of the rearview mirror when looking at the rearview mirror.

[0054] Next, the determination unit 232 determines the gaze position of the driver 40 using the determination criterion changed by the change unit 233 based on the line of sight of the driver 40 (step S205), and ends the series of processes. When the time during which the line of sight of the driver 40 intersects with the same area exceeds a predetermined reference time, the determination unit 232 may determine that the average position of the intersecting position within this reference time is the gaze position of the driver 40.

[0055] For example, in the above-mentioned step S204, the modification unit 233 may shorten the determination time for which it is determined that the gaze area is being gazed upon by the driver 40, compared to the determination time for which it is determined that the adjacent area adjacent to the gaze area is being gazed upon by the driver 40.

[0056] As described above in detail, the monitoring device of the present embodiment determines the driver's gaze position based on the determination criterion that is changed based on the direction of the driver's face, and therefore can accurately estimate the driver's gaze position.

[0057] In the present disclosure, the monitoring device, the monitoring computer program, and the monitoring method of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

[0058] For example, in the above embodiment, the area in the vehicle cabin is classified into an inattentive area and a non-inattentive area, but the area in the vehicle cabin does not have to be classified into an inattentive area and a non-inattentive area. In this case, the monitoring device may set the area in the vehicle cabin toward which the driver faces as a gaze area, and set areas of a predetermined range adjacent to the gaze area as adjacent areas. [Explanation of symbols]

[0059] 1. Surveillance System 2. Surveillance cameras 3 User Interface (UI) 3a Display device 10 Vehicles 11 Automatic control device 12 Monitoring equipment 21 Communication Interface 22 Memory 23 Processors 231 Estimation Department 232 Judgment section 233 Changes 234 Control Unit 14 In-vehicle network 30 Cabin 31 Driver's seat 32 Steering wheel 33 Brake pedal 34 Accelerator pedal

Claims

[Claim 1] a first estimation unit that estimates a face direction and a line of sight direction of a driver based on an image showing the vicinity of the driver's seat; a second estimation unit that estimates a gaze area in the vehicle cabin toward which the driver's face is directed based on the direction of the driver's face estimated by the first estimation unit; a determination unit that determines a gaze position of a driver using a predetermined determination criterion based on the gaze direction estimated by the first estimation unit; a change unit that changes the determination criterion so that the determination unit is more likely to determine that the gaze position is included in the gaze region than in an adjacent region adjacent to the gaze region; having The determination unit determines the gaze position of the driver by using the determination criterion changed by the change unit. A monitoring device comprising:

Images