Vehicle distraction monitoring device

Abstract

To improve certainty regarding a determination of looking-aside of an occupant such as a driver in a vehicle.SOLUTION: A looking-aside monitoring apparatus 10 comprises: an imaging device 36 capable of capturing a picked-up image including an occupant of a vehicle 1; and a determination unit 34 for determining looking-aside of the occupant based on the picked-up image. The determination unit 34 determines an area to which the occupant faces in the picked-up image from among a plurality of areas 41-48 dividing a visible range to which the occupant is possible to face, and determines looking-aside of the occupant through determination processing for each determined area.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an inattentive monitoring device for a vehicle. [Background technology]

[0002] Patent Documents 1 and 2 disclose inattentive monitoring devices for a vehicle that determine whether an occupant, such as a driver, is inattentive to the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-108784 A [Patent Document 2] JP 2020-077220 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when determining whether a passenger such as a driver is looking away, the determination is basically based on the direction of the passenger's face. However, the passenger's actual line of sight may differ from the direction of the passenger's face. For example, a passenger may look toward the inside of the vehicle while facing forward, which is the direction in which the vehicle is moving. Therefore, in order to increase the accuracy of determining whether an occupant such as a driver is looking away, it is possible to use, for example, the gaze direction in addition to the face direction, which is the direction of the face, in the determination. However, the gaze of an occupant is not necessarily captured in a captured image in a state that can be used for the determination. For example, if the occupant turns his face away from the imaging device, the image captured by the imaging device cannot properly include the eyeballs. In this case, if the determination of the looking away is made based on the face direction and the gaze direction, the gaze direction will be incorrect based on the eyeballs that are not properly captured, and as a result, the possibility of incorrectly determining the looking away increases.

[0005] Thus, in a vehicle, there is a demand for increasing the accuracy of judging whether an occupant such as a driver is looking away from the vehicle. [Means for solving the problem]

[0006] The vehicle inattentive monitoring device according to the present invention includes an imaging device capable of capturing an image including an occupant of a vehicle, and a determination unit that determines whether the occupant is inattentive based on the captured image, the determination unit comprising: Identifying the facial direction of the occupant and the gaze direction of the occupant's eyes based on the captured image; The area to which the occupant is facing in the captured image is determined from among a plurality of areas that divide the visible range to which the occupant may be facing, and the determined area Using at least one of the face direction and the gaze direction that have been specified according to Judging whether passengers are looking away It is , The visual range is divided into a plurality of areas in the pitch direction or the up-down direction of the vehicle, and in at least one of the areas, the occupant's looking away is judged using only the line of sight direction without using the face direction.

[0007] Preferably, the determination unit: In at least one of the areas obtained by dividing the visible range in the yaw direction or the left-right direction of the vehicle, the driver is prevented from looking away from the vehicle by using only the face direction without using the line of sight direction. It is good to judge.

[0008] Preferably, The determination unit is The visible range of Area in front of the vehicle in the direction of travel and before The front area is divided into a surrounding area and In at least one of the areas, the determined face direction and the determined line of sight direction are used to determine whether the occupant is looking away. , it is good. Effect of the Invention

[0014] In the present invention, the visual range into which a vehicle occupant may be directed is divided into a plurality of areas. A determination unit that determines whether an occupant is looking away based on an image captured by an imaging device capable of capturing an image including the vehicle occupant determines an area into which the occupant is looking away in the captured image from among the plurality of areas into which the visual range into which the occupant may be directed is divided, and determines whether the occupant is looking away by a determination process for each determined area. This makes it possible to determine whether the occupant is looking away by a determination process set for each area according to the relative positional relationship between each area and the occupant. It becomes unnecessary to determine whether the occupant is looking away by a uniform determination process for the entire visual range into which the occupant may be looking. In contrast, the captured image state of the occupant's face or eyes changes depending on the direction the occupant is facing. For this reason, if the determination unit performs a uniform inattentiveness determination process in the entire visible range in which the occupant may face, the accuracy of the inattentiveness determination for the direction in which the occupant is facing may decrease depending on the direction in which the occupant is facing. For example, if the inattentiveness determination process is a uniform determination process using both the face direction in which the face is facing and the gaze direction in which the eyes are facing, and the eyeballs are captured in a captured image in a state in which the gaze direction cannot be determined, the determination unit may erroneously determine inattentiveness because the gaze direction in which the eyeballs are facing can only be uncertainly identified in the captured image. In the present invention, the visibility range in which the occupant may be oriented is divided into multiple areas, and different determination processes can be executed for each area, thereby reducing the possibility of such erroneous determination of inattentive driving. The present invention is expected to increase the accuracy of the determination of inattentive driving by the occupant and suppress erroneous determination of inattentive driving. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic plan view of an automobile to which an inattentive monitoring device according to an embodiment of the present invention is applied. [Diagram 2] FIG. 2 is a schematic vertical cross-sectional view of the automobile of FIG. [Diagram 3] FIG. 3 is an explanatory diagram of a control system that functions as an inattentive monitoring device in the automobile of FIG. [Figure 4] FIG. 4 is a flowchart of passenger monitoring control by the CPU of the passenger monitoring device of FIG. [Diagram 5] FIG. 5 is a detailed flowchart of the inattentive driving determination in FIG. 4 according to this embodiment. [Figure 6] FIG. 6 is an explanatory diagram of an example of a plurality of areas that divide the driver's visible range in the vertical direction. [Figure 7] FIG. 7 is an explanatory diagram of an example of a plurality of areas dividing the driver's visual range in the left-right direction. [Figure 8] FIG. 8 is an explanatory diagram of an example of how to combine a plurality of areas according to FIGS. 6 and 7. [Figure 9A] FIG. 9A is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FR area. [Figure 9B] FIG. 9B is an explanatory diagram of the inattentive viewing determination process used when the face direction area is determined to be the FF area. [Figure 9C] FIG. 9C is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FC area. [Figure 9D] FIG. 9D is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FS area. [Figure 9E] FIG. 9E is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FL area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is a schematic plan view of an automobile 1 to which an inattentive monitoring device according to an embodiment of the present invention is applied. FIG. 2 is a schematic vertical cross-sectional view of the automobile 1 of FIG. The automobile 1 is an example of a vehicle.

[0018] The automobile 1 in Fig. 1 and Fig. 2 has a vehicle body 2. The vehicle body 2 is provided with a passenger compartment 3. The passenger compartment 3 is provided with multiple rows of seats 4 in which passengers such as the driver who ride in the automobile 1 are seated. A dashboard 5 extending in the left-right direction of the vehicle body 2 of the automobile 1 is provided in the front part of the passenger compartment 3, in front of the front row of seats 4. A steering wheel 7 and an in-vehicle camera 36 of an occupant monitoring device 11, which will be described later, are provided on the dashboard 5. A windshield 6 is provided above the dashboard 5 at the front of the passenger compartment 3. The in-vehicle camera 36 is provided in the center of the dashboard 5 in the left-right direction. The steering wheel 7 is provided on the left side of the dashboard 5 so as to be located in front of the seat 4 in which the driver sits. A rearview mirror 8 is provided in the center of the vehicle body 2 in the left-right direction. The driver controls the running of the automobile 1 by operating the steering wheel 7, accelerator pedal, brake pedal, shift lever, etc. of the automobile 1.

[0019] FIG. 3 is an explanatory diagram of a control system 10 that functions as an inattentive monitoring device in the automobile 1 of FIG. The control system 10 in FIG. 3 includes an occupant monitoring device 11, a UI device 12, a driving assistance device 13, a drive device 14, a braking device 15, a steering device 16, and an in-vehicle network 17 to which these are connected.

[0020] The in-vehicle network 17 may be a wired communication network for the automobile 1, for example, compliant with CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 17 may be a communication network such as a LAN, or a combination of these. A wireless communication network may be included as a part of the in-vehicle network 17. The various devices 11 to 16 connected to the in-vehicle network 17 transmit and receive data to and from each other through the in-vehicle network 17. The data transmitted and received through the in-vehicle network 17 may be encrypted, for example, and transmitted and received through the in-vehicle network 17 as packet data to which the ID of the transmission source device and the ID of the transmission destination device are added.

[0021] The drive device 14 has, for example, a power source and generates a driving force for accelerating the automobile 1. The drive device 14 may generate the driving force from an internal combustion engine that burns gasoline or ethanol, an electric motor that uses stored power, or the like, a power source that uses hydrogen, or a hybrid power source that combines these. The drive device 14 may generate a driving force according to the amount of operation of an accelerator pedal, for example.

[0022] The braking device 15 has, for example, a brake device, and generates a braking force for decelerating and stopping the automobile 1. The braking device 15 may generate a braking force according to the amount of operation of a brake pedal, for example.

[0023] The steering device 16 has, for example, a steering mechanism, and changes the traveling direction of the automobile 1. For example, the traveling direction of the automobile 1 may be changed by a steering amount corresponding to an operation amount of the steering wheel 7.

[0024] The driving support device 13 supports the driver's operation to control the driving of the automobile 1. The driving support device 13 may output a control command corresponding to the driver's operation amount to the driving device 14, the braking device 15, and the steering device 16, may output a control command adjusted from the driver's operation amount to the driving device 14, the braking device 15, and the steering device 16, or may output a control command to the driving device 14, the braking device 15, and the steering device 16 in an automatic driving state in which there is no driver's operation amount.

[0025] The UI device 12 is a user interface device for a passenger such as a driver, etc. To the UI device 12, user interface devices such as an operation device 21, a display device 22, a sound input device 23, and a sound output device 24 are connected. The display device 22 is, for example, a liquid crystal device. The display device 22 may be provided in the center of the dashboard 5 in the left-right direction, in front of the driver, or the like. The operation device 21 may be, for example, a touch panel provided over the display screen of a liquid crystal device, or various buttons and switches. The sound input device 23 is, for example, a microphone. The sound output device 24 is, for example, a speaker. Such a UI device 12 may output information acquired from the other devices 11, 13 to 16 of the control system 10 from a display device 22 or a sound output device . Furthermore, when an operation is performed on the operation device 21 or a voice instruction is given to the sound input device 23, the UI device 12 generates information corresponding to the operation or voice instruction and outputs the information to the other devices 11, 13 to 16 of the control system 10.

[0026] The occupant monitoring device 11 identifies and monitors occupants, including the driver, seated in the seat 4 in the passenger compartment 3 of the automobile 1. For safety, the driver and other occupants are basically required to pay attention to the surroundings of the traveling automobile 1. However, the driver and other occupants may look away from the direction of travel of the automobile 1, fall asleep, or suffer a sudden seizure, especially in an automatic driving system. The occupant monitoring device 11 monitors the occurrence of such situations and causes the control system 10, such as the driving assistance device 13, to execute a corresponding control. The occupant monitoring device 11 has an input / output unit 31, a timer 32, a memory 33, a CPU 34, a floodlight LED 35, an in-vehicle camera 36, ​​and a system bus 37 to which these are connected. Similarly to the occupant monitoring device 11, the other devices 12 to 16 of the control system 10 may also have an input / output section, a timer, a memory, a CPU, and a system bus to which these are connected.

[0027] 1 and 2, the in-vehicle camera 36 is provided in the left-right central portion of the dashboard 5 so as to capture an image of the rear side. The in-vehicle camera 36, ​​as an imaging device, is capable of capturing an image including an occupant such as the driver in the vehicle compartment 3.

[0028] The floodlight LED 35 irradiates, for example, infrared light to a driver or other occupant whose image is captured by the in-vehicle camera 36. The floodlight LED 35 may be provided next to the in-vehicle camera 36 in the center portion of the dashboard 5 in the left-right direction.

[0029] The input / output unit 31 is connected to the in-vehicle network 17. The input / output unit 31 inputs and outputs data between the in-vehicle network 17 and other devices in the control system 10.

[0030] The timer 32 measures the time and duration.

[0031] The memory 33 records the programs executed by the CPU 34 and various data. The memory 33 may be a non-volatile memory, a volatile memory, or a combination of these. Examples of non-volatile memory include semiconductor memories with data retention functions such as EEPROM and SSD, and hard disk drives. Examples of volatile memory include semiconductor memories such as RAM.

[0032] The CPU 34 reads and executes the programs recorded in the memory 33. In this way, the CPU 34 functions as a control unit of the occupant monitoring device 11. The CPU 34 serving as a control unit of the occupant monitoring device 11 controls the imaging of the in-vehicle camera 36 and the light projection of the light projecting LED 35 . The CPU 34 identifies and monitors occupants such as the driver in the vehicle compartment 3 based on images captured by the in-vehicle camera 36 and the like. For example, when the CPU 34 determines based on a captured image that the driver is looking away or falling asleep, it issues an alarm to the driver, and if the situation does not change after the alarm is issued, it instructs the driving support device 13 to execute control to decelerate and stop the automobile 1 on the shoulder of the road.

[0033] FIG. 4 is a flowchart of passenger monitoring control by the CPU 34 of the passenger monitoring device 11 of FIG. The CPU 34 of the occupant monitoring device 11 repeatedly executes the occupant monitoring control of FIG.

[0034] In step ST1, the CPU 34 determines whether a new image has been captured by the in-vehicle camera 36. The image capturing camera periodically captures images of the interior of the vehicle. If a new image has not been captured by the in-vehicle camera 36, ​​the CPU 34 repeats this process. If a new image has been captured by the in-vehicle camera 36, ​​the CPU 34 advances the process to step ST2.

[0035] In step ST2, the CPU 34 determines whether a passenger such as a driver is in the vehicle based on the new captured image. The CPU 34 may, for example, compare the previous captured image with the new captured image and determine whether a passenger such as a driver is in the vehicle by face pattern recognition or the like for the captured portion where a difference occurs. If a passenger such as a driver is not in the vehicle, the CPU 34 returns the process to step ST1. If a passenger such as a driver is in the vehicle, the CPU 34 advances the process to step ST3.

[0036] In step ST3, the CPU 34 determines whether the automobile 1 is moving. The CPU 34 may determine whether the automobile 1 is moving, for example, based on detection values ​​of a speed sensor, an acceleration sensor, and a GPS receiver (not shown) provided in the automobile 1. If the automobile 1 is not moving, the CPU 34 returns the process to step ST1. If the automobile 1 is moving, the CPU 34 advances the process to step ST4.

[0037] In step ST4, the CPU 34 judges whether the driver is looking away based on the new captured image. The CPU 34 may also judge whether passengers other than the driver are looking away.

[0038] In step ST5, the CPU 34 determines whether the driver is dozing off based on the new captured image. The CPU 34 may also determine whether passengers other than the driver are dozing off.

[0039] In step ST6, the CPU 34 judges an emergency state of the occupants including the driver based on the new captured image. The occupants including the driver may develop myocardial infarction, cerebral hemorrhage, etc. while the automobile 1 is traveling. When the captured image contains infrared components, the infrared image components contain information on the blood flow of each occupant. The CPU 34 may judge whether the pulse rate that can be determined from the blood flow is within a normal range. The CPU 34 may judge the emergency state by judging the driving posture of the driver in the captured image. For example, when the driver's head is lowered, the CPU 34 may judge the emergency state based on the captured image.

[0040] In step ST7, the CPU 34 judges whether or not an alarm is necessary. For example, if the CPU 34 judges that the driver is looking away or falling asleep, or that an emergency situation involving passengers including the driver is determined, the CPU 34 judges that an alarm is necessary and proceeds to step ST8. If such an emergency situation is not determined, the CPU 34 skips the process of step ST8 and proceeds to step ST9.

[0041] In step ST8, the CPU 34 outputs a warning to the occupants including the driver. The CPU 34 outputs the warning to, for example, the UI device 12. The UI device 12 outputs the warning to the display device 22 and the sound output device 24. This calls the attention of the occupants including the driver. The occupants including the driver can stop the driver from looking away or falling asleep and return to a state suitable for driving. In addition, the occupants including the driver can notice other occupants in an emergency and provide care for them.

[0042] In step ST9, the CPU 34 judges whether or not an emergency state exists. If an emergency state is judged in step ST6, and the driver does not return to a state suitable for driving even after the warning in step ST8, the CPU 34 judges that an emergency state exists and advances the process to step ST10. If no such emergency state exists, the CPU 34 skips the process of step ST10 and ends this control.

[0043] In step ST10, the CPU 34 executes emergency processing. The CPU 34 instructs, for example, the driving support device to make an emergency stop. The driving support device executes automatic driving control to decelerate the automobile 1 and stop it in the lane in which it is traveling or on the shoulder of the road. The CPU 34 also instructs, for example, a wireless device (not shown) to make an emergency call. The wireless device of the automobile 1 executes an emergency call to a server device of an emergency response organization via a base station, for example, by predetermined wireless communication. This allows an ambulance or the like to immediately rush to the scene where the automobile 1 is located.

[0044] Incidentally, when determining whether the driver or the like is looking away based on the captured image, the CPU 34 basically determines whether the driver is looking away based on the direction of the passenger's face. However, the passenger's actual line of sight may differ from the direction of the passenger's face. For example, the passenger may look left or right while facing forward, which is the direction in which the automobile 1 is moving. For this reason, in order to increase the accuracy of the determination of the inattentive behavior of an occupant such as a driver, it is considered desirable to use, for example, the gaze direction in addition to the face direction, which is the direction of the face, in the determination. However, the gaze of an occupant is not necessarily captured in a captured image so that it can be determined with a high degree of accuracy. For example, if an occupant turns his face in the opposite direction to the in-vehicle camera 36, ​​the captured image will not include an image of the eyeball. If the inattentive behavior determination process is performed based on the face direction and the gaze direction of the eyeball for a captured image that does not include an image of the eyeball, the inattentive behavior determination process cannot be performed correctly. If the inattentive behavior determination based on the gaze direction of the eyeball is forcibly performed, the gaze direction of the eyeball that is not captured will be erroneously determined, and as a result, there is a high possibility that the inattentive behavior determination will be erroneously determined. Thus, in the automobile 1, there is a demand for increasing the accuracy of judging whether a passenger such as a driver is looking away.

[0045] FIG. 5 is a detailed flowchart of the inattentive driving determination in FIG. 4 according to this embodiment. The CPU 34 of the occupant monitoring device 11 repeatedly executes the inattentive driving determination process shown in FIG. 5 during the occupant monitoring control shown in FIG.

[0046] In step ST21, the CPU 34 identifies the face direction (angle) based on the image of the driver's face included in the captured image. When the direction in which the driver's face is facing changes, the image of the driver's face included in the captured image also changes. The CPU 34 may identify the face direction (angle) by analyzing the state of the image of the driver's face included in the captured image. The CPU 34 may identify the angle of the face direction based on, for example, the direction in which the driver faces straight ahead of the automobile 1.

[0047] In step ST22, the CPU 34 identifies the gaze direction (angle) based on the image of the driver's eyeballs included in the captured image. When the driver's face direction or the direction in which the glasses are facing on the face changes, the image of the driver's eyeballs included in the captured image also changes. The CPU 34 may identify the gaze direction (angle) by analyzing the state of the image of the driver's eyeballs included in the captured image. The CPU 34 may identify the angle of the gaze direction based on, for example, the direction in which the driver faces straight ahead of the automobile 1.

[0048] In step ST23, the CPU 34 determines the area in the captured image in the direction of the driver's face from among a plurality of areas dividing the visible range into which the driver may turn while seated in the seat 4. The visible range into which the driver may turn while seated in the seat 4 is basically the range in front of the seat 4. The visible range may be divided, at a minimum, into a front area in the traveling direction of the automobile 1 into which the driver turns his / her gaze while the automobile 1 is traveling, and a peripheral area around the front area. In this case, the CPU 34 can determine the area in the direction of the driver's face by determining whether the face direction is within the range of the front area. In addition, in the area determination in step ST23, the CPU 34 may determine the area of ​​the gaze direction together with the area of ​​the face direction. In the area determination in step ST23, the CPU 34 may determine the area of ​​the combined direction of the face direction and the gaze direction.

[0049] In step ST24, the CPU 34 judges whether the face direction area in which the driver's face is facing is an area determined only by the face direction among the face direction and the line of sight direction. If the face direction area is not an area determined only by the face direction, the CPU 34 advances the process to step ST25. If the face direction area is an area determined only by the face direction, the CPU 34 advances the process to step ST26.

[0050] In step ST25, the CPU 34 judges whether the face direction area in which the driver's face is directed is an area determined only by the line of sight direction. If the face direction area is not an area determined only by the line of sight direction, the CPU 34 advances the process to step ST28. If the face direction area is an area determined only by the line of sight direction, the CPU 34 advances the process to step ST27.

[0051] In step ST26, the CPU 34 determines whether the driver is looking away based only on the direction of the driver's face. After that, the CPU 34 advances the process to step ST29.

[0052] In step ST27, the CPU 34 determines whether the driver is looking away based only on the direction of the driver's line of sight. After that, the CPU 34 advances the process to step ST29.

[0053] In step ST28, the CPU 34 judges whether the driver is looking away based on the driver's face direction and judges whether the driver is looking away based on the driver's line of sight direction. If the driver is looking away based on either of the two, the CPU 34 judges that the driver is looking away. If the driver is not looking away based on either of the two, the CPU 34 judges that the driver is not looking away. After that, the CPU 34 proceeds to step ST29. Through the processing of steps ST24 to ST28, the CPU 34 can determine whether the driver is looking away through the determination processing for each determined area. In the determination process for each area, CPU 34 does not execute a uniform determination process for all of the multiple areas into which the visible range is divided, but can execute different determination processes depending on the position in the visible range of the determined area.

[0054] In step ST29, the CPU 34 judges whether or not it is determined that the driver is looking away. If the driver is looking away, the CPU 34 advances the process to step ST30. If the driver is not looking away, the CPU 34 advances the process to step ST31.

[0055] In step ST30, the CPU 34 records inattentive information indicating that the driver is looking away in the memory 33.

[0056] In step ST31, the CPU 34 records in the memory 33 non-inattentive information indicating that the driver is not inattentive.

[0057] The CPU 34 repeatedly executes the process of Fig. 5 in step ST4 of Fig. 4. As a result, the memory 33 accumulates and records the determination result of the driver's inattentiveness based on the captured images that are captured repeatedly. Further, the CPU 34 can use the history of the driver's inattentiveness determination results stored in the memory 33 to determine whether or not a warning is required in step ST7 of FIG. For example, if the latest driver's inattentiveness determination result is inattentive information indicating that the driver is inattentive, CPU 34 can determine in step ST7 that an alarm is necessary and output an alarm to the occupants including the driver in step ST8. In addition, for example, if the results of a number of recent driver inattentiveness judgments continue to be inattentive information indicating that the driver is inattentive, CPU 34 can determine in step ST9 that an emergency state exists and execute emergency processing in step ST10.

[0058] Next, the multiple areas used in the determination of step ST23 will be described. In the above description, the visible range to which the driver may turn while seated in the seat 4 is divided into two areas: a front area in the traveling direction of the automobile 1 to which the driver turns his / her gaze while the automobile 1 is traveling, and a peripheral area around the front area. In the following, specific multiple areas that can be actually used will be described.

[0059] FIG. 6 is an explanatory diagram of an example of a plurality of areas that divide the driver's visible range in the vertical direction. 6 shows a plurality of areas that divide the driver's visual range in the vertical direction or the pitch direction of the automobile 1 on a schematic vertical cross-sectional view of the automobile 1 similar to that of FIG. 2. Here, the plurality of areas are an area U, an area F, an area B1, and an area B2.

[0060] The F area is a front area that the driver of the automobile 1 turns his / her head to view while driving in order to confirm the traveling direction of the automobile 1. The F area may be a range that overlaps with the windshield 6 of the automobile 1.

[0061] The U area is the area above the F area. Basically, the driver who drives the automobile 1 does not need to turn his / her face to look at the U area above the windshield 6 of the automobile 1 while driving the automobile 1.

[0062] Area B1 is a first lower area below Area F. Area B1 may be an area overlapping with the dashboard 5. A driver operating the automobile 1 may turn his / her face temporarily to look at the meter or display device 22 on the dashboard 5 to check the running state of the automobile 1 while driving.

[0063] The B2 area is a second lower area below the B1 area. The B2 area may be a range below the dashboard 5. Basically, the driver who drives the automobile 1 does not need to turn his / her face to view the B2 area, which is below the dashboard 5 of the automobile 1, while driving the automobile 1.

[0064] In this way, the visual range in front of seat 4, toward which the driver may turn while seated in seat 4, may be divided into four areas, from top to bottom: area U, area F, area B1, and area B2.

[0065] FIG. 7 is an explanatory diagram of an example of a plurality of areas dividing the driver's visual range in the left-right direction. 7 shows a plurality of areas that divide the driver's visual range in the left-right direction or the yaw direction of the automobile 1 on a schematic plan view of the automobile 1 similar to that of FIG 1. Here, the plurality of areas are an FR area, an FF area, an FC area, an FS area, and an FL area.

[0066] The FF area is a front area that the driver of the automobile 1 turns his / her head to view while driving in order to confirm the direction of travel of the automobile 1. The FF area may be an area that is approximately the right half of the windshield 6 of the automobile 1 on the steering wheel 7 side.

[0067] The FR area is the rightmost area on the right side of the FF area. Basically, the driver who drives the automobile 1 turns his / her face or gaze temporarily when checking the rear or right side of the automobile 1 in the door mirror 9 on the right side of the automobile 1.

[0068] The FC area is the central area on the left side of the FF area. The FC area may be an area that is the central portion of the windshield 6 of the automobile 1. The driver who drives the automobile 1 turns his / her face and gaze temporarily toward the center of the automobile 1 when checking the dashboard 5 or the rearview mirror 8.

[0069] The FS area is the area to the left of the FC area. The FS area may be an area that is approximately the left half of the windshield 6 of the automobile 1. When the driver operating the automobile 1 checks the left front side of the automobile 1, he or she turns his or her face or gaze temporarily to the left side of the automobile 1.

[0070] The FL area is the leftmost area to the left of the FS area. Basically, the driver who drives the automobile 1 turns his / her face or gaze temporarily when checking the rear or left side of the automobile 1 in the left door mirror 9 of the automobile 1.

[0071] In this way, the visual range in front of seat 4, to which the driver may turn while seated in seat 4, can be divided into five areas, from right to left: FR area, FF area, FC area, FS area, and FL area.

[0072] FIG. 8 is an explanatory diagram of an example of how to combine a plurality of areas according to FIGS.

[0073] 8, the F area, which is the second area from the top in the vertical direction in FIG. 6, is divided into five areas, FR area 42, FF area 43, FC area 44, FS area 45, and FL area 46, similarly to FIG. As a result, eight areas are shown in FIG. 8: U area 41, FR area 42, FF area 43, FC area 44, FS area 45, FL area 46, B1 area 47, and B2 area 48. Three areas, a FF area 43, an FC area 44, and an FS area 45, overlap on the windshield 6. The FR area 42 on the right side of the FF area 43 overlaps with the right door mirror 9. The FL area 46 on the left side of the FS area 45 overlaps with the left door mirror 9. The U area 41 is provided above the FR area 42, the FF area 43, the FC area 44, the FS area 45, and the FL area 46, and the B1 area 47 is provided below them. Below the B1 area 47, the B2 area 48 is provided. In this way, the visible range in FIG. 8 is divided into a plurality of areas in the pitch direction or up-down direction of the automobile 1, and is also divided into a plurality of areas in the yaw direction or left-right direction of the automobile 1.

[0074] Here, the U area 41 in FIG. 8 is an area where inattentiveness is judged based only on the face direction among the face direction and the line of sight direction of the driver. The FR area 42 is an area in which inattentive driving is judged based only on the direction of the driver's face. The FF area 43 is an area in which inattentive driving is judged based on both the face direction and line of sight direction of the driver. The FC area 44 is an area for determining whether the driver is looking away based only on the direction of the driver's face. The FS area 45 is an area in which inattentive driving is judged based on both the face direction and line of sight direction of the driver. The FL area 46 is an area in which inattentiveness of the driver is judged based only on the direction of the driver's face. The B1 area 47 is an area in which inattentiveness is judged based only on the driver's line of sight. The B2 area 48 is an area for determining whether the driver is looking away based only on the direction of the driver's face.

[0075] In this way, the visible range to which the driver may turn while seated in the seat 4 is divided into a plurality of areas 41-48. In step ST23 in FIG. 5, the CPU 34 determines a face direction area to which the driver's face is directed from among the areas 41 to 48 in FIG. Furthermore, in steps ST24 to ST25, CPU34 determines whether the face direction area determined to include the driver's face direction is an area in which inattentiveness is determined only by the face direction, an area in which inattentiveness is determined only by the gaze direction, or an area in which inattentiveness is determined only by the face direction and the gaze direction.

[0076] When the face direction area determined to include the face direction is the U area 41, the CPU 34 determines whether the driver is looking away based only on the face direction of the driver. As shown in Fig. 6, a driver who drives an automobile 1 basically directs his / her face and gaze within the range from the windshield 6 to the dashboard 5 of the automobile 1 while driving. The upper and lower visual confirmation areas viewed while driving are the non-looking-away angle range in which the driver is not looking away. When the driver's face or gaze is directed above or below the non-looking-away angle range, the driver is basically looking away. In this case, CPU 34 may determine whether the driver's face direction is included in the non-looking-away angle range, and determine whether the driver is looking away based only on the driver's face direction. When the face direction of the driver is included in the non-looking-away angle range, the CPU 34 may determine that the driver is not looking away. When the face direction of the driver is not included in the non-looking-away angle range, the CPU 34 may determine that the driver is looking away.

[0077] When the face direction area determined to include the face direction is the B1 area 47, the CPU 34 determines whether the driver is looking away based only on the line of sight of the driver. In this case, CPU 34 may determine whether the driver's line of sight is included in the non-avoidance angle range, and determine whether the driver is looking away based only on the line of sight. When the driver's line of sight direction is not included in the non-looking-away angle range, the CPU 34 may determine that the driver is looking away. When the driver's line of sight direction is included in the non-looking-away angle range, the CPU 34 may determine that the driver is not looking away. Even if the driver's line of sight is within the non-looking-away angle range, the CPU 34 may determine that the driver is looking away if the driver's line of sight is directed toward the dashboard 5 continuously for a predetermined period that can be measured by the timer 32. The CPU 34 may determine that the driver is looking away if the driver's line of sight is directed toward the dashboard 5 three or more times in a row, for example. Area B1 47, which determines whether the driver is looking away using only the line of sight without using the face direction, is provided as at least one of a plurality of areas that divide the visible range in the pitch direction or the up-down direction of automobile 1.

[0078] When the face direction area determined to include the face direction is the B2 area 48, the CPU 34 determines whether the driver is looking away based only on the face direction of the driver. In this case, CPU 34 may determine whether the driver's face direction is included in the non-looking-away angle range, and determine whether the driver is looking away based only on the driver's face direction. When the face direction of the driver is not included in the non-looking-away angle range, the CPU 34 may determine that the driver is looking away. When the face direction of the driver is included in the non-looking-away angle range, the CPU 34 may determine that the driver is not looking away.

[0079] FIG. 9 is an explanatory diagram of the inattentive driving determination process for each of a plurality of areas arranged in the left-right direction in area F in FIG. FIG. 9A is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FR area 42. FIG. FIG. 9B is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FF area 43. In FIG. FIG. 9C is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FC area 44. In FIG. FIG. 9D is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FS area 45. As shown in FIG. FIG. 9E is an explanatory diagram of the inattentiveness determination process used when the face direction area is determined to be the FL area 46. In FIG.

[0080] When the face direction area determined to include the face direction is the FR area 42, the CPU 34 determines whether the driver is looking away based only on the face direction of the driver. In this case, CPU34 determines whether the driver is looking away based only on the non-looking away angle range RF indicated by the solid line among the non-looking away angle range RF for the face direction and the non-looking away angle range RE for the line of sight shown in FIG. 9A. The CPU 34 may determine whether the driver is looking away by determining, for example, whether the direction of the driver's face is included in the non-looking away angle range RF in FIG. 9A. When the face direction of the driver is not included in the non-looking-aside angle range RF of FIG. 9A, the CPU 34 may determine that the driver is looking aside. When the face direction of the driver is included in the non-looking-aside angle range RF of FIG. 9A, the CPU 34 may determine that the driver is not looking aside. The FR area 42, which determines whether the driver is looking away using only the face direction without using the line of sight direction, is provided as at least one of a plurality of areas 42-46 that divide the visible range in the yaw direction or left-right direction of the automobile 1.

[0081] When the face direction area determined as including the face direction is the FF area 43, the CPU 34 determines whether the driver is looking away based on both the face direction and line of sight direction of the driver. In this case, CPU 34 determines whether the driver is looking away by using each of non-looking away angle range RF for the face direction and non-looking away angle range RE for the line of sight direction shown in FIG. 9B. The CPU 34 may determine whether the driver is looking away, for example, by determining whether the direction of the driver's face is within the non-looking away angle range RF of FIG. 9B and whether the direction of the driver's line of sight is within the non-looking away angle range RE of FIG. 9B. When the driver's face direction is included in the non-looking aside angle range RF in FIG. 9B and the line of sight direction is included in the non-looking aside angle range RE in FIG. 9B, CPU 34 may determine that the driver is not looking aside. Otherwise, CPU 34 may determine that the driver is distracted.

[0082] When the face direction area determined as including the face direction is the FC area 44, the CPU 34 determines whether the driver is looking away based only on the face direction of the driver. In this case, CPU34 determines whether the driver is looking away based only on the non-looking away angle range RF indicated by the solid line among the non-looking away angle range RF for the face direction and the non-looking away angle range RE for the line of sight shown in FIG. 9C. The CPU 34 may determine whether the driver is looking away by determining, for example, whether the direction of the driver's face is included in the non-looking away angle range RF in FIG. 9C. When the face direction of the driver is not included in the non-looking-aside angle range RF of FIG. 9C, the CPU 34 may determine that the driver is looking aside. When the face direction of the driver is included in the non-looking-aside angle range RF of FIG. 9C, the CPU 34 may determine that the driver is not looking aside. An FC area 44 for determining whether the driver is looking away using only the face direction without using the line of sight direction is provided as at least one of a plurality of areas 42 to 46 that divide the visible range in the yaw direction or left-right direction of the automobile 1.

[0083] When the face direction area determined as including the face direction is the FS area 45, the CPU 34 determines whether the driver is looking away based on both the face direction and line of sight direction of the driver. In this case, the CPU 34 determines whether the driver is looking away by using each of the non-looking away angle range RF for the face direction and the non-looking away angle range RE for the line of sight direction shown in FIG. 9D. The non-looking aside angle range RF for the face direction shown in FIG. 9D is set to the left as compared with the non-looking aside angle range RF for the face direction shown in FIGS. 9A to 9C. The non-looking aside angle range RE for the line of sight direction shown in FIG. 9D is set to the left as compared with the non-looking aside angle range RE for the line of sight direction shown in FIGS. 9A to 9C. The CPU 34 may determine whether the driver is looking away, for example, by determining whether the direction of the driver's face is within the non-looking away angle range RF of FIG. 9D and whether the direction of the driver's line of sight is within the non-looking away angle range RE of FIG. 9D. When the driver's face direction is included in the non-looking aside angle range RF in FIG. 9D and the line of sight direction is included in the non-looking aside angle range RE in FIG. 9D, CPU 34 may determine that the driver is not looking aside. Otherwise, CPU 34 may determine that the driver is distracted.

[0084] When the face direction area determined to include the face direction is the FL area 46, the CPU 34 determines whether the driver is looking away based only on the face direction of the driver. In this case, CPU34 determines whether the driver is looking away based only on the non-looking away angle range RF indicated by the solid line among the non-looking away angle range RF for the face direction and the non-looking away angle range RE for the line of sight shown in FIG. 9E. The CPU 34 may determine whether the driver is looking away by determining, for example, whether the face direction of the driver is included in the non-looking away angle range RF of FIG. 9E. When the face direction of the driver is not included in the non-looking-aside angle range RF of FIG. 9E, the CPU 34 may determine that the driver is looking aside. When the face direction of the driver is included in the non-looking-aside angle range RF of FIG. 9E, the CPU 34 may determine that the driver is not looking aside. An FL area 46 for determining whether the driver is looking away using only the face direction without using the line of sight direction is provided as at least one of a plurality of areas 42 to 46 that divide the visible range in the yaw direction or left-right direction of the automobile 1.

[0085] In this way, CPU 34, as a determination unit, determines whether the driver is looking away using both the identified face direction and gaze direction in some (43, 45) of the multiple areas 41-48 that divide the visible range. Also, CPU 34 determines whether the driver is looking away using only the gaze direction without using the face direction in the identified face direction and gaze direction in the remaining part (47) of the multiple areas 41-48 that divide the visible range. Also, CPU 34 determines whether the driver is looking away using only the face direction without using the gaze direction in the identified face direction and gaze direction in the remaining parts (41, 42, 44, 46, 48) of the multiple areas 41-48 that divide the visible range.

[0086] As described above, in this embodiment, the visual range to which the driver of the automobile 1 may turn is divided and set into a plurality of areas 41-48. Then, the CPU 34, which serves as a determination unit for determining whether the driver is looking away based on an image captured by an imaging device capable of capturing an image including the driver of the automobile 1, determines a face direction area to which the driver is looking in the captured image from among the plurality of areas 41-48 that divide the visual range to which the driver may turn, and determines whether the driver is looking away by a determination process for each determined area. This makes it possible to determine whether the driver is looking away by a determination process set for each area according to the relative positional relationship between each area and the driver. It is no longer necessary to determine whether the driver is looking away by a uniform determination process for the entire visual range to which the driver may turn. On the other hand, the imaging state of the driver's face and eyes changes depending on the direction the driver is facing. For this reason, if the CPU 34 performs a uniform inattentiveness determination process in the entire visible range in which the driver may face, the accuracy of the inattentiveness determination for the direction the driver is facing may decrease depending on the direction the driver is facing. For example, if the inattentiveness determination process is a uniform determination process using both the face direction and the gaze direction of the eyes, and the eyes are captured in a captured image in a state in which their gaze direction cannot be determined, the CPU 34 may erroneously determine inattentiveness because the gaze direction of the eyes can only be uncertainly identified in the captured image. In this embodiment, the visible range to which the driver may be directed is divided into a plurality of areas 41 to 48, and different determination processes can be executed for each area, so that it is possible to reduce the possibility of such an erroneous determination of inattentiveness. As a result, in this embodiment, it is difficult to erroneously determine whether the driver is inattentive. In this embodiment, it is possible to increase the certainty of the determination of whether the driver is inattentive.

[0087] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.

[0088] For example, in the embodiment described above, in the control system 10 of the automobile 1, the CPU 34 of the passenger monitoring device 11 executes all the controls shown in FIG. 4 and FIG. In the control system 10 of the automobile 1, the other devices 12 to 16 also have CPUs. The multiple CPUs constituting the control system 10 may execute part of the control shown in Figures 4 to 5. The multiple CPUs provided in the control system 10 may cooperate to execute the control shown in Figures 4 to 5 in a distributed manner.

[0089] In the above-described embodiment, the CPU 34 of the occupant monitoring device 11 determines the face direction area for each of the areas 41 to 48 in FIG. 8 that divide the visible range. Alternatively, for example, CPU 34 may determine face direction areas for a plurality of areas in FIG. 6 that divide the visible range, or for a plurality of areas in FIG. 7 that divide the visible range. Furthermore, the CPU 34 may determine the face direction area for the multiple areas in FIG. 6 into which the visible range is divided, or for the multiple areas in FIG. 7 into which the visible range is divided. Furthermore, the CPU 34 may determine the face direction area for two areas: the front area and the surrounding area around it.

[0090] In the above-described embodiment, the CPU 34 of the occupant monitoring device 11 determines whether the driver in the automobile 1 is looking away. In addition, for example, the CPU 34 of the occupant monitoring device 11 may also determine whether occupants other than the driver in the automobile 1 are looking away. (Example of the aspect disclosed in the present embodiment) In this embodiment, for example, the following contents are disclosed. The vehicle inattentive monitoring device of this embodiment has an imaging device capable of capturing an image including the vehicle occupants, and a judgment unit that judges whether the occupant is looking away based on the captured image. The judgment unit judges an area in the captured image to which the occupant is facing from among a number of areas that divide the visible range to which the occupant may be facing, and judges whether the occupant is looking away by performing judgment processing for each determined area. Preferably, the determination unit identifies the facial direction in which the occupant's face is facing from the captured image, and determines the area in which the identified facial direction is facing from among a plurality of areas that divide the visible range into which the occupant may be facing. Preferably, the visible range is divided into at least a front area in the traveling direction of the vehicle and a peripheral area around the front area. Preferably, the visual range is divided into a plurality of areas at least in the pitch direction or up-down direction of the vehicle, or at least in the yaw direction or left-right direction of the vehicle. Preferably, the determination unit identifies the facial direction in which the occupant's face is facing as well as the gaze direction in which the eyes are facing based on the captured image, and in the determination process for each area, performs different determination processes depending on the position within the visible range for the determined area. Preferably, in some of the multiple areas dividing the visible range, the judgment unit judges whether the occupant is looking away using both the specified face direction and gaze direction, in some of the remaining areas dividing the visible range, the judgment unit judges whether the occupant is looking away using only the gaze direction without using the face direction between the specified face direction and gaze direction, and in the remaining areas dividing the visible range, the judgment unit judges whether the occupant is looking away using only the face direction without using the gaze direction between the specified face direction and gaze direction. Preferably, the area in which the occupant's looking away is determined using only the face direction without using the line of sight direction is provided as at least one of a plurality of areas that divide the visible range in the yaw direction or left-right direction of the vehicle. Preferably, the area in which the occupant's looking away is determined using only the line of sight direction without using the face direction is provided as at least one of a plurality of areas that divide the visible range in the pitch direction or vertical direction of the vehicle. [Explanation of symbols]

[0091] 1...Automobile (vehicle), 2...Vehicle body, 3...Vehicle interior, 4...Seat, 5...Dashboard, 6...Windshield, 7...Steering wheel, 8...Rearview mirror, 9...Door mirror, 10...Control system (distraction monitoring device), 11...Occupant monitoring device, 12...UI device, 13...Driving assistance device, 14...Drive system, 15...Braking system, 16...Steering system, 17...In-vehicle network, 21...Operation device, 22...Display device, 23...Sound input device, 24...Sound output device, 31...Input / output unit, 32...Timer, 33...Memory, 34...CPU (determination unit), 35...Floodlight LED, 36...In-vehicle camera (imaging device), 37...System bus, 41...U area, 42...FR area, 43...FF area, 44...FC area, 45...FS area, 46...FL area, 47...B1 area, 48...B2 area

Claims

1. An imaging device capable of capturing an image including an occupant of a vehicle; a determination unit for determining whether an occupant is looking away based on the captured image; having The determination unit is Identifying the facial direction of the occupant and the gaze direction of the occupant's eyes based on the captured image; an area in the captured image to which the occupant is facing is determined from among a plurality of areas that divide a visual recognition range to which the occupant may be facing, and whether the occupant is looking away is determined using at least one of the identified face direction and the identified line of sight direction according to the determined area, In at least one of the areas obtained by dividing the visible range in a pitch direction or a vertical direction of the vehicle, the inattentiveness of the occupant is determined using only the line of sight direction without using the face direction. Vehicle distraction monitoring device.

2. The determination unit is determining whether the occupant is looking away using only the face direction without using the line of sight direction in at least one of a plurality of areas obtained by dividing the visible range in a yaw direction or a left-right direction of the vehicle; 2. A vehicle inattentive monitoring device according to claim 1.

3. The determination unit: determining whether an occupant is looking away using both the identified face direction and the identified line of sight direction in at least one of areas obtained by dividing the visible range into a front area in a traveling direction of the vehicle and a peripheral area around the front area; 3. A vehicle inattentive monitoring device according to claim 1 or 2.

Images