Occupant monitoring device, vehicle control system, occupant monitoring method, and program

Abstract

An occupant monitoring device capable of reducing unnecessary restrictions on a driving assistance function is provided.　An occupant monitoring device (2) comprises: an image acquisition unit (10) that acquires, from an image capturing device (8) that is mounted on a vehicle (1) and captures an image of the inside of the vehicle (1), a captured image in which an occupant in the vehicle (1) is captured; an element detection unit (11) that uses the captured image acquired by the image acquisition unit (10) to detect a facial or bodily element of the occupant; an occupant state detection unit (12) that uses the element detected by the element detection unit (11) to detect an inattentive state indicating a state in which the occupant is in a distracted state or in a state with lowered arousal level; an operation monitoring unit (13) that monitors whether or not the occupant state detection unit (12) is able to detect the element; and an execution feasibility determination unit (15) that uses the result from the operation monitoring unit (13) and determines the execution feasibility of a driving assistance function by an operation control device (21) that executes braking / driving control of the vehicle (1).

Description

Occupant monitoring device, vehicle control system, occupant monitoring method, and program

[0001] The present disclosure relates to an occupant monitoring device, a vehicle control system, an occupant monitoring method, and a program.

[0002] A driving assistance device is disclosed that is provided in a vehicle and includes a plurality of monitoring devices that monitor the driver's condition, a device inspection unit that inspects whether the plurality of monitoring devices are normal or abnormal, and an assistance level determination unit that determines the degree of assistance (whether or not it can be used) of a driving assistance function that assists the driver's driving operation based on the inspection results by the device inspection unit (for example, Patent Document 1).

[0003] International Publication No. 2018 / 011872

[0004] However, in a device that monitors a driver, various functions are correlated with each other to detect a driver's inattention. In the above-described conventional technology, whether or not a device that monitors a driver is operating normally is determined for each device that monitors the driver. On the other hand, even if a device is not operating normally due to a malfunction or the like, it may not be necessary to immediately disable a driving assistance function if, for example, eyes can be detected. Even in cases where it is not necessary to immediately disable a driving assistance function, there is a problem in that the driving assistance function may be disabled, that is, the driving assistance function may be unnecessarily disabled.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an occupant monitoring device and a vehicle control system that reduce unnecessary suppression of driving assistance functions.

[0006] The occupant monitoring device according to the present disclosure includes an image acquisition unit that acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle that images the interior of the vehicle, an element detection unit that detects elements of the occupant's face or body using the image acquired by the image acquisition unit, an occupant state detection unit that uses the elements detected by the element detection unit to detect an inattentive state indicating that the occupant is in a distracted state or has a reduced level of alertness, an operation monitoring unit that monitors whether the occupant state detection unit has detected the elements, and an execution feasibility determination unit that uses the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that performs braking and driving control of the vehicle.

[0007] In addition, the vehicle control system according to the present disclosure includes the above-mentioned occupant monitoring device and a driving control device that controls the driving of the vehicle, and when the occupant monitoring device determines that it is necessary to suppress the driving assistance function, it outputs a signal to the driving control device indicating that the driving assistance function cannot be executed by the driving control device based on the detection result of the occupant state detection unit, and when the driving control device acquires the signal indicating that the driving assistance function cannot be executed, it outputs a signal to the steering mechanism or braking / driving mechanism installed in the vehicle to cancel the suppression of the driving assistance function by the driving control device.

[0008] In addition, the occupant monitoring method according to the present disclosure includes the steps of: an image acquisition unit acquiring an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing images of the interior of the vehicle; an element detection unit detecting elements of the occupant's face or body using the image acquired by the image acquisition unit; an occupant state detection unit detecting an inattentive state indicating that the occupant is in a distracted state or has a reduced level of alertness using the elements detected by the element detection unit; an operation monitoring unit monitoring whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit using the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that performs braking and driving control of the vehicle.

[0009] In addition, the program of the present disclosure causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle that captures images of the interior of the vehicle; an element detection unit detects elements of the occupant's face or body using the image acquired by the image acquisition unit; an occupant state detection unit uses the elements detected by the element detection unit to detect an inattentive state indicating that the occupant is in a state of absent-mindedness or a state of reduced alertness; an operation monitoring unit monitors whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit uses the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that performs braking and driving control of the vehicle.

[0010] According to the occupant monitoring device of the present disclosure, unnecessary suppression of driving assistance functions can be reduced. Furthermore, according to the vehicle control system of the present disclosure, unnecessary suppression of driving assistance functions can be reduced. Furthermore, according to the occupant monitoring method of the present disclosure, unnecessary suppression of driving assistance functions can be reduced. Furthermore, according to the program of the present disclosure, unnecessary suppression of driving assistance functions can be reduced.

[0011] 1 is a block diagram showing a portion of a vehicle equipped with an occupant monitoring device according to a first embodiment of the present disclosure. 2 is a block diagram showing a vehicle control system equipped with an occupant monitoring device according to a first embodiment of the present disclosure. 3 is a block diagram showing an occupant monitoring device according to a first embodiment of the present disclosure. 4 is a diagram for explaining a plurality of posture imbalance types. 5 is a flowchart showing an example of an operation of the occupant monitoring device according to the first embodiment of the present disclosure. 6 is a diagram showing an example of a hardware configuration of the occupant monitoring device according to the first embodiment of the present disclosure. 7 is a diagram showing an example of a hardware configuration of the occupant monitoring device according to the first embodiment of the present disclosure. 8 is a block diagram showing an occupant monitoring device according to a second embodiment of the present disclosure. 9 is a flowchart showing an example of an operation of the occupant monitoring device according to the second embodiment of the present disclosure. 10 is a block diagram showing an occupant monitoring device according to a third embodiment of the present disclosure. 11 is a flowchart showing an example of an operation of the occupant monitoring device according to the third embodiment of the present disclosure. 12 is a block diagram showing an occupant monitoring device according to a fourth embodiment of the present disclosure. 13 is a flowchart showing an example of an operation of the occupant monitoring device according to the fourth embodiment of the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the drawings are schematic, and the relative sizes and positions shown in different drawings are not necessarily limited to those described and may be changed as appropriate. In the following description, similar components are denoted by the same reference numerals, and their names and functions are assumed to be the same or similar. Therefore, detailed descriptions thereof may be omitted.

[0013] Embodiment 1. Figure 1 is a block diagram showing a portion of a vehicle 1 equipped with an occupant monitoring device 2 according to this embodiment. The vehicle 1 is equipped with a vehicle control system 3. The occupant monitoring device 2 constitutes a portion of the vehicle control system 3. The vehicle control system 3 includes an occupant monitoring system 4, a driving control system 5, a braking / driving mechanism 6, and a steering mechanism 7. The occupant monitoring system 4 includes the occupant monitoring device 2, an imaging device 8, and an output device 9. The occupant monitoring device 2 constitutes a portion of the occupant monitoring system 4.

[0014] The imaging device 8 is mounted on the vehicle 1. The imaging device 8 is a camera provided on a movable part inside the vehicle 1, and captures images of the interior of the vehicle 1. The imaging device 8 captures an image of an imaging area that spreads from the installation position at a preset camera angle of view. The imaging device 8 is installed so that the imaging area includes the faces of the occupants inside the vehicle 1, who are the imaging targets. The imaging device 8 captures images of the occupants inside the vehicle 1 at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured images to the occupant monitoring device 2. The captured images may be still images, moving images, or both still images and moving images.

[0015] The output device 9 includes at least one of a display, a speaker, and a vibrator. The display is configured, for example, by a liquid crystal display, an organic EL (Electro-Luminescence) display, or a HUD (Head-Up Display). The display is provided, for example, on the dashboard of the vehicle 1. The speaker is provided, for example, on the dashboard of the vehicle 1. The vibrator is provided, for example, on the steering wheel of the vehicle 1 or on the driver's seat of the vehicle 1.

[0016] The occupant monitoring device 2 monitors the state of the occupant, and when the occupant is in an inattentive state indicating a distracted state or a low level of alertness, outputs a command to the output device 9 to notify the driver that the driving assistance function has been suppressed. Note that the output device 9 may also be configured to output a command to notify the driver in advance of the suppression of the driving assistance function.

[0017] As shown in Figure 1, the occupant monitoring device 2 includes an image acquisition unit 10, an element detection unit 11, an occupant state detection unit 12, an operation monitoring unit 13, a memory unit 14, an execution feasibility determination unit 15, and a command unit 16, each of which is connected to a communication bus 17, and data can be sent and received via the communication bus 17.

[0018] The image acquisition unit 10 acquires an image of an occupant in the vehicle from the imaging device 8 and outputs the acquired image to the element detection unit 11 .

[0019] The element detection unit 11 detects elements of the face or body of the occupant. The element detection unit 11 outputs the detected elements of the face or body of the occupant and the captured image acquired from the image acquisition unit 10 to the occupant state detection unit 12.

[0020] Next, we will explain the configuration of the driving control system 5 that provides driving assistance using the detection results of the occupant monitoring system 4. Figure 2 is a block diagram showing the vehicle control system 3 of the vehicle 1. As shown in Figure 2, the driving control system 5 is configured to be able to communicate with the occupant monitoring system 4 wirelessly or via a wire.

[0021] The driving control system 5 includes a driving control device 21 that controls the steering mechanism 7 and braking / driving mechanism 6 of the vehicle 1 by outputting control signals based on the detection results of the occupant monitoring system 4. The driving control system 5 also includes a map information storage device (not shown) that stores map information, a surrounding condition monitoring device 22 that monitors the surrounding conditions of the vehicle, and a vehicle state acquisition device 23 that acquires information indicating the state of the vehicle 1. The occupant monitoring system 4, the driving control device 21, the surrounding condition monitoring device 22, and the vehicle state acquisition device 23 included in the driving control system 5 are each connected to a communication bus 17, and are able to send and receive data via the communication bus 17.

[0022] The braking / driving mechanism 6 is a mechanism for controlling the traveling speed of the vehicle 1 and for switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 61. The steering mechanism 7 is a mechanism for determining the traveling direction of the vehicle 1, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 71. The braking / driving actuator 61 that controls the braking / driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, and the steering actuator 71 that controls the steering mechanism 7 is composed of, for example, an EPS (Electric Power Steering) motor, and the like.

[0023] The surrounding condition monitoring device 22 monitors the conditions around the vehicle 1 and includes a GPS (Global Positioning System) receiver 221, an on-board communication device 222, an external sensor 223, and a navigation system 224. The GPS receiver 221 receives signals transmitted from GPS positioning satellites to detect the current location of the vehicle 1. The on-board communication device 222 communicates with communication devices of other vehicles or communication devices installed on the road to acquire information on the positions of other vehicles or pedestrians, traffic information, etc. Here, traffic information includes, for example, congestion information, traffic regulation information, construction zone information, etc. The on-board communication device 222 can also perform vehicle-to-vehicle communication via wireless communication with on-board communication devices of other vehicles located around the vehicle 1. In addition, the on-board communication device 222 may perform mobile communication with a base station external to the vehicle 1. The external sensor 223 is, for example, composed of at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and detects the positions of vehicles, pedestrians, obstacles, etc. present around the vehicle 1 or their distance from the vehicle 1. The navigation system 224 calculates a route from the current position of the vehicle 1 to a destination and provides guidance along the calculated route. The navigation system 224 has a display unit composed of, for example, a liquid crystal display, and is housed in the instrument panel. The navigation system 224 also has an operation unit such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. Note that the operation unit of the navigation system 224 may be composed of a microphone or the like to be able to accept operations via voice uttered by the occupant.

[0024] The vehicle state acquisition device 23 acquires information indicating the state of the vehicle 1 and includes a steering angle sensor 231, a vehicle speed sensor 232, a steering torque sensor 233, an accelerator position sensor 234, and a brake position sensor 235. The steering angle sensor 231 is provided, for example, on an EPS (Electric Power Steering) motor or a steering wheel and detects the steering angle of the vehicle 1. The vehicle speed sensor 232 is provided, for example, on a wheel and detects the traveling speed of the vehicle 1. The steering torque sensor 233 is provided, for example, on the steering wheel and detects the magnitude of the steering wheel operation force applied by the driver. The accelerator position sensor 234 detects the amount of accelerator pedal depression by the driver. The brake position sensor 235 detects the amount of brake pedal depression by the driver.

[0025] The driving control device 21 of the driving control system 5 controls the driving of the vehicle 1. The driving control device 21 controls a braking / driving actuator 61 or a steering actuator 71 mounted on the vehicle 1 to assist the driver in driving. The driving control device 21 performs automatic driving control of the vehicle 1, for example, by outputting a signal to the braking / driving actuator 61 or the steering actuator 71. The driving control device 21 performs braking / driving control of the vehicle 1, such as by controlling the braking / driving actuator 61, which is composed of an electronically controlled throttle and a brake actuator, to activate the brakes to decelerate or stop the vehicle 1. The driving control device 21 performs steering control of the vehicle 1, such as by controlling the steering actuator 71, which is composed of an EPS motor, to maintain the lane in which the vehicle 1 is traveling. Furthermore, the driving control device 21 executes multiple driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function is referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function is referred to as LKA (Lane Keeping Assist).

[0026] When the driving control device 21 executes ACC, it controls the traveling speed of the vehicle 1 by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device 22. If no vehicle ahead is detected, the driving control device 21 causes the vehicle 1 to travel at a constant speed at a target speed set in advance by the driver or the like. On the other hand, if a vehicle ahead is detected, the ACC causes the vehicle 1 to follow the vehicle ahead while maintaining a distance from the vehicle ahead.

[0027] Furthermore, when the driving control device 21 executes LKA, the driving control device 21 controls the steering force and steering holding force based on shape information of the lane markings in the traveling direction acquired from the surrounding conditions monitoring device 22. LKA causes the vehicle 1 to travel along the lane by applying a steering force to the steering wheel in a direction that prevents the vehicle from approaching the lane markings. Note that road information output by the surrounding conditions monitoring device 22 may be used for vehicle control by ACC and LKA.

[0028] Furthermore, when the driving control device 21 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the vehicle 1. When the automatic evacuation control is started, the driving control device 21 causes the surrounding condition monitoring device 22 to search for an evacuation location where the vehicle 1 should be stopped. Then, the driving control device 21 moves the vehicle 1 to the evacuation location set by the surrounding condition monitoring device 22 and stops the vehicle 1 there. Note that, on expressways, the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle 1 is traveling, and on ordinary roads, it may be a location that avoids a shoulder outside the lane in which the vehicle 1 is traveling, as well as intersections, railroad crossings, sidewalks, and locations where there is a high possibility of the presence of moving objects such as other vehicles, trains, and pedestrians.

[0029] 3 is a block diagram showing the occupant monitoring device 2. The image acquisition unit 10 outputs the captured image acquired from the imaging device 8 to the element detection unit 11.

[0030] The element detection unit 11 acquires a captured image from the image acquisition unit 10. The element detection unit 11 uses the captured image to detect elements of the occupant's face or body. The element detection unit 11 can detect the position and size of elements of the occupant's face or body. The elements of the occupant's face or body include, for example, facial features and head orientation. The facial features include, for example, the eyes, nose, and mouth.

[0031] The element detection unit 11 may detect elements of the occupant's face or body using a known image recognition technique on the captured image. For example, the element detection unit 11 detects position information of the outer corners and inner corners of the eyes, upper eyelids, and lower eyelids as information about the eyes. For example, the element detection unit 11 detects position information of the root, tip, back, and wings of the nose as information about the nose. For example, the element detection unit 11 detects position information of the upper lip, lower lip, and corners of the mouth as information about the mouth.

[0032] The element detection unit 11 detects the head position of the occupant in real space based on the occupant's face in the captured image. The head position of the occupant in the captured image is indicated, for example, by the center of the occupant's eyebrows. The element detection unit 11 detects, for example, a point in real space corresponding to the center of the occupant's eyebrows in the captured image as the occupant's head position. Note that this is merely an example, and the position of the occupant's head in the captured image may be indicated, for example, by the center of the occupant's facial area or the center of a line connecting the inner corners of the occupant's eyes. In this case, the element detection unit 11 detects, for example, a point in real space corresponding to the center of the occupant's facial area in the captured image or the center of a line connecting the inner corners of the occupant's eyes as the occupant's head position in real space. The occupant's facial area is, for example, the smallest rectangle that surrounds the outline of the occupant's face. The occupant's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle in the captured image.

[0033] The element detection unit 11 may detect the head position of the occupant by using, for example, a known coordinate transformation technique for transforming points on the captured image into points in real space. The head position of the occupant is expressed, for example, by coordinates in real space. The information on the head position of the occupant includes coordinate information of the head position of the occupant.

[0034] The element detection unit 11 outputs the detected information about the occupant's eyes, nose, mouth, and head position (hereinafter referred to as "head position information") to the occupant state detection unit 12. The element detection unit 11 outputs the detected information about the occupant's eyes, nose, mouth, and head position information to the eye opening degree determination unit 121, gaze detection unit 122, facial direction detection unit 123, drowsiness detection unit 124, and posture determination unit 128 of the occupant state detection unit 12. In addition, the element detection unit 11 outputs the captured image acquired from the image acquisition unit 10 to the occupant state detection unit 12.

[0035] The element detection unit 11 can detect the position and size of the occupant's face or body elements using, for example, the method described above. The element detection unit 11 outputs the occupant's face or body elements to the movement monitoring unit 13. The element detection unit 11 outputs the detected information about the occupant's eyes, nose, mouth, and head position information to the movement monitoring unit 13.

[0036] The occupant state detection unit 12 detects the occupant's distracted state or alert state using the captured image acquired by the image acquisition unit 10. That is, the occupant state detection unit 12 detects the occupant's inattentive state, which indicates that the occupant is in a distracted state or has a low level of alertness, using the captured image acquired by the image acquisition unit 10.

[0037] The occupant state detection unit 12 acquires information about the occupant's eyes, nose, mouth, and head position from the element detection unit 11. Note that the occupant state detection unit 12 may acquire the captured image directly from the image acquisition unit 10 without going through the element detection unit 11, and detect the state of the occupant.

[0038] The occupant state detection unit 12 includes an eye opening degree determination unit 121 , a gaze detection unit 122 , a face direction detection unit 123 , a drowsiness detection unit 124 , an eye state determination unit 125 , an inattentive driving determination unit 126 , a dozing determination unit 127 , and a posture determination unit 128 .

[0039] The eye-opening degree determination unit 121 determines the eye-opening degree of the occupant based on the captured image acquired by the image acquisition unit 10. Specifically, the eye-opening degree determination unit 121 determines the eye-opening degree using the occupant's upper eyelid and lower eyelid detected by the element detection unit 11. For example, an area in the captured image where the occupant's face may be present (hereinafter referred to as a "face detection area") is set in advance, and the eye-opening degree determination unit 121 performs known edge detection on the face detection area in the captured image to extract feature points indicating the occupant's upper eyelid and lower eyelid. Next, the eye-opening degree determination unit 121 calculates, for example, the vertical distance between the occupant's upper eyelid and lower eyelid in the captured image (hereinafter referred to as "eyelid distance"), starting from left to right in the captured image, and calculates the maximum eyelid distance. Then, the eye-opening degree determination unit 121 determines the eye-opening degree based on the ratio between the calculated maximum eyelid distance and a reference value for the eye-opening degree. The reference value of the eye openness is set to, for example, the average value of the eye openness of an adult of average build who is awake. The reference value of the eye openness may be, for example, the average value of the eye openness of the occupant for a predetermined time after the occupant gets into the vehicle, or the average value of all the eye openness determined by the eye openness determination unit up to now. The above-described method of determining the eye openness is merely an example. The eye openness determination unit 121 may determine the eye openness using various known algorithms. The eye openness determination unit 121 outputs information related to the determination result of the eye openness of the occupant (hereinafter referred to as "eye openness information") to the eye state determination unit 125. The eye openness information is, for example, information that associates the eye openness of the occupant with the captured image acquired by the image acquisition unit.

[0040] The gaze detection unit 122 detects the gaze of the occupant based on the captured image acquired by the image acquisition unit 10. The gaze detection unit 122 may detect the gaze direction of the occupant using a known image recognition technique on the captured image. For example, if the imaging device 8 is a visible light camera, the gaze detection unit 122 detects the inner corners of the occupant's eyes and irises in the captured image and detects the gaze direction of the occupant from the positional relationship between the inner corners of the eyes and the irises. If the imaging device 8 is an infrared camera, the gaze detection unit 122 detects the gaze direction of the occupant from the positional relationship between the pupil and a Purkinje image obtained by reflecting a near-infrared point light source from the infrared camera. Note that these are merely examples, and the gaze detection unit 122 may detect the gaze direction of the occupant using various known algorithms. The gaze direction of the occupant is expressed as an angle relative to a predetermined reference direction. For example, the gaze direction of the occupant is expressed as a gaze angle that indicates how many degrees the gaze direction has rotated horizontally and vertically from the state in which the occupant is looking straight ahead while sitting in the seat. The gaze detection unit 122 outputs information relating to the detected gaze of the occupant (hereinafter referred to as "gaze information") to the eye state determination unit 125. The gaze information is, for example, information in which information indicating the gaze direction of the occupant is associated with a captured image. If the gaze detection unit 122 is unable to detect the gaze of the occupant, it outputs gaze information indicating that the gaze of the occupant could not be detected to the eye state determination unit 125.

[0041] The face direction detection unit 123 detects the face direction of the occupant based on the captured image acquired by the image acquisition unit 10. The face direction detection unit 123 may detect the face direction of the occupant using a known image recognition technique on the captured image. For example, the face direction detection unit 123 may detect the face direction of the occupant using general pattern recognition based on the captured image. This is merely one example, and the face direction detection unit 123 may detect the face direction of the occupant using various known algorithms. The face direction of the occupant is expressed as an angle with respect to a predetermined reference direction. For example, the face direction of the occupant is expressed as a face direction angle that indicates how many degrees the face direction has rotated horizontally and vertically from a state in which the occupant is looking straight ahead in the seat. The face direction detection unit 123 outputs information regarding the detected face direction of the occupant (hereinafter referred to as "face direction information") to the eye state determination unit 125. The face direction detection unit 123 may also output the detected face direction information to the posture determination unit 128. The face direction information is, for example, information in which the face direction of the occupant is associated with a captured image.

[0042] The drowsiness detection unit 124 detects the drowsiness level of the occupant based on the captured image acquired by the image acquisition unit 10. The drowsiness detection unit 124 may detect the drowsiness level of the occupant using a known image recognition technique on the captured image. For example, the drowsiness detection unit 124 may determine the number of blinks of the occupant per a predetermined unit time based on the captured image. The drowsiness detection unit 124 detects the drowsiness level of the occupant based on the number of blinks per unit time. The drowsiness level corresponding to the number of blinks per unit time is determined in advance. For example, the drowsiness level may be set to a value such that a higher drowsiness level indicates greater drowsiness of the occupant, and a lower drowsiness level indicates a more awake occupant. The drowsiness detection unit 124 may detect the drowsiness level of the occupant using various known algorithms. Note that, although the drowsiness detection unit 124 detects the drowsiness level of the occupant based on the captured image in this example, this is merely an example. The drowsiness detection unit 124 may, for example, acquire biological information of the occupant, such as body temperature, from a biological sensor provided in the vehicle 1 and detect the drowsiness level of the occupant from the biological information. The drowsiness detection unit 124 outputs information related to the detected drowsiness level of the occupant (hereinafter referred to as "drowsiness level information") to the dozing determination unit 127. The drowsiness level information is, for example, information in which the drowsiness level of the occupant is associated with a captured image.

[0043] When the eye opening degree of the occupant determined by the eye opening degree determination unit 121 is less than a first threshold, the eye state determination unit 125 determines whether the occupant has their eyes downcast or closed, based on the gaze direction of the occupant detected by the gaze detection unit 122 before it was determined that the eye opening degree was less than the first threshold and the facial direction of the occupant detected by the facial direction detection unit. Specifically, when the eye opening degree determined by the eye opening degree determination unit 121 is less than the first threshold, if the gaze direction of the occupant detected by the gaze detection unit 122 is directed downward and the eye opening degree becomes less than the first threshold after the facial direction of the occupant detected by the facial direction detection unit 123 is directed downward, the eye state determination unit 125 determines that the occupant has their eyes downcast. On the other hand, the eye state determination unit 125 determines that the occupant has their eyes closed if the gaze direction of the occupant detected by the gaze detection unit 122 is directed downward and the degree of eye opening does not become less than the first threshold after the occupant's facial direction detected by the face direction detection unit 123 has turned downward. The determination of whether the occupant has their eyes down or closed, performed by the eye state determination unit 125, is also referred to as a "downcast eyes determination." The eye state determination unit 125 determines whether the degree of eye opening of the occupant has become less than the first threshold based on the eye opening degree information output from the eye opening degree determination unit 121. Note that if the degree of eye opening of the occupant determined by the eye opening degree determination unit 121 is equal to or greater than the first threshold, the eye state determination unit 125 does not perform a downcast eyes determination and determines that the occupant has their eyes open. The downcast eyes determination process performed by the eye state determination unit 125 is described below.

[0044] Eye state determination unit 125 stores, in chronological order, the eye opening degree information output from eye opening degree determination unit 121, the gaze information output from gaze detection unit 122, and the facial direction information output from facial direction detection unit 123. The information stored in chronological order by eye state determination unit 125, in which the eye opening degree information, gaze information, and facial direction information are associated with each other, is referred to as "eye-related information."

[0045] When the eye state determination unit 125 determines that the degree of eye opening of the occupant has become less than the first threshold, it refers to the eye-related information and obtains the gaze information of the occupant before the degree of eye opening of the occupant became less than the first threshold.

[0046] Next, based on the acquired gaze information of the occupant, the eye state determination unit 125 calculates a trajectory of the occupant's gaze direction from the gaze direction of the occupant detected by the gaze detection unit 122 before it was determined that the degree of eye opening was less than the first threshold. The eye state determination unit 125 refers to the eye-related information to acquire facial direction information of the occupant before the degree of eye opening of the occupant became less than the first threshold. Based on the acquired facial direction information of the occupant, the eye state determination unit 125 calculates a trajectory of the occupant's facial direction from the facial direction of the occupant detected by the facial direction detection unit 123 before it was determined that the degree of eye opening was less than the first threshold.

[0047] The eye state determination unit 125 determines whether the calculated trajectory of the occupant's gaze direction is directed downward and whether the calculated trajectory of the occupant's facial direction is directed downward. Specifically, the eye state determination unit 125 determines whether the calculated trajectory of the occupant's gaze direction is directed toward a predetermined downward viewing area and whether the calculated trajectory of the occupant's facial direction is directed toward the downward viewing area. If the trajectory of the occupant's gaze direction is directed toward the downward viewing area, the eye state determination unit 125 determines that the occupant's gaze direction is directed downward. Furthermore, if the trajectory of the occupant's facial direction is directed toward the downward viewing area, the eye state determination unit 125 determines that the occupant's facial direction is directed downward. The downward viewing area is an area set below the image capture device 8 within the vehicle 1. The downward viewing area is set as an area where it is assumed that the occupant may be looking aside if, for example, the occupant's line of sight remains within the area while looking downward for a predetermined period of time. For example, the downward viewing area is set as an area including an area where devices that the occupant can operate below the imaging device 8, such as the display of a car navigation device, are located.

[0048] The eye state determination unit 125 determines that the occupant has their eyes downcast if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are pointing downward. In other words, the eye state determination unit 125 determines that the occupant has their eyes downcast if the occupant's line of sight and the occupant's facial direction tend to be pointing downward and the degree of eye opening of the occupant is less than a first threshold.

[0049] The eye state determination unit 125 determines that the occupant has their eyes closed if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are not pointing downward. In other words, the eye state determination unit 125 determines that the occupant has their eyes closed unless the degree of eye opening of the occupant is less than the first threshold while the occupant's line of sight and the occupant's facial direction tend to point downward.

[0050] The eye state determination unit 125 outputs information indicating whether the occupant has their eyes downcast, closed, or open (hereinafter referred to as "eye state information") to the inattentive driving determination unit 126 and the drowsy driving determination unit 127.

[0051] When the eye state determination unit 125 outputs eye state information indicating that the occupant is looking down, in other words, when the eye state determination unit 125 determines that the occupant is looking down, the inattentiveness determination unit 126 determines whether the occupant is looking aside (hereinafter referred to as "inattentiveness determination") based on whether a preset condition (hereinafter referred to as "condition for inattentiveness determination") is satisfied. An example of the condition for inattentiveness determination is that "the occupant's downcast eyes continue for a preset time period (hereinafter referred to as "time period for inattentiveness determination") or more."

[0052] When the eye state determination unit 125 outputs eye state information indicating that the occupant is looking down, the inattentiveness determination unit 126 counts up the time (hereinafter referred to as the "time after downcast eye determination"). When the counted up time after downcast eye determination is equal to or greater than the inattentiveness determination time, the inattentiveness determination unit 126 determines that the occupant's downcast eye state has continued for equal to or greater than the inattentiveness determination time, and determines that the occupant is looking aside. The inattentiveness determination unit 126 stores the counted up time after downcast eye determination. On the other hand, when the inattentiveness determination unit 126 determines that the occupant's downcast eye state has not continued for equal to or greater than the inattentiveness determination time, it determines that the occupant is not looking aside. When the inattentiveness determination unit 126 determines that the occupant is looking aside as a result of performing the inattentiveness determination, it outputs information indicating that the occupant is looking aside to the execution feasibility determination unit 15 and the memory unit 14.

[0053] When the eye state determination unit 125 outputs eye state information indicating that the occupant has closed eyes, in other words, when the eye state determination unit 125 determines that the occupant has closed eyes, the drowsiness determination unit 127 determines whether the occupant is dozing off (hereinafter also referred to as "drowsiness determination") based on the drowsiness level of the occupant detected by the drowsiness detection unit 124. The drowsiness determination unit 127 may identify the drowsiness level of the occupant from the drowsiness level information output from the drowsiness detection unit 124. Specifically, the drowsiness determination unit 127 determines that the occupant is dozing off when a state (hereinafter referred to as a "drowsiness state") in which the occupant's drowsiness level is equal to or higher than a preset threshold (hereinafter referred to as a "drowsiness determination threshold") continues for a preset time (hereinafter referred to as a "drowsiness determination time") or longer. The doze determination unit 127 determines that the occupant is in a drowsy state when the eye state determination unit 125 outputs eye state information indicating that the occupant has closed eyes and the occupant's drowsiness level detected by the drowsiness detection unit 124 is equal to or greater than a drowsiness determination threshold. When the doze determination unit 127 determines that the occupant is in a drowsy state, it counts up a time (hereinafter referred to as the "post-drowsiness determination time"). The doze determination unit 127 stores the counted post-drowsiness determination time. When the counted post-drowsiness determination time is equal to or greater than the doze determination time, the doze determination unit 127 determines that the occupant's drowsy state has continued for equal to or greater than the doze determination time, and determines that the occupant is dozing. When the doze determination unit 127 determines that the occupant is dozing as a result of performing the doze determination, it outputs information indicating that the occupant is dozing to the execution feasibility determination unit 15 and the storage unit 14.

[0054] The posture determination unit 128 determines which of a plurality of predetermined posture types the posture of the vehicle occupant corresponds to based on the captured image. Occupants may lose their posture due to symptoms such as loss of consciousness or epilepsy. Figure 4 (partially modified from "Basic Design Document for Automatic Occupant Abnormality Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) shows the types of posture loss.

[0055] As shown in Figure 4, there are multiple modes of posture imbalance. Figure 4 shows "head down" in which the occupant leans forward and continues to have their face close to the steering wheel, "head down" in which the occupant continues to have their face facing down, "backward lean" in which the occupant's upper body tilts backward and continues to have their face facing up, "backward lean" in which the occupant's upper body arches up and continues to have their face facing up, "head tilted to the side" in which the occupant's face tilts to the left or right, "sideways lean" in which the occupant's upper body tilts to the left or right and their face tilts in the same direction, and "leaning to the side" in which the occupant's upper body tilts to the left or right. In this disclosure, "leaning face down," "head down," "arching backward," and "arching backward" are classified as posture type 1 (first posture type), and "head tilted to the side only," "head tilted to the side," and "leaning to the side" are classified as posture type 2 (second posture type). Posture type 1 is a posture in which the occupant's head, face, or upper body is tilted forward and along the vehicle, and posture type 2 is a posture in which the occupant's head, face, or upper body is tilted to the left side relative to the fore-and-aft direction of the vehicle.

[0056] The posture determination unit 128 determines whether the occupant is experiencing poor posture using the facial direction input from the facial direction detection unit 123 and the head position input from the element detection unit 11. For example, the type of poor posture is determined by determining whether the difference between the occupant's head position and facial direction obtained from the current image frame and a reference posture is equal to or greater than a predetermined threshold. As the reference posture, for example, the average value of the occupant's head position and facial direction obtained when the vehicle speed is 25 km / h or higher and the steering angle is within a range of ±20 degrees over a predetermined period of time (e.g., 3 seconds) can be used.

[0057] When the posture determination unit 128 determines that the occupant has lost their posture, it counts up the time (hereinafter referred to as the "time after posture loss determination"). The posture determination unit 128 stores the counted up time after posture loss determination. When the counted up time after posture loss determination is equal to or greater than the posture loss determination time, the posture determination unit 128 determines that the state in which the occupant has lost their posture has continued for equal to or greater than the posture loss determination time, and determines that the occupant's posture is abnormal. When the posture determination unit 128 determines that the occupant's posture is abnormal as a result of determining whether or not the occupant's posture is abnormal, it outputs information that the occupant's posture is abnormal to the execution feasibility determination unit 15 and the memory unit 14.

[0058] The inattention state, which indicates a state of absent-mindedness or low level of alertness, includes the occupant looking away, the occupant falling asleep, and the occupant having an abnormal posture.

[0059] The movement monitoring unit 13 monitors whether the occupant state detection unit 12 has detected the face or body elements of the occupant. The movement monitoring unit 13 acquires the face or body elements of the occupant from the element detection unit 11.

[0060] If the element detection unit 11 is unable to detect elements of the occupant's face or body, the elements of the occupant's face or body are not output from the element detection unit 11 to the occupant state detection unit 12, and the occupant state detection unit 12 is unable to acquire information regarding the elements of the occupant's face or body. That is, the occupant state detection unit 12 is unable to detect elements of the occupant's face or body. Furthermore, even if a captured image is output from the image acquisition unit 10 to the occupant state detection unit 12 without going through the element detection unit 11, if the element detection unit 11 is unable to detect elements of the occupant's face or body, it can be determined that the occupant state detection unit 12 is also unable to detect elements of the occupant's face or body. Therefore, the operation monitoring unit 13 can determine and monitor whether the element detection unit 11 is able to detect elements of the occupant's face or body by determining and monitoring whether the element detection unit 11 is able to detect elements of the occupant's face or body.

[0061] The operation monitoring unit 13 determines whether or not the occupant state detection unit 12 has detected all the elements. If the operation monitoring unit 13 determines that the occupant state detection unit 12 has detected all the elements, the operation monitoring unit 13 determines that the occupant state detection unit 12 can normally detect the state of the occupant.

[0062] The execution possibility determination unit 15 obtains a determination result as to whether or not the occupant state detection unit 12 has detected all elements from the operation monitoring unit 13. The execution possibility determination unit 15 uses the determination result of the operation monitoring unit 13 to determine whether or not the driving assistance function can be executed by the driving control device 21.

[0063] The execution feasibility determination unit 15 can execute the driving assistance function when the operation monitoring unit 13 can determine that the state of the occupant can be normally detected. Furthermore, the execution feasibility determination unit 15 may be able to determine that the driving assistance function can be executed even when the operation monitoring unit 13 determines that the occupant state detection unit 12 has not been able to detect some elements. For example, if it is considered that being able to detect the eyes, among the body elements of the occupant, is important for executing the driving assistance function, the execution feasibility determination unit 15 can determine that the driving assistance function can be executed when the occupant state detection unit 12 has been able to detect the eyes, among the body elements of the occupant, even if other elements such as the nose or mouth have not been detected.

[0064] Furthermore, for example, if the execution feasibility determination unit 15 considers that it is important to be able to detect the eyes among the elements of the occupant's face or body in order to execute the driving assistance function, it can determine that the driving assistance function cannot be executed if the occupant state detection unit 12 is unable to detect the eyes among the elements of the occupant's face or body.

[0065] Furthermore, for example, if the occupant state detection unit 12 is unable to detect more than half of the facial or bodily elements of the occupant, the execution possibility determination unit 15 can determine that the execution of the driving assistance function is impossible. Note that the execution of a driving assistance function also includes the continuation of a driving assistance function that is already being executed.

[0066] When the execution possibility determination unit 15 determines that the driving assistance function cannot be executed, it outputs a signal indicating that the driving assistance function cannot be executed to the driving control device 21 that executes braking / driving control of the vehicle 1. Furthermore, when the execution possibility determination unit 15 determines that the driving assistance function can be executed, it may output a signal indicating that the driving assistance function can be executed to the driving control device 21 that executes braking / driving control of the vehicle 1.

[0067] In addition, the execution feasibility determination unit 15 may determine that, for example, if the occupant state detection unit 12 is unable to detect the eyes among the facial features, only the cruise control function among the multiple driving assistance functions can be executed.

[0068] The execution feasibility determination unit 15 obtains a detection result indicating whether or not the occupant is in an inattentive state from the occupant state detection unit 12. The execution feasibility determination unit 15 uses the detection result from the occupant state detection unit 12 to determine whether or not the driving assistance function of the driving control device 21 that executes braking / driving control of the vehicle 1 can be executed. The execution feasibility determination unit 15 obtains information indicating that the occupant is in an inattentive state from the inattentiveness determination unit 126. The execution feasibility determination unit 15 also obtains information indicating that the occupant is in an inattentive state from the drowsiness determination unit 127. The execution feasibility determination unit 15 also obtains information indicating that the occupant's posture is abnormal from the posture determination unit 128.

[0069] The execution feasibility determination unit 15 can determine that a notification should be issued to the occupant when it receives information that the occupant is in an inattentive state, information that the occupant is in a dozing state, or information that the occupant's posture is abnormal from the inattentiveness determination unit 126, the drowsiness determination unit 127, or the posture determination unit 128. Furthermore, when it determines that a notification should be issued to the occupant, the execution feasibility determination unit 15 can output a determination result that a notification should be issued to the command unit 16.

[0070] The execution feasibility determination unit 15 determines that the driving assistance function cannot be executed when the occupant state detection unit 12 detects that the occupant is in an inattentive state. More specifically, the execution feasibility determination unit 15 can determine that the driving assistance function needs to be suppressed when the execution feasibility determination unit 15 acquires information that the occupant is in an inattentive state, information that the occupant is in a drowsy state, or information that the occupant's posture is abnormal a predetermined number of times or more from the inattentiveness determination unit 126, the drowsiness determination unit 127, or the posture determination unit 128. In other words, the execution feasibility determination unit 15 can determine that the driving assistance function needs to be suppressed when the number of times it has been determined that a notification should be issued is a predetermined number or more. Note that, in determining whether the driving assistance function can be executed, it is not necessarily necessary to determine that a notification should be issued and to issue a notification.

[0071] Furthermore, when the execution possibility determination unit 15 acquires information indicating that the occupant is looking away, that the occupant is dozing, or that the occupant's posture is abnormal a predetermined number of times or more, that is, when it determines that the driving assistance function cannot be executed, it can output a signal indicating that the driving assistance function cannot be executed to the driving control device 21. Note that the predetermined number of times can be set to any number of times. For example, the predetermined number of times can be set to three times.

[0072] When the execution possibility determination unit 15 determines that the driving assistance function cannot be executed, it outputs a signal indicating that the driving assistance function cannot be executed by the driving control device 21 to the driving control device 21. When the driving control device 21 acquires the signal indicating that the driving assistance function cannot be executed, it suppresses the driving assistance function.

[0073] The execution possibility determination unit 15 outputs factors that determine that the execution of the driving assistance function is impossible to the memory unit 14 and the command unit 16. Factors that determine that the execution of the driving assistance function is impossible include, for example, the occupant being inattentive, the occupant being dozing, or the occupant having an abnormal posture. In other words, the occupant being in a state of inattention. Furthermore, a factor that determines that the execution of the driving assistance function is impossible is, for example, the occupant state detection unit 12 being unable to detect elements of the occupant's face or body.

[0074] If the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed, it can determine that the occupant should be notified that the driving assistance function cannot be executed, and output the determination result that the occupant should be notified that the driving assistance function cannot be executed to the command unit 16.

[0075] The storage unit 14 acquires information that the occupant is in an inattentive state, information that the occupant is in a drowsy state, and information that the occupant's posture is abnormal from the inattentiveness determination unit 126, the drowsiness determination unit 127, and the posture determination unit 128. The storage unit 14 also acquires factors that determine that execution of the driving assistance function is not possible from the execution feasibility determination unit 15. The storage unit 14 stores the factors that cause the execution feasibility determination unit 15 to determine that execution of the driving assistance function is not possible.

[0076] When the command unit 16 receives a signal from the execution feasibility determination unit 15 indicating that the occupant should be notified that the driving assistance function cannot be executed, the command unit 16 commands the output device 9 to issue a notification indicating that the driving assistance function cannot be executed. Furthermore, when the command unit 16 receives from the memory unit 14 or the execution feasibility determination unit 15 the reason why the execution feasibility determination unit 15 determined that the driving assistance function cannot be executed, the command unit 16 commands the output device 9 to notify the occupant of the reason why it determined that the driving assistance function cannot be executed.

[0077] Factors that cause the execution feasibility determination unit 15 to determine that the driving assistance function is not executable include an inattentive state of the occupant, such as the occupant looking away, the occupant falling asleep, or the occupant having an abnormal posture, and the inability to detect a facial or bodily element of the occupant. When the execution feasibility determination unit 15 determines that the driving assistance function is not executable due to an inattentive state, the command unit 16 outputs a command to the output device 9 to cause a notification indicating that it has been determined that the driving assistance function is not executable due to an inattentive state of the occupant. When the execution feasibility determination unit 15 determines that the driving assistance function is not executable due to an inability to detect a facial or bodily element of the occupant, the command unit 16 outputs a command to the output device 9 to cause a notification indicating that it has been determined that the driving assistance function is not executable due to an inability to detect a facial or bodily element of the occupant.

[0078] The command unit 16 outputs a command to the output device 9 to notify the driver that it has been determined that the driving assistance function cannot be executed due to the driver's inattention. This allows the driver to know that the driving assistance function has been suppressed due to the driver's inattention. Furthermore, the command unit 16 outputs a command to the output device 9 to notify the driver that it has been determined that the driver's face or body element cannot be detected. This allows the driver to know that it has been determined that the driving assistance function cannot be executed due to the driver's face or body element cannot be detected.

[0079] The output device 9 receives from the command unit 16 a command to issue a notification indicating that it has been determined that execution of the driving assistance function is impossible due to the occupant's inattention. The output device 9 outputs a notification indicating that it has been determined that execution of the driving assistance function is impossible due to the occupant's inattention. The output device 9 receives from the command unit 16 a command to issue a notification indicating that it has been determined that execution of the driving assistance function is impossible due to the occupant's face or body elements not being detected. The output device 9 outputs a notification indicating that it has been determined that execution of the driving assistance function is impossible due to the occupant's face or body elements not being detected.

[0080] Next, the operation of the occupant monitoring device 2 will be described. Fig. 5 is a flowchart showing an example of the operation of the occupant monitoring device 2 according to the first embodiment. The occupant monitoring device 2 starts operation, for example, when the imaging device 8 is activated, and repeats processing operations while the vehicle 1 is traveling. The following description will be given assuming that the occupant state detection unit 12 is detecting an occupant at "Start (A)" in Fig. 5.

[0081] In step ST101, the image acquisition unit 10 acquires a captured image from the imaging device 8. The image acquisition unit 10 outputs the acquired captured image to the element detection unit 11.

[0082] In step ST102, the element detection unit 11 detects elements of the face or body of the occupant using the captured image acquired by the image acquisition unit 10. The element detection unit 11 outputs the elements of the face or body of the occupant to the movement monitoring unit 13.

[0083] In step ST103, the operation monitoring unit 13 determines whether the element detection unit 11 has detected all elements. That is, it monitors whether the occupant state detection unit 12 has detected elements. If the element detection unit 11 has not detected all elements (ST103; NO), the operation monitoring unit 13 proceeds to the processing of step ST104. If the element detection unit 11 has detected all elements (ST103; YES), the operation monitoring unit 13 determines that the occupant state can be detected normally, and proceeds to the processing of step ST107.

[0084] In step ST104, the execution feasibility determination unit 15 determines whether the driving assistance function is executable. For example, if the occupant state detection unit 12 can detect the eyes among the body elements of the occupant, the execution feasibility determination unit 15 can determine that the driving assistance function is executable even if other elements such as the nose or mouth cannot be detected. Furthermore, for example, if the occupant state detection unit 12 cannot detect the eyes among the face or body elements of the occupant, the execution feasibility determination unit 15 can determine that the driving assistance function is not executable. Furthermore, if the occupant state detection unit 12 cannot detect more than half of the face or body elements of the occupant, the execution feasibility determination unit 15 can determine that the driving assistance function is not executable.

[0085] If the execution possibility determination unit 15 determines that the driving assistance function can be executed (ST104; YES), the process proceeds to step ST101. Note that when returning to step ST101, a notification that the driving assistance function can be executed but that some elements of the occupant's face or body have not been detected may be output to the command unit 16. The command unit 16 may output a command to the output device 9 to output a notification that some elements of the occupant's face or body have not been detected. The output device 9 may output a notification that some elements of the occupant's face or body have not been detected. If the execution possibility determination unit 15 determines that the driving assistance function cannot be executed (ST104; YES), that is, if it is determined that the driving assistance function cannot be executed, the process proceeds to step ST105.

[0086] In step ST105, the execution possibility determination unit 15 outputs a signal to the driving control device 21 indicating that the driving support function cannot be executed by the driving control device 21.

[0087] In step ST106, when the driving control device 21 receives a signal requesting suppression of the driving support function, the driving control device 21 suppresses the execution of the driving support function.

[0088] In step ST107 , the image acquisition unit 10 acquires a captured image from the imaging device 8 .

[0089] In step ST108, the occupant state detection unit 12 detects the state of the occupant.

[0090] In step ST109, the occupant state detection unit 12 determines whether the occupant is in a state of absent-mindedness or a state of low alertness. If the occupant is in an inattentive state (ST109; YES), the process proceeds to step ST110. If the occupant is not in an inattentive state (ST109; NO), the process returns to step ST101. The occupant is detected to be in an inattentive state. Inattentive states that indicate an occupant is in an absent-minded state or a state of low alertness include the occupant being distracted, the occupant being dozing, or the occupant having an abnormal posture.

[0091] In step ST110, the execution possibility determination unit 15 determines whether the detection result of the occupant state detection unit 12 satisfies a predetermined condition. The predetermined condition may be, for example, that the occupant state detection unit 12 has detected an inattentive state, including the occupant looking away, the occupant falling asleep, or the occupant's abnormal posture, a predetermined number of times or more. In other words, the execution possibility determination unit 15 can determine that the predetermined condition is satisfied when it has obtained information indicating that the occupant is in an inattentive state a predetermined number of times or more.

[0092] If the detection result of the occupant state detection unit 12 satisfies a predetermined condition (ST110; YES), the execution possibility determination unit 15 determines that the driving assistance function needs to be suppressed, and proceeds to the processing of ST111. If the detection result of the occupant state detection unit 12 does not satisfy the predetermined condition (ST110; NO), the processing returns to the processing of step ST101. Note that, when returning to the processing of step ST101 from step ST104, the execution possibility determination unit 15 may store the number of times that information indicating that the occupant state is in an inattentive state in the storage unit 14, and when returning to the processing of step ST110 again, the number of times that information indicating that the occupant state is in an inattentive state is acquired may be added.

[0093] In step ST111, the execution possibility determination unit 15 outputs a signal to the driving control device 21 indicating that the driving support function cannot be executed by the driving control device 21.

[0094] In step ST112, when the driving control device 21 receives a signal requesting suppression of the driving support function, the driving control device 21 suppresses the execution of the driving support function.

[0095] 6 and 7 are diagrams showing an example of the hardware configuration of the occupant monitoring device 2 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 10, the occupant state detection unit 12, the execution feasibility determination unit 15, and the command unit 16 are realized by a processing circuit 1001. That is, the occupant monitoring device 2 includes the processing circuit 1001 for controlling the monitoring of the state of the vehicle occupant based on the captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 6, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 7.

[0096] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0097] When the processing circuit is the processor 1004, the functions of the image acquisition unit 10, the occupant state detection unit 12, the execution feasibility determination unit 15, and the command unit 16 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 10, the occupant state detection unit 12, the execution feasibility determination unit 15, the storage unit 14, and the command unit 16 by reading and executing the program stored in the memory 1005. That is, the occupant monitoring device 2 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST10 in FIG. 8 described above. The program stored in the memory 1005 can also be said to cause a computer to execute the processing procedures or methods of the image acquisition unit 10, the occupant state detection unit 12, the execution feasibility determination unit 15, and the command unit 16. Here, memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0098] It is also possible to realize some of the functions of the image acquisition unit 10, the occupant state detection unit 12, the execution feasibility determination unit 15, and the command unit 16 with dedicated hardware and some with software or firmware. For example, the image acquisition unit 10 may be realized by a processing circuit 1001 as dedicated hardware, and the occupant state detection unit 12, the execution feasibility determination unit 15, and the command unit 16 may be realized by the processor 1004 reading and executing programs stored in the memory 1005.

[0099] The storage unit 14 is configured by, for example, a memory 1005. The occupant monitoring device 2 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 8 or the output device 9.

[0100] The occupant monitoring device 2 according to this embodiment includes an image acquisition unit 10 that acquires an image of an occupant in the vehicle interior 1 from an imaging device 8 mounted on the vehicle 1 and capturing an image of the interior of the vehicle 1, an element detection unit 11 that detects elements of the occupant's face or body using the image acquired by the image acquisition unit 10, an occupant state detection unit 12 that detects an inattentive state indicating that the occupant is in a distracted state or has a reduced level of alertness using the elements detected by the element detection unit 11, an operation monitoring unit 13 that monitors whether the occupant state detection unit 12 has detected the elements, and an execution feasibility determination unit 15 that uses the results of the operation monitoring unit 13 to determine whether a driving assistance function can be executed by a driving control device 21 that executes braking / driving control of the vehicle 1. Thus, the occupant monitoring device 2 can reduce unnecessary suppression of the driving assistance function.

[0101] Furthermore, the vehicle control system 3 according to this embodiment includes the above-described occupant monitoring device 2 and a driving control device 21 that controls the driving of the vehicle 1. When the occupant monitoring device 2 determines that it is necessary to suppress the driving assistance function by the driving control device 21, the occupant monitoring device 2 outputs a signal to the driving control device 21 indicating that execution of the driving assistance function by the driving control device 21 is not possible based on the detection result of the occupant state detection unit 12, and when the driving control device 21 acquires the signal indicating that execution of the driving assistance function is not possible, the driving control device 21 outputs a signal to the steering mechanism 7 or the braking / driving mechanism 6 mounted on the vehicle 1 to cancel the suppression of the driving assistance function by the driving control device 21. Therefore, the vehicle control system 3 can reduce unnecessary suppression of the driving assistance function.

[0102] The occupant monitoring method according to the present embodiment includes the steps of: an image acquisition unit 10 acquiring an image of an occupant inside the vehicle 1 from an imaging device 8 mounted on the vehicle 1 and capturing an image of the interior of the vehicle 1; an element detection unit detecting elements of the occupant's face or body using the image acquired by the image acquisition unit; an occupant state detection unit detecting an inattentive state indicating that the occupant is in a distracted state or a state of reduced alertness using the elements detected by the element detection unit; an operation monitoring unit monitoring whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit using the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that executes braking / driving control of the vehicle. Therefore, the occupant monitoring method can reduce unnecessary suppression of the driving assistance function.

[0103] Furthermore, the program according to the present embodiment causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing an image of the interior of the vehicle; an element detection unit detects elements of the occupant's face or body using the image acquired by the image acquisition unit; an occupant state detection unit detects an inattentive state indicating that the occupant is in a distracted state or a state of reduced alertness using the elements detected by the element detection unit; an operation monitoring unit monitors whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit determines whether a driving assistance function can be executed by a driving control device that executes braking / driving control of the vehicle using the results of the operation monitoring unit. Therefore, the program can reduce unnecessary suppression of the driving assistance function.

[0104] Embodiment 2 An occupant monitoring device 31 according to embodiment 2 will be described with reference to Figures 8 and 9. Description of configurations similar to those of embodiment 1 will be omitted. In Figures 8 and 9, the same reference numerals as those in Figures 1 to 7 indicate the same or corresponding parts. In addition, steps that are the same as those in the processing of the occupant monitoring device 2 according to embodiment 1 are assigned the same reference numerals as those shown in Figure 5, and description thereof will be omitted or simplified. The following description will focus on differences from embodiment 1.

[0105] The passenger monitoring device 31 according to this embodiment differs from the passenger monitoring device 2 according to the first embodiment in that a cancellation request unit 32 is further provided.

[0106] The occupant monitoring device 31 includes a cancellation request unit 32. The cancellation request unit 32 is connected to the communication bus 17 and is capable of transmitting and receiving data to and from the image acquisition unit 10, the occupant state detection unit 12, the operation monitoring unit 13, the memory unit 14, the execution feasibility determination unit 15, and the command unit 16 via the communication bus 17.

[0107] 8 is a block diagram showing an occupant monitoring device 31 according to Embodiment 2. The execution possibility determination unit 15 outputs to the cancellation request unit 32 the reason for determining that the execution of the driving assistance function is impossible.

[0108] When the execution possibility determination unit 15 determines that the execution of the driving assistance function is not possible, the release request unit 32 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function by the driving control device 21 based on the detection result of the occupant state detection unit 12. In other words, when it can be determined based on the detection result of the occupant state detection unit 12 that the factor that caused the execution possibility determination unit 15 to determine that the execution of the driving assistance function is not possible has been removed, the release request unit 32 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function by the driving control device 21.

[0109] The release request unit 32 requests the release of the suppression of the driving assistance function by the driving control device 21 based on the detection results of the occupant status detection unit 12, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed.

[0110] The release request unit 32 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function by the driving control device 21 if the time that the vehicle 1 has been continuously stopped after the driving assistance function has been suppressed is equal to or longer than a predetermined stopping reference time, or if the detection result of the occupant state detection unit 12 when the vehicle 1 is moving after being stopped for equal to or longer than the stopping reference time indicates that the occupant's state is not in an inattentive state, and this state continues for equal to or longer than a predetermined first attention duration time.

[0111] The stopping reference time is, for example, the time required for a rest to recover from fatigue or other factors that may have led to the occupant's inattentive state. The stopping reference time can be set to, for example, 15 minutes. The first attention duration is the time considered necessary to determine that the occupant has recovered from the inattentive state when the vehicle 1 has been stopped continuously for a period of time equal to or longer than the predetermined stopping reference time. The first attention duration can be set to, for example, 5 minutes.

[0112] If the vehicle 1 is continuously stopped for a period of time equal to or longer than a predetermined stopping reference time, it can be assumed that the occupant has had a sufficient rest and recovered from fatigue. Therefore, if the detection result of the occupant state detection unit 12 when the vehicle 1 is stopped for a period of time equal to or longer than the stopping reference time and then starts driving indicates that the occupant is not in an inattentive state and continues for a predetermined first attention duration time or longer, the release request unit 32 can determine that the occupant has recovered from the inattentive state. In other words, it can be determined that the factor that caused the execution feasibility determination unit 15 to determine that the driving assistance function cannot be executed has been removed.

[0113] The release request unit 32 determines that the occupant has left the inattentive state and outputs a signal to the driving control device 21 requesting that the driving control device 21 release the suppression of the driving assistance function, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed when the vehicle 1 is continuously stopped for a period of time that is equal to or longer than a predetermined stopping reference time.

[0114] In addition, the release request unit 32 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function by the driving control device 21 if the time that the vehicle 1 has been continuously stopped after the driving assistance function has been suppressed is less than a predetermined stopping reference time, or if the detection result of the occupant state detection unit 12 when driving after the driving assistance function has been suppressed indicates that the occupant's state is not in an inattentive state for more than a predetermined second attention duration time.

[0115] The second attention duration is a time considered necessary to determine that the occupant has ceased to be inattentive when the vehicle 1 has been continuously stopped for less than a predetermined stopping reference time. When the vehicle 1 has been continuously stopped for less than the predetermined stopping reference time, it is considered that a longer time is necessary to determine that the occupant has ceased to be inattentive compared to when the vehicle 1 has been continuously stopped for more than the predetermined stopping reference time. Therefore, the second attention duration can be set to, for example, a time longer than the first attention duration. The second attention duration can be set to, for example, 120 minutes.

[0116] If the time that the vehicle 1 is continuously stopped is less than a predetermined stop reference time, it is possible that the occupant has not been able to take a sufficient rest and recover from fatigue, etc. However, if the detection result of the occupant state detection unit 12 when driving after the driving assistance function is suppressed indicates that the occupant is not in an inattentive state for a predetermined second attention duration time or more, the cancellation request unit 32 can determine that the occupant has recovered from the inattentive state. In other words, it can determine that the factor that caused the execution feasibility determination unit 15 to determine that the driving assistance function cannot be executed has been removed.

[0117] The release request unit 32 determines that the occupant has escaped from the inattentive state and outputs a signal to the driving control device 21 requesting that the driving control device 21 release the suppression of the driving assistance function, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed when the vehicle 1 is continuously stopped for less than a predetermined stopping reference time.

[0118] Furthermore, the release request unit 32 can output information requesting the release of the suppression of the driving assistance function to the command unit 16 .

[0119] In addition, when the cancellation request unit 32 outputs a signal to the driving control unit 21 requesting cancellation of the suppression of the driving assistance function by the driving control unit 21, the cancellation request unit 32 can reset the number of times that the execution feasibility determination unit 15 has received information that the occupant is looking away, that the occupant is dozing, or that the occupant's posture is abnormal.

[0120] When the driving control device 21 receives a signal requesting cancellation of the suppression of the driving support function from the cancellation request unit 32, the driving control device 21 cancels the suppression of the driving support function.

[0121] The command unit 16 can acquire information requesting the cancellation of suppression of the driving assistance function from the cancellation request unit 32. The command unit 16 can also acquire, from the memory unit 14, factors that are stored in the memory unit 14 and that cause the execution feasibility determination unit 15 to determine that the driving assistance function cannot be executed.

[0122] When the release request unit 32 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function, the command unit 16 can issue a command to the output device 9 installed in the vehicle 1 to send a notification indicating the cause based on the cause stored in the memory unit 14.

[0123] Next, a description will be given of the operation of the occupant monitoring device 31. Fig. 8 is a flowchart showing an example of the operation of the occupant monitoring device 31 according to the second embodiment.

[0124] In step ST201, the release request unit 32 determines whether the time during which the vehicle 1 has been continuously stopped is equal to or longer than a predetermined stopping reference time. If the time during which the vehicle 1 has been continuously stopped is equal to or longer than the predetermined stopping reference time (ST201; YES), the process proceeds to step ST202. If the time during which the vehicle 1 has been continuously stopped is not equal to or longer than the predetermined stopping reference time (ST203; NO), that is, if the time during which the vehicle 1 has been continuously stopped is shorter than the predetermined stopping reference time, the process proceeds to step ST204.

[0125] Note that the acquisition of the captured image executed in step ST101, the element detection executed in step ST102, the detection of the occupant's state executed in step ST108, and the determination of whether the occupant is in an inattentive state executed in step ST109 are executed continuously, for example, even when the vehicle 1 is traveling after being stopped. Furthermore, the processes of step ST101 and step ST102 may be executed even when the vehicle 1 is stopped.

[0126] Furthermore, when determining whether the occupant state is in an inattentive state, the occupant state detection unit 12 can measure the duration of a state indicating that the occupant state is not inattentive when the vehicle 1 is stopped for a stopping reference time or longer and then starts driving. Furthermore, when determining whether the occupant state is in an inattentive state, the occupant state detection unit 12 can measure the duration of a state indicating that the occupant state is not inattentive when the vehicle 1 is driven after the driving assistance function is suppressed. Furthermore, the occupant state detection unit 12 may output the duration of a state indicating that the occupant state is not inattentive to the memory unit 14 and store it in the memory unit 14.

[0127] In step ST202, the release request unit 32 determines whether the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration (ST202; YES), the process proceeds to step ST204. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration (ST202; NO), the process returns to step ST201.

[0128] In step ST203, the release request unit 32 determines whether the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration (ST203; YES), the process proceeds to step ST204. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration (ST203; NO), the process returns to step ST201.

[0129] In step ST204, the release request unit 32 outputs a signal to the driving control device 21 requesting the driving control device 21 to release the suppression of the driving support function.

[0130] In step ST205, the driving control device 21 cancels the suppression of the driving support function. After the process of step ST205, the process returns to the process of step ST101.

[0131] The occupant monitoring device 31 according to this embodiment further includes a release request unit 32 that, when the execution possibility determination unit 15 determines that execution of the driving assistance function is not possible, outputs a signal to the driving control device 21 requesting release of suppression of the driving assistance function by the driving control device 21 based on the detection result of the occupant state detection unit 12. Therefore, the occupant monitoring device 31 can prevent a decrease in convenience for the user.

[0132] Embodiment 3 An occupant monitoring device 51 according to embodiment 3 will be described with reference to Figures 10 and 11. Descriptions of configurations similar to those of embodiment 1 or embodiment 3 will be omitted. In Figures 10 and 11, the same reference numerals as those in Figures 1 to 9 indicate the same or corresponding parts. In addition, steps that are the same as those in the processing of the occupant monitoring device 2 according to embodiment 1 or the occupant monitoring device 31 according to embodiment 2 are assigned the same reference numerals as those shown in Figures 5 and 9, and descriptions thereof will be omitted or simplified. The following description will focus on differences from embodiment 1.

[0133] The occupant monitoring device 51 according to this embodiment differs from the occupant monitoring device 2 according to the first embodiment in that an occlusion determining unit 52 is further provided.

[0134] The occupant monitoring device 51 includes an occlusion determination unit 52. The occlusion determination unit 52 is connected to a communication bus 17, and is capable of transmitting and receiving data to and from the image acquisition unit 10, the element detection unit 11, the occupant state detection unit 12, the operation monitoring unit 13, the memory unit 14, the execution feasibility determination unit 15, the command unit 16, and the release request unit 32 via the communication bus 17.

[0135] 10 is a block diagram showing an occupant monitoring device 51 according to embodiment 3. An occlusion determination unit 52 acquires a captured image from the image acquisition unit 10. Using a known image recognition technique, the occlusion determination unit 52 determines whether or not there is an obstruction between the imaging device and the occupant that obstructs the capture of an image of the occupant.

[0136] The occupant state detection unit 12 outputs the detection result of the occupant state to the execution possibility determination unit 15 and the storage unit 14 .

[0137] The occlusion determination unit 52 outputs the determination result of whether or not an obstruction exists to the execution feasibility determination unit 15 and the storage unit 14. When the execution feasibility determination unit 15 acquires the determination result that an obstruction exists from the occlusion determination unit 52, it can determine that it is necessary to suppress the driving assistance function. Furthermore, when the execution feasibility determination unit 15 acquires the determination result that an obstruction exists from the occlusion determination unit 52, it can determine that it is necessary to suppress the driving assistance function and output a signal requesting the suppression of the driving assistance function to the driving control device 21.

[0138] Furthermore, when the execution possibility determination unit 15 obtains the determination result that an obstruction exists from the obstruction determination unit 52 and determines that the driving assistance function needs to be suppressed, the execution possibility determination unit 15 can output information that the factor that determined that the driving assistance function needs to be suppressed is the presence of an obstruction to the storage unit 14 and store the information in the storage unit 14. Furthermore, the execution possibility determination unit 15 outputs information that the factor that determined that the driving assistance function cannot be executed is the presence of an obstruction to the cancellation request unit 32.

[0139] Furthermore, the execution feasibility determination unit 15 can determine that a notification should be given to the occupant when it acquires a determination result that an obstruction exists from the occupancy determination unit 52. Furthermore, the execution feasibility determination unit 15 can output to the command unit 16 information that the cause of the execution feasibility determination unit 15 determining that the driving assistance function cannot be executed is the presence of an obstruction, and a command to the effect that a notification should be given to the occupant.

[0140] The execution feasibility determination unit 15 can determine that the driving assistance function needs to be suppressed when the information obtained from the occupant state detection unit 12 indicates that the occupant is in an inattentive state, or when the information obtained from the occupant occupancy determination unit 52 indicates that an obstruction is present.

[0141] In other words, the execution feasibility determination unit 15 can determine that it is necessary to suppress the driving assistance function if the information obtained from the occupant state detection unit 12 indicates that the occupant is in an inattentive state and the information obtained from the occupancy determination unit 52 indicates that an obstruction is present.

[0142] In addition, the execution feasibility determination unit 15 can determine that it is necessary to suppress the driving assistance function if the information obtained from the occupant state detection unit 12 indicates that the occupant is in an inattentive state and the information obtained from the occupancy determination unit 52 indicates that no occupant is present.

[0143] In addition, the execution feasibility determination unit 15 can determine that it is necessary to suppress the driving assistance function if the information obtained from the occupant state detection unit 12 indicates that the occupant's state is not in an inattentive state and the information obtained from the occupancy determination unit 52 indicates that an obstruction is present.

[0144] In addition, the execution feasibility determination unit 15 can determine that there is no need to suppress the driving assistance function if the information obtained from the occupant state detection unit 12 indicates that the occupant is not in an inattentive state and the information obtained from the occupancy determination unit 52 indicates that no occupant is present.

[0145] In addition, the execution feasibility determination unit 15 may not obtain information indicating the occupant's status from the occupant status detection unit 12, and may determine that the driving assistance function needs to be suppressed based only on the information obtained from the occupancy determination unit 52 indicating the presence of an obstruction.

[0146] The storage unit 14 can acquire the determination result of whether or not an obstruction exists from the obstruction determination unit 52. The storage unit 14 can also acquire and store information from the execution feasibility determination unit 15 that the factor that caused the execution feasibility determination unit 15 to determine that the driving assistance function cannot be executed is the presence of an obstruction.

[0147] When the command unit 16 acquires from the execution possibility determination unit 15 or the memory unit 14 information that the factor that caused the execution of the driving assistance function to be impossible is the presence of an obstruction and a command to notify the occupant, the command unit 16 outputs a command to the output device 9 to make a notification indicating that an obstruction exists. That is, when the factor that caused the execution possibility determination unit 15 to determine that the execution of the driving assistance function to be impossible is the presence of an obstruction, the command unit 16 outputs a command to the output device 9 to make a notification indicating that an obstruction exists.

[0148] When the execution feasibility determination unit 15 determines that execution of the driving assistance function is not possible, the release request unit 32 outputs a signal requesting the release of suppression of the driving assistance function by the driving control device 21 to the driving control device 21. When the factor acquired from the execution feasibility determination unit 15 that caused the execution of the driving assistance function by the execution feasibility determination unit 15 to determine that execution of the driving assistance function is not possible is the presence of an obstruction, and when the release request unit 32 acquires a determination result from the obstruction determination unit 52 that the obstruction has been removed after the driving assistance function was suppressed, the release request unit 32 outputs a signal requesting the release of suppression of the driving assistance function by the driving control device 21 to the driving control device 21.

[0149] Next, a description will be given of the operation of the occupant monitoring device 51. Fig. 11 is a flowchart showing an example of the operation of the occupant monitoring device 51 according to the second embodiment.

[0150] In step ST301, the image acquisition unit 10 acquires a captured image from the imaging device 8. The image acquisition unit 10 outputs the acquired captured image to the occlusion determination unit 52.

[0151] In step ST302, the occlusion determination unit 52 determines whether or not there is an obstruction between the imaging device and the occupant that blocks the imaging of the occupant. If the occlusion determination unit 52 determines that there is an obstruction (ST302; YES), the process proceeds to step ST111. If the occlusion determination unit 52 determines that there is no obstruction (ST302; NO), the process returns to step ST301.

[0152] In step ST111, the execution possibility determination unit 15 outputs a signal to the driving control device 21 requesting the driving control device 21 to suppress the driving assistance function.

[0153] In step ST112, when the driving control device 21 receives a signal requesting suppression of the driving support function, the driving control device 21 suppresses the driving support function.

[0154] In step ST303, the command unit 16 commands the execution possibility determination unit 15 to issue a notification indicating the cause for determining that the driving assistance function cannot be executed. When the occlusion determination unit 52 determines that an obstruction exists, the command unit 16 outputs, to the output device 9, a command to issue a notification indicating that an obstruction exists.

[0155] In step ST304, the output device 9 outputs a notification indicating that an obstruction is present.

[0156] In step ST305, the image acquiring unit 10 acquires a captured image from the imaging device 8. The image acquiring unit 10 outputs the acquired captured image to the occlusion determining unit 52.

[0157] In step ST306, the occlusion determination unit 52 determines whether or not an obstruction that blocks the capture of an image of the occupant is present between the image capture device and the occupant after the driving assistance function is suppressed. If no obstruction is present (ST306; NO), the occlusion determination unit 52 determines that the obstruction has been removed and proceeds to the processing of step ST110. If an obstruction is present (ST306; YES), the processing returns to the processing of step ST205.

[0158] In step ST204, the release request unit 32 outputs a signal to the driving control device 21 requesting the driving control device 21 to release the suppression of the driving support function.

[0159] In step ST205, the driving control device 21 cancels the suppression of the driving support function. After the process of step ST111, the process returns to the process of step ST101.

[0160] The occupant monitoring device 51 in this embodiment is equipped with an occupant obstruction determination unit 52 that determines whether or not there is an obstruction between the imaging device 8 and the occupant that prevents the occupant from being photographed. If the occupancy determination unit 52 determines that an obstruction exists, the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed. If the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed because of the presence of an obstruction, the command unit 16 outputs a command to the output device 9 to cause a notification indicating the presence of an obstruction. Thus, even if an obstruction exists between the imaging device 8 and the occupant, the occupant monitoring device 51 can prevent a decrease in user convenience.

[0161] Embodiment 4 An occupant monitoring device 53 according to embodiment 4 will be described with reference to Figures 12 and 13. Descriptions of configurations similar to those of embodiments 1 to 3 will be omitted. Also, in Figures 12 and 13, the same reference numerals as those in Figures 1 to 11 indicate the same or corresponding parts. Also, steps that are the same as those in the processing of the occupant monitoring device 2 according to embodiment 1, the occupant monitoring device 31 according to embodiment 2, and the occupant monitoring device 51 according to embodiment 3 are assigned the same reference numerals as those shown in Figures 5, 9, and 11, and descriptions thereof will be omitted or simplified. The following description will focus on differences from embodiment 1.

[0162] The occupant monitoring device 53 according to this embodiment differs from the occupant monitoring device 2 according to the first embodiment in that a calibration unit 54 and a reliability calculation unit 55 are further provided.

[0163] 11 is a block diagram showing an occupant monitoring device 53 according to the fourth embodiment. The occupant monitoring device 53 includes a calibration unit 54 and a reliability calculation unit 55. The calibration unit 54 and the reliability calculation unit 55 are connected to a communication bus 17, and can transmit and receive data to and from the image acquisition unit 10, the element detection unit 11, the occupant state detection unit 12, the operation monitoring unit 13, the storage unit 14, the execution feasibility determination unit 15, the command unit 16, and the release request unit 32 via the communication bus 17.

[0164] The calibration unit 54 transmits and receives information to and from the occupant state detection unit 12 and performs calibration processing. The calibration unit 54 executes calibration processing for image processing in the occupant state detection unit 12. The calibration unit 54 executes calibration processing, which is processing for correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit 10. The calibration processing is processing for correcting individual differences between occupants who are the target of image processing. Generally, the detection results by the occupant state detection unit 12 involve detection errors due to individual differences between occupants. Therefore, the calibration unit 54 executes calibration processing to ensure the detection accuracy of the occupant state detection unit 12. Specific examples of calibration processing are shown below. Note that the examples are not limited to the following specific examples, and other known means may also be used.

[0165] First, an example will be described in which the calibration unit 54 corrects the standard model used for gaze direction detection. In this case, the occupant state detection unit 12 holds a standard model for gaze direction detection that associates a captured image of the occupant's face with the occupant's gaze direction. The occupant state detection unit 12 then uses the standard model to detect the occupant's gaze direction from the captured image and determine, for example, whether or not the occupant is looking away from the road. Here, the standard model held in the occupant state detection unit 12 needs to be corrected to accommodate individual differences between occupants.

[0166] Therefore, the calibration unit 54 corrects the standard model based on the actual gaze direction of the occupant and the captured image in that state. For example, the calibration unit 54 detects the gaze direction of the occupant and estimates the vehicle accessory the occupant is gazing at using the result. Then, the calibration unit 54 determines the direction of the vehicle accessory as seen by the occupant as a reference for the gaze direction, and corrects the standard model based on the error between this reference gaze direction and the detected gaze direction. Alternatively, the calibration unit 54 may correct the standard model by correlating the gaze of the occupant when operating each device in the vehicle 1 with the position of the device being operated. This allows the occupant state detection unit 12 to accurately detect the gaze direction regardless of individual differences between occupants.

[0167] The calibration unit 54 may also correct the standard model used for facial direction detection. In this case, the occupant state detection unit 12 holds a 3D model that is the standard model, and uses the 3D model to detect the facial direction of the occupant from the captured image and determine, for example, whether or not the occupant is looking away from the road. The 3D model is a 3D model that reflects, for example, the size of a standard adult's face and the characteristic points of facial features. In other words, the 3D model does not reflect the size of the occupant's face and the characteristic points of facial features.

[0168] Therefore, the calibration unit 54 extracts feature points from the occupant in the captured image that correspond to the feature points of the 3D model. The method for extracting the feature points of the occupant from the captured image is not particularly limited, and the feature points may be extracted, for example, by a feature point extraction technique using machine learning, an image recognition technique such as edge extraction, or template matching. The calibration unit 54 then compares the positions of the feature points of the 3D model with the positions of the feature points of the occupant extracted from the captured image on the captured image. Next, the calibration unit 54 calibrates the positions of the feature points of the 3D model to match the positions of the feature points of the occupant extracted from the captured image so that the error between the positions of the feature points of the 3D model and the positions of the feature points of the occupant extracted from the captured image is within a predetermined threshold. This allows the occupant state detection unit 12 to accurately detect the face direction regardless of individual differences between occupants.

[0169] The calibration unit 54 may also correct the standard model used to detect the degree of eye opening. In this case, the occupant state detection unit 12 holds a standard model for detecting the degree of eye opening of the occupant. The occupant state detection unit 12 then detects the occupant's eyes from the captured image and uses the standard model to detect the degree of eye opening, which is the degree to which the occupant's eyes are open. The standard model for detecting the degree of eye opening includes information on a reference flattening ratio, which is, for example, a value obtained by dividing the width of the eyes when the eyes are open by the height of the eyes. However, the reference flattening ratio differs depending on the individual occupant.

[0170] Therefore, the calibration unit 54 calculates the occupant's eye width and eye height from the captured image and calculates the occupant's reference flattening.The calibration unit 54 then corrects the standard model based on the error between the standard model's reference flattening and the calculated occupant's reference flattening.This allows the occupant state detection unit 12 to accurately detect the eye opening degree regardless of individual occupant differences.The calibration unit 54 outputs information indicating the corrected standard model, which is the result of the calibration process, to the storage unit 14.

[0171] The execution possibility determination unit 15 determines that the execution of the driving assistance function is impossible when the occupant state detection unit 12 detects that the occupant is in an inattentive state. Furthermore, the execution possibility determination unit 15 outputs information that the factor for determining that the execution of the driving assistance function is impossible is the occupant's inattentive state to the storage unit 14, and causes the storage unit 14 to store the information.

[0172] When the execution feasibility determination unit 15 obtains information from the execution feasibility determination unit 15 or the memory unit 14 that the factor that causes the execution of the driving assistance function to be determined to be impossible is that the occupant is in an inattentive state, the command unit 16 outputs a command to the output device 9 to issue a notification indicating that the driving assistance function has been suppressed due to the occupant's inattentive state.

[0173] In addition, when the execution possibility determination unit 15 determines that the driving assistance function cannot be executed because the occupant is in an inattentive state, the command unit 16 outputs a command to the calibration unit 54 to execute a calibration process.

[0174] The calibration unit 54 executes the calibration process when it receives a command to execute the calibration process from the command unit 16. The calibration unit 54 executes the calibration process based on the captured image in order to determine whether the occupant is looking aside, etc., based on the state in which the occupant is facing forward. However, there is a possibility that the calibration process will be executed by mistake, assuming that the captured image in which the occupant is not facing forward is an image in which the occupant is facing forward. In other words, the result of the calibration process may be incorrect. If the occupant state detection unit 12 detects an inattentive state of the occupant, the result of the calibration process executed by the calibration unit 54 may be incorrect, which may have caused the occupant state detection unit 12 to make an erroneous detection. By executing the calibration process by the calibration unit 54, it is possible to correct the result of the calibration process, which may have been incorrect.

[0175] Furthermore, when the execution possibility determination unit 15 determines that the driving assistance function cannot be executed because of inattention and the command unit 16 acquires a signal indicating that the vehicle 1 is stopped or traveling on a route without congestion from the driving control device 21, the command unit 16 may output a command to the calibration unit 54 to execute a calibration process. When the command unit 16 acquires a signal indicating that the vehicle 1 is stopped or traveling on a route without congestion from the driving control device 21, the command unit 16 outputs a command to the calibration unit 54 to execute a calibration process, so that the calibration unit 54 can perform the calibration process with the occupant facing forward.

[0176] The reliability calculation unit 55 calculates the reliability of the detection of the inattention state by the occupant state detection unit 12. The reliability is expressed, for example, as a value between 0 and 100. The reliability is smaller when a part of the eyes, nose, or mouth is blocked, for example, when the occupant covers their mouth with their hand, rubs their eyes with their hand, or looks back from the driver's seat to look into the backseat. The reliability is also smaller when the occupant is wearing sunglasses or other eyewear that blocks the infrared sensor light, because the contrast ratio around the eyes is reduced. The reliability is also smaller when the eyelids are lowered, because the iris and pupil cannot be captured. The reliability is also smaller when the face angle relative to the image capture device 8 is greater than a certain value, because the contrast ratio of the iris and pupil is reduced due to the shadow of the eyelashes or eyelids. The reliability calculation unit 55 determines whether the reliability is lower than a predetermined threshold.

[0177] If the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed because of inattention and the reliability is lower than a predetermined threshold, the command unit 16 outputs a command to the calibration unit 54 to execute a calibration process.

[0178] When the reliability is lower than the threshold, the results of the calibration process performed up until it is determined that the driving assistance function cannot be executed, i.e., the results previously performed by the calibration unit 54, are more likely to be incorrect than when the reliability is equal to or higher than the threshold. In other words, there is a high possibility that an inattentive state has been erroneously detected. When the reliability is lower than the predetermined threshold, the calibration unit 54 can be made to perform the calibration process again in the case where an inattentive state has been erroneously detected, thereby correcting the results of the calibration process that may have been erroneous.

[0179] If the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed because the cause is an inattentive state and the reliability is equal to or greater than a predetermined threshold, the command unit 16 outputs a command to the occupant state detection unit 12 to detect the occupant's inattentive state using the results of past execution by the calibration unit 54.

[0180] When the reliability is equal to or higher than the threshold, the result previously performed by the calibration unit 54 is more likely to be correct than when the reliability is lower than the threshold. When the reliability is equal to or higher than the predetermined threshold, the calibration unit 54 outputs a command to the occupant state detection unit 12 to detect the state of inattention of the occupant using the result previously performed by the calibration unit 54, so that the occupant state detection unit 12 can detect the state of the occupant using the result of the calibration process that is more likely to be correct.

[0181] When it is detected that the occupant is in an inattentive state, the calibration process may be performed even if the reliability is not low.

[0182] In addition, when the release request unit 32 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function, the command unit 16 can issue a command to the occupant status detection unit 12 to eliminate the factor based on the factor stored in the memory unit 14.

[0183] When the occupant state detection unit 12 receives a command from the command unit 16 to execute elimination of the cause, it can output a signal to the calibration unit 54 to execute calibration processing. When the calibration unit 54 receives a signal to execute calibration processing from the occupant state detection unit 12, it executes calibration processing. Execution of calibration processing by the calibration unit 54 leads to elimination of the cause.

[0184] When the release request unit 32 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function, the command unit 16 issues at least one of a command to the output device 9 mounted on the vehicle 1 to notify the output device 9 of the cause based on the cause stored in the memory unit 14, and a command to the occupant status detection unit 12 to resolve the cause, thereby further preventing a decrease in user convenience.

[0185] Next, a description will be given of the operation of the occupant monitoring device 53. Fig. 13 is a flowchart showing an example of the operation of the occupant monitoring device 51 according to the second embodiment.

[0186] In step ST401, the reliability calculation unit 55 determines whether the reliability is lower than a predetermined threshold. If the reliability is lower than the predetermined threshold (ST401; YES), the process proceeds to step ST402. If the reliability is equal to or higher than the predetermined threshold (ST401; NO), the process proceeds to step ST404.

[0187] In step ST402, if the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed because the occupant is in an inattentive state, the command unit 16 outputs a command to the calibration unit 54 to execute the configuration processing.

[0188] In step ST403, the calibration unit 54 executes a calibration process. After the process of step ST403, the process proceeds to step ST201, and the process of step ST202 or step ST203 is executed. That is, the process of step ST202 and step ST203 can be executed using the determination result of the occupant state executed after step ST403. That is, the occupant state can be detected based on the result of the calibration process, which is likely to be correct.

[0189] In step ST404, if the execution feasibility determination unit 15 determines that the driving assistance function cannot be executed because the occupant is in an inattentive state, the command unit 16 outputs a command to the occupant state detection unit 12 to detect the occupant's inattentive state using the results of previous execution by the calibration unit 54.

[0190] Note that even when the reliability is equal to or greater than a predetermined threshold (ST401; NO), the command unit 16 may output a command to the calibration unit 54 to execute the configuration process.

[0191] In the above-described embodiments, the materials, materials, dimensions, shapes, relative positional relationships, and implementation conditions of each component may be described. However, these are merely examples in all respects and are not intended to limit the scope of each embodiment. Therefore, countless variations not exemplified are contemplated within the scope of each embodiment. For example, these include cases where any component is modified, added, or omitted, and even cases where at least one component in at least one embodiment is extracted and combined with a component in another embodiment.

[0192] It goes without saying that various design modifications are possible within the scope of the present invention, as long as the object of the present invention can be achieved and the gist of the present invention is not deviated from.

[0193] REFERENCE SIGNS LIST 1 Vehicle, 2, 31, 51, 53 Occupant monitoring device, 3 Vehicle control system, 4 Occupant monitoring system, 5 Driving control system, 6 Braking / driving mechanism, 7 Steering mechanism, 8 Imaging device, 9 Output device, 10 Image acquisition unit, 11 Element detection unit, 12 Occupant state detection unit, 13 Action monitoring unit, 14 Memory unit, 15 Execution feasibility determination unit, 16 Command unit, 21 Driving control device, 32 Cancellation request unit, 52 Obstruction determination unit, 54 Calibration unit, 55 Reliability calculation unit

Claims

1. An occupant monitoring device comprising: an image acquisition unit that acquires images of occupants inside a vehicle from an imaging device mounted on the vehicle that images the interior of the vehicle; an element detection unit that detects elements of the face or body of the occupant using the image acquired by the image acquisition unit; an occupant state detection unit that uses the elements detected by the element detection unit to detect an inattentive state indicating that the occupant is in a distracted state or a state of reduced alertness; an operation monitoring unit that monitors whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit that uses the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that executes braking / driving control of the vehicle.

2. The passenger monitoring device according to claim 1, wherein the elements include facial features and head orientation.

3. The passenger monitoring device according to claim 2, wherein the facial features include eyes, a nose, and a mouth.

4. The occupant monitoring device according to claim 3, wherein the execution feasibility determination unit determines that the driving assistance function can be executed by the driving control device if the occupant state detection unit can detect the eyes but cannot detect the mouth among the elements.

5. The occupant monitoring device according to claim 1, wherein the elements are multiple, and the execution feasibility determination unit determines that the driving assistance function cannot be executed by the driving control device if the occupant state detection unit cannot detect more than half of the elements.

6. An occupant monitoring device as described in claim 1, further comprising a release request unit that, when the execution feasibility determination unit determines that the driving control device cannot execute the driving assistance function, outputs a signal to the driving control device requesting the release of the suppression of the driving assistance function by the driving control device based on the detection result of the occupant state detection unit.

7. The occupant monitoring device of claim 6, wherein the release request unit outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device when the time the vehicle has been continuously stopped after the driving assistance function by the driving control device has been suppressed is equal to or longer than a predetermined stopping reference time, and when the detection result of the occupant state detection unit when the vehicle is moving after having been stopped for equal to or longer than the stopping reference time indicates that the occupant is not in an inattentive state for equal to or longer than a predetermined first attention duration time.

8. The occupant monitoring device of claim 6, wherein the release request unit outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device when the time the vehicle has been continuously stopped after the driving assistance function by the driving control device has been suppressed is less than a predetermined stopping reference time, and when the detection result of the occupant state detection unit when driving after the driving assistance function by the driving control device has been suppressed indicates that the occupant is not in an inattentive state for more than a predetermined second attention duration time.

9. An occupant monitoring device as described in claim 6, comprising: a memory unit that stores factors that cause the execution feasibility determination unit to determine that it is necessary for the driving control device to suppress the driving assistance function; and a command unit that, when the release request unit outputs a signal to the driving control device requesting the release of suppression of the driving assistance function by the driving control device, issues at least one of a command to an output device mounted on the vehicle to notify the driver of the factor based on the factor stored in the memory unit and a command to the occupant status detection unit to eliminate the factor, 10. An occupant monitoring device as described in claim 9, further comprising an occupant obstruction determination unit that determines whether or not there is an obstruction that obstructs the imaging of the occupant between the imaging device and the occupant, wherein the execution feasibility determination unit determines that suppression of the driving assistance function is necessary when the occupancy determination unit determines that the obstruction exists, and the command unit outputs a command to the output device to issue a notification indicating the presence of the obstruction when the execution feasibility determination unit determines that suppression of the driving assistance function by the driving control device is necessary because of the presence of the obstruction.

11. The occupant monitoring device according to claim 10, wherein the release request unit outputs a signal to the driving control device requesting the release of the suppression of the driving assistance function by the driving control device when the occlusion determination unit determines that the occlusion has been removed.

12. The occupant monitoring device of claim 9, wherein the command unit outputs a command to the output device to notify the driver that the driving assistance function of the driving control device has been suppressed due to the driver's inattention when the execution feasibility determination unit determines that the suppression of the driving assistance function of the driving control device is necessary due to the driver's inattention.

13. An occupant monitoring device as described in claim 9, further comprising a calibration unit that executes a calibration process which is a process of correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit, wherein the execution feasibility determination unit determines that it is necessary to suppress the driving assistance function by the driving control device when the occupant state detection unit detects that the occupant is in an inattentive state, and the command unit outputs a command to the calibration unit to execute the calibration process when the inattentive state is the cause of the execution feasibility determination unit's determination that it is necessary to suppress the driving assistance function by the driving control device.

14. An occupant monitoring device as described in claim 9, comprising: a calibration unit that executes a calibration process, which is a process of correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit; and a reliability calculation unit that calculates the reliability of the detection of an inattentive state by the occupant state detection unit, wherein the command unit outputs a command to the calibration unit to execute the calibration process when the execution feasibility determination unit determines that the driving assistance function of the driving control device needs to be suppressed because of the inattentive state and the reliability is lower than a predetermined threshold value.

15. The occupant monitoring device according to claim 14, wherein the command unit outputs a command to the calibration unit to execute a calibration process when the execution feasibility determination unit determines that the driving assistance function by the driving control device needs to be suppressed because of the inattention state and the reliability is lower than a predetermined threshold, and outputs a command to the occupant state detection unit to detect the occupant's inattention state using results of previous executions by the calibration unit when the execution feasibility determination unit determines that the driving assistance function by the driving control device needs to be suppressed because of the inattention state and the reliability is equal to or higher than a predetermined threshold.

16. A vehicle control system comprising: an occupant monitoring device according to any one of claims 1 to 15; and a driving control device that controls the driving of the vehicle, wherein when the occupant monitoring device determines that it is necessary to suppress a driving assistance function by the driving control device, it outputs a signal to the driving control device that the driving assistance function cannot be executed based on the detection result of the occupant state detection unit, and when the driving control device acquires the signal that it is impossible to execute a driving assistance function by the driving control device, it outputs a signal to a steering mechanism or braking / driving mechanism mounted on the vehicle to cancel the suppression of the driving assistance function by the driving control device.

17. An occupant monitoring method comprising the steps of: an image acquisition unit acquiring an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing images of the interior of the vehicle; an element detection unit detecting elements of the face or body of the occupant using the image acquired by the image acquisition unit; an occupant state detection unit detecting an inattentive state indicating that the occupant is in a distracted state or a state of reduced alertness using the elements detected by the element detection unit; an operation monitoring unit monitoring whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit using the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that executes braking and driving control of the vehicle.

18. A program that causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle that captures images of the interior of the vehicle; an element detection unit detects elements of the face or body of the occupant using the image acquired by the image acquisition unit; an occupant state detection unit uses the elements detected by the element detection unit to detect an inattentive state indicating that the occupant is in a state of absent-mindedness or reduced alertness; an operation monitoring unit monitors whether the occupant state detection unit has detected the elements; and an execution feasibility determination unit uses the results of the operation monitoring unit to determine whether a driving assistance function can be executed by a driving control device that executes braking and driving control of the vehicle.

Images