Occupant-monitoring device, vehicle control system, occupant-monitoring method, and program

Abstract

The purpose of the present invention is to provide an occupant-monitoring device with which it is possible to suppress deterioration of user convenience.　This occupant-monitoring device (2) comprises: an image acquisition unit (10) that acquires, from an imaging device (8), a captured image of an occupant in a vehicle (1); an occupant state detection unit (11) that uses the captured image acquired by the image acquisition unit (10) to detect a careless state indicating that the occupant is in an inattention state or a state of low arousal; a suppression determination unit (12) that, when it is detected that the occupant is in a careless state by the occupant state detection unit (11), determines whether or not it is necessary to suppress a driving assistance function by an operation control device (21) that executes braking / driving control of the vehicle (1); and a cancellation request unit (14) that, when it is determined that the suppression determination unit (12) needs to suppress the driving assistance function, outputs to the operation control device (21) a signal requesting cancellation of the suppression of the driving assistance function by the operation control device (21) on the basis of the detection result of the occupant state detection unit (11).

Description

Occupant monitoring device, vehicle control system, occupant monitoring method, and program

[0001] The present disclosure relates to an occupant monitoring device, a vehicle control system, an occupant monitoring method, and a program.

[0002] Prior art discloses a vehicle control device that can resume automatic driving control after a driver abnormality stopping assistance function is activated during automatic driving control, only if the driver's behavior satisfies certain restrictive conditions related to the vehicle's driving control (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-140423

[0004] However, in the above-mentioned conventional technology, there is no consideration given to releasing the suppression of the driving assistance function depending on the state of the occupant detected using an image of the occupant, and there was a problem that once the driving assistance function was suppressed, user convenience was reduced.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an occupant monitoring device and a vehicle control system that suppress a decrease in user convenience.

[0006] The occupant monitoring device disclosed herein includes an image acquisition unit that acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle that images the interior of the vehicle; an occupant state detection unit that uses the image acquired by the image acquisition unit to detect an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness; a suppression determination unit that determines, when the occupant state detection unit detects that the occupant is in an inattentive state, that it is necessary to suppress the driving assistance function of a driving control device that executes braking and driving control of the vehicle; and a release request unit that, when the suppression determination unit determines that it is necessary to suppress the driving assistance function of the driving control device, outputs a signal to the driving control device to request that suppression of the driving assistance function of the driving control device be released based on the detection result of the occupant state detection unit.

[0007] In addition, the vehicle control system according to the present disclosure includes the above-mentioned occupant monitoring device and a driving control device that controls the driving of the vehicle, and when the occupant monitoring device determines that it is necessary to suppress the driving assistance function, it outputs a signal to the driving control device requesting the cancellation of the suppression of the driving assistance function based on the detection result of the occupant state detection unit, and when the driving control device acquires a signal requesting the cancellation of the suppression of the driving assistance function by the driving control device, it outputs a signal to the steering mechanism or braking / driving mechanism installed in the vehicle to cancel the suppression of the driving assistance function by the driving control device.

[0008] In addition, the occupant monitoring method according to the present disclosure includes the steps of: an image acquisition unit acquiring an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing images of the interior of the vehicle; an occupant state detection unit detecting an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness using the image acquired by the image acquisition unit; a suppression determination unit determining whether or not it is necessary to suppress the driving assistance function of the driving control device that executes braking and driving control of the vehicle when the detection result of the occupant state detection unit determines that it is necessary to suppress the driving assistance function of the driving control device; and a release request unit outputting a signal to the driving control device, based on the detection result of the occupant state detection unit, to request release of the suppression of the driving assistance function of the driving control device when the suppression determination unit determines that it is necessary to suppress the driving assistance function of the driving control device.

[0009] In addition, the program of the present disclosure causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle that captures images of the interior of the vehicle; an occupant state detection unit uses the image acquired by the image acquisition unit to detect an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness; a suppression determination unit determines whether it is necessary to suppress the driving assistance function by the driving control device that executes braking and driving control of the vehicle when the detection result of the occupant state detection unit indicates an inattentive state; and a release request unit outputs a signal to the driving control device based on the detection result of the occupant state detection unit to request release of the suppression of the driving assistance function by the driving control device when the suppression determination unit determines that it is necessary to suppress the driving assistance function.

[0010] The occupant monitoring device according to the present disclosure can prevent a decrease in user convenience. Furthermore, the vehicle control system according to the present disclosure can prevent a decrease in user convenience. Furthermore, the occupant monitoring method according to the present disclosure can prevent a decrease in user convenience. Furthermore, the program according to the present disclosure can prevent a decrease in user convenience.

[0011] 1 is a block diagram showing a portion of a vehicle equipped with an occupant monitoring device according to a first embodiment of the present disclosure. 2 is a block diagram showing a vehicle control system equipped with an occupant monitoring device according to a first embodiment of the present disclosure. 3 is a block diagram showing an occupant monitoring device according to a first embodiment of the present disclosure. 4 is a diagram for explaining a plurality of posture imbalance types. 5 is a flowchart showing an example of operation of the occupant monitoring device according to the first embodiment of the present disclosure. 6 is a diagram showing an example of a hardware configuration of the occupant monitoring device according to the first embodiment of the present disclosure. 7 is a diagram showing an example of a hardware configuration of the occupant monitoring device according to the first embodiment of the present disclosure. 8 is a block diagram showing an occupant monitoring device according to a second embodiment of the present disclosure. 9 is a flowchart showing an example of operation of the occupant monitoring device according to the second embodiment of the present disclosure. 10 is a block diagram showing an occupant monitoring device according to a third embodiment of the present disclosure. 11 is a flowchart showing an example of operation of the occupant monitoring device according to the third embodiment of the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the drawings are schematic, and the relative sizes and positions shown in different drawings are not necessarily limited to those described and may be changed as appropriate. In the following description, similar components are denoted by the same reference numerals, and their names and functions are assumed to be the same or similar. Therefore, detailed descriptions thereof may be omitted.

[0013] Embodiment 1. Figure 1 is a block diagram showing a portion of a vehicle 1 equipped with an occupant monitoring device 2 according to this embodiment. The vehicle 1 is equipped with a vehicle control system 3. The occupant monitoring device 2 constitutes a portion of the vehicle control system 3. The vehicle control system 3 includes an occupant monitoring system 4, a driving control system 5, a braking / driving mechanism 6, and a steering mechanism 7. The occupant monitoring system 4 includes the occupant monitoring device 2, an imaging device 8, and an output device 9. The occupant monitoring device 2 constitutes a portion of the occupant monitoring system 4.

[0014] The imaging device 8 is mounted on the vehicle 1. The imaging device 8 is a camera provided on a movable part inside the vehicle 1, and captures images of the interior of the vehicle 1. The imaging device 8 captures an image of an imaging area that spreads from the installation position at a preset camera angle of view. The imaging device 8 is installed so that the imaging area includes the faces of the occupants inside the vehicle 1, who are the imaging targets. The imaging device 8 captures images of the occupants inside the vehicle 1 at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured images to the occupant monitoring device 2. The captured images may be still images, moving images, or both still images and moving images.

[0015] The output device 9 includes at least one of a display, a speaker, and a vibrator. The display is configured, for example, by a liquid crystal display, an organic EL (Electro-Luminescence) display, or a HUD (Head-Up Display). The display is provided, for example, on the dashboard of the vehicle 1. The speaker is provided, for example, on the dashboard of the vehicle 1. The vibrator is provided, for example, on the steering wheel of the vehicle 1 or on the driver's seat of the vehicle 1.

[0016] The occupant monitoring device 2 monitors the state of the occupant, and when the occupant is in an inattentive state indicating a distracted state or a low level of alertness, outputs a command to the output device 9 to notify the driver that the driving assistance function has been suppressed. Note that the output device 9 may also be configured to output a command to notify the driver in advance of the suppression of the driving assistance function.

[0017] As shown in Figure 1, the occupant monitoring device 2 includes an image acquisition unit 10, an occupant state detection unit 11, a suppression judgment unit 12, a memory unit 13, a release request unit 14, and a command unit 15, each of which is connected to a communication bus 16, and data can be sent and received via the communication bus 16.

[0018] The image acquisition unit 10 acquires an image of an occupant in the vehicle from the imaging device 8 and outputs the acquired image to the occupant state detection unit 11 .

[0019] Next, we will explain the configuration of the driving control system 5 that provides driving assistance using the detection results of the occupant monitoring system 4. Figure 2 is a block diagram showing the vehicle control system 3 of the vehicle 1. As shown in Figure 2, the driving control system 5 is configured to be able to communicate with the occupant monitoring system 4 wirelessly or via a wire.

[0020] The driving control system 5 includes a driving control device 21 that controls the steering mechanism 7 and braking / driving mechanism 6 of the vehicle 1 by outputting control signals based on the detection results of the occupant monitoring system 4, thereby controlling the driving of the vehicle 1. The driving control system 5 also includes a map information storage device (not shown) that stores map information, a surrounding condition monitoring device 22 that monitors the surrounding conditions of the vehicle, and a vehicle state acquisition device 23 that acquires information indicating the state of the vehicle 1. The occupant monitoring system 4, the driving control device 21, the surrounding condition monitoring device 22, and the vehicle state acquisition device 23 included in the driving control system 5 are each connected to a communication bus 16, and are capable of transmitting and receiving data via the communication bus 16.

[0021] The braking / driving mechanism 6 is a mechanism for controlling the traveling speed of the vehicle 1 and for switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 61. The steering mechanism 7 is a mechanism for determining the traveling direction of the vehicle 1, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 71. The braking / driving actuator 61 that controls the braking / driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, and the steering actuator 71 that controls the steering mechanism 7 is composed of, for example, an EPS (Electric Power Steering) motor, and the like.

[0022] The surrounding condition monitoring device 22 monitors the conditions around the vehicle 1 and includes a GPS (Global Positioning System) receiver 221, an on-board communication device 222, an external sensor 223, and a navigation system 224. The GPS receiver 221 receives signals transmitted from GPS positioning satellites to detect the current location of the vehicle 1. The on-board communication device 222 communicates with communication devices of other vehicles or communication devices installed on the road to acquire information on the positions of other vehicles or pedestrians, traffic information, etc. Here, traffic information includes, for example, congestion information, traffic regulation information, construction zone information, etc. The on-board communication device 222 can also perform vehicle-to-vehicle communication via wireless communication with on-board communication devices of other vehicles located around the vehicle 1. In addition, the on-board communication device 222 may perform mobile communication with a base station external to the vehicle 1. The external sensor 223 is, for example, composed of at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and detects the positions of vehicles, pedestrians, obstacles, etc. present around the vehicle 1 or their distance from the vehicle 1. The navigation system 224 calculates a route from the current position of the vehicle 1 to a destination and provides guidance along the calculated route. The navigation system 224 has a display unit composed of, for example, a liquid crystal display, and is housed in the instrument panel. The navigation system 224 also has an operation unit such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. Note that the operation unit of the navigation system 224 may be composed of a microphone or the like to be able to accept operations via voice uttered by the occupant.

[0023] The vehicle state acquisition device 23 acquires information indicating the state of the vehicle 1 and includes a steering angle sensor 231, a vehicle speed sensor 232, a steering torque sensor 233, an accelerator position sensor 234, and a brake position sensor 235. The steering angle sensor 231 is provided, for example, on an EPS (Electric Power Steering) motor or a steering wheel and detects the steering angle of the vehicle 1. The vehicle speed sensor 232 is provided, for example, on a wheel and detects the traveling speed of the vehicle 1. The steering torque sensor 233 is provided, for example, on the steering wheel and detects the magnitude of the steering wheel operation force applied by the driver. The accelerator position sensor 234 detects the amount of accelerator pedal depression by the driver. The brake position sensor 235 detects the amount of brake pedal depression by the driver.

[0024] The driving control device 21 of the driving control system 5 controls the driving of the vehicle 1. The driving control device 21 controls a braking / driving actuator 61 or a steering actuator 71 mounted on the vehicle 1 to assist the driver in driving. The driving control device 21 performs automatic driving control of the vehicle 1, for example, by outputting a signal to the braking / driving actuator 61 or the steering actuator 71. The driving control device 21 performs braking / driving control of the vehicle 1, such as by controlling the braking / driving actuator 61, which is composed of an electronically controlled throttle and a brake actuator, to activate the brakes to decelerate or stop the vehicle 1. The driving control device 21 performs steering control of the vehicle 1, such as by controlling the steering actuator 71, which is composed of an EPS motor, to maintain the lane in which the vehicle 1 is traveling. Furthermore, the driving control device 21 executes multiple driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function is referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function is referred to as LKA (Lane Keeping Assist).

[0025] When the driving control device 21 executes ACC, it controls the traveling speed of the vehicle 1 by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device 22. If no vehicle ahead is detected, the driving control device 21 causes the vehicle 1 to travel at a constant speed at a target speed set in advance by the driver or the like. On the other hand, if a vehicle ahead is detected, the ACC causes the vehicle 1 to follow the vehicle ahead while maintaining a distance from the vehicle ahead.

[0026] Furthermore, when the driving control device 21 executes LKA, the driving control device 21 controls the steering force and steering holding force based on shape information of the lane markings in the traveling direction acquired from the surrounding conditions monitoring device 22. LKA causes the vehicle 1 to travel along the lane by applying a steering force to the steering wheel in a direction that prevents the vehicle from approaching the lane markings. Note that road information output by the surrounding conditions monitoring device 22 may be used for vehicle control by ACC and LKA.

[0027] Furthermore, when the driving control device 21 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the vehicle 1. When the automatic evacuation control is started, the driving control device 21 causes the surrounding condition monitoring device 22 to search for an evacuation location where the vehicle 1 should be stopped. Then, the driving control device 21 moves the vehicle 1 to the evacuation location set by the surrounding condition monitoring device 22 and stops the vehicle 1 there. Note that, on expressways, the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle 1 is traveling, and on ordinary roads, it may be a location that avoids a shoulder outside the lane in which the vehicle 1 is traveling, as well as intersections, railroad crossings, sidewalks, and locations where there is a high possibility of the presence of moving objects such as other vehicles, trains, and pedestrians.

[0028] 3 is a block diagram showing the occupant monitoring device 2. The image acquisition unit 10 outputs the captured image acquired from the imaging device 8 to an eye opening degree determination unit 111, a gaze detection unit 112, a face direction detection unit 113, a drowsiness detection unit 114, and a head position detection unit 115.

[0029] The occupant state detection unit 11 detects the occupant's distracted state or alert state using the captured image acquired by the image acquisition unit 10. That is, the occupant state detection unit 11 detects the inattention state indicating that the occupant is in a distracted state or has a low level of alertness using the captured image acquired by the image acquisition unit 10. The occupant state detection unit 11 includes an eye opening degree determination unit 111, a gaze detection unit 112, a face direction detection unit 113, a drowsiness detection unit 114, a head position detection unit 115, an eye state determination unit 116, an aside-looking determination unit 117, a dozing determination unit 118, and a posture determination unit 119.

[0030] The eye-opening degree determination unit 111 determines the degree of eye opening of the occupant based on the captured image acquired by the image acquisition unit 10. Specifically, the eye-opening degree determination unit 111 detects the occupant's upper eyelid and lower eyelid using a known image recognition technique on the captured image. For example, an area in the captured image where the occupant's face may be present (hereinafter referred to as a "face detection area") is set in advance, and the eye-opening degree determination unit 111 performs known edge detection on the face detection area of ​​the captured image to extract feature points indicating the occupant's upper eyelid and lower eyelid. Next, the eye-opening degree determination unit 111 calculates, for example, the vertical distance between the occupant's upper eyelid and lower eyelid (hereinafter referred to as "eyelid distance") in the captured image, starting from left to right, and calculates the maximum eyelid distance. Then, the eye-opening degree determination unit 111 determines the degree of eye opening based on the ratio between the calculated maximum eyelid distance and a reference value for the eye-opening degree. The reference value of the eye openness is set to, for example, the average value of the eye openness of an adult of average build who is awake. The reference value of the eye openness may be, for example, the average value of the eye openness of the occupant for a predetermined time after the occupant gets into the vehicle, or the average value of all the eye openness determined by the eye openness determination unit up to now. The above-described method of determining the eye openness is merely an example. The eye openness determination unit 111 may determine the eye openness using various known algorithms. The eye openness determination unit 111 outputs information related to the determination result of the eye openness of the occupant (hereinafter referred to as "eye openness information") to the eye state determination unit 116. The eye openness information is, for example, information that associates the eye openness of the occupant with the captured image acquired by the image acquisition unit.

[0031] The gaze detection unit 112 detects the gaze of the occupant based on the captured image acquired by the image acquisition unit 10. The gaze detection unit 112 may detect the gaze direction of the occupant using a known image recognition technique on the captured image. For example, if the imaging device 8 is a visible light camera, the gaze detection unit 112 detects the inner corners of the occupant's eyes and irises in the captured image and detects the gaze direction of the occupant from the positional relationship between the inner corners of the eyes and the irises. If the imaging device 8 is an infrared camera, the gaze detection unit 112 detects the gaze direction of the occupant from the positional relationship between the pupil and a Purkinje image obtained by reflecting a near-infrared point light source from the infrared camera. Note that these are merely examples, and the gaze detection unit 112 may detect the gaze direction of the occupant using various known algorithms. The gaze direction of the occupant is expressed as an angle with respect to a predetermined reference direction. For example, the gaze direction of the occupant is expressed as a gaze angle that indicates how many degrees the gaze direction has rotated horizontally and vertically from the state in which the occupant is looking straight ahead while sitting in the seat. The gaze detection unit 112 outputs information relating to the detected gaze of the occupant (hereinafter referred to as "gaze information") to the eye state determination unit 116. The gaze information is, for example, information in which information indicating the gaze direction of the occupant is associated with a captured image. If the gaze detection unit 112 is unable to detect the gaze of the occupant, it outputs gaze information indicating that the gaze of the occupant could not be detected to the eye state determination unit 116.

[0032] The face direction detection unit 113 detects the face direction of the occupant based on the captured image acquired by the image acquisition unit 10. The face direction detection unit 113 may detect the face direction of the occupant using a known image recognition technique on the captured image. For example, the face direction detection unit 113 may detect the face direction of the occupant using general pattern recognition based on the captured image. This is merely one example, and the face direction detection unit 113 may detect the face direction of the occupant using various known algorithms. The face direction of the occupant is expressed as an angle with respect to a predetermined reference direction. For example, the face direction of the occupant is expressed as a face direction angle that indicates how many degrees the face direction has rotated horizontally and vertically from a state in which the occupant is looking straight ahead in the seat. The face direction detection unit 113 outputs information regarding the detected face direction of the occupant (hereinafter referred to as "face direction information") to the eye state determination unit 116. The face direction detection unit 113 may also output the detected face direction information to the posture determination unit 119. The face direction information is, for example, information in which the face direction of the occupant is associated with a captured image.

[0033] The drowsiness detection unit 114 detects the drowsiness level of the occupant based on the captured image acquired by the image acquisition unit 10. The drowsiness detection unit 114 may detect the drowsiness level of the occupant using a known image recognition technique on the captured image. For example, the drowsiness detection unit 114 may determine the number of blinks of the occupant per a predetermined unit time based on the captured image. The drowsiness detection unit 114 detects the drowsiness level of the occupant based on the number of blinks per unit time. The drowsiness level corresponding to the number of blinks per unit time is determined in advance. For example, the drowsiness level may be set to a value such that a higher drowsiness level indicates greater drowsiness of the occupant, and a lower drowsiness level indicates a more awake occupant. The drowsiness detection unit 114 may detect the drowsiness level of the occupant using various known algorithms. Note that, although the drowsiness detection unit 114 detects the drowsiness level of the occupant based on the captured image in this example, this is merely an example. The drowsiness detection unit 114 may, for example, acquire biological information of the occupant, such as body temperature, from a biological sensor provided in the vehicle 1 and detect the drowsiness level of the occupant from the biological information. The drowsiness detection unit 114 outputs information related to the detected drowsiness level of the occupant (hereinafter referred to as "drowsiness level information") to the dozing determination unit 118. The drowsiness level information is, for example, information in which the drowsiness level of the occupant is associated with a captured image.

[0034] When the eye opening degree of the occupant determined by the eye opening degree determination unit 111 is less than a first threshold, the eye state determination unit 116 determines whether the occupant has their eyes downcast or closed, based on the gaze direction of the occupant detected by the gaze detection unit 112 before it was determined that the eye opening degree was less than the first threshold and the facial direction of the occupant detected by the facial direction detection unit. Specifically, when the eye opening degree determined by the eye opening degree determination unit 111 is less than the first threshold, if the gaze direction of the occupant detected by the gaze detection unit 112 is directed downward and the degree of eye opening becomes less than the first threshold after the facial direction of the occupant detected by the facial direction detection unit 113 is directed downward, the eye state determination unit 116 determines that the occupant has their eyes downcast. On the other hand, the eye state determination unit 116 determines that the occupant has their eyes closed if the gaze direction of the occupant detected by the gaze detection unit 112 is directed downward and the degree of eye opening does not become less than the first threshold after the occupant's facial direction detected by the face direction detection unit 113 has turned downward. The determination of whether the occupant has their eyes down or closed, performed by the eye state determination unit 116, is also referred to as a "downcast eyes determination." The eye state determination unit 116 determines whether the degree of eye opening of the occupant has become less than the first threshold based on the eye opening degree information output from the eye opening degree determination unit 111. Note that if the degree of eye opening of the occupant determined by the eye opening degree determination unit 111 is equal to or greater than the first threshold, the eye state determination unit 116 does not perform a downcast eyes determination and determines that the occupant has their eyes open. The downcast eyes determination process performed by the eye state determination unit 116 is described below.

[0035] Eye state determination unit 116 stores, in chronological order, the eye opening degree information output from eye opening degree determination unit 111, the gaze information output from gaze detection unit 112, and the facial direction information output from facial direction detection unit 113. The information stored in chronological order by eye state determination unit 116, in which the eye opening degree information, gaze information, and facial direction information are associated with each other, is referred to as "eye-related information."

[0036] When the eye state determination unit 116 determines that the degree of eye opening of the occupant has become less than the first threshold, it refers to the eye-related information and obtains the gaze information of the occupant before the degree of eye opening of the occupant became less than the first threshold.

[0037] Next, based on the acquired gaze information of the occupant, the eye state determination unit 116 calculates a trajectory of the occupant's gaze direction from the gaze direction of the occupant detected by the gaze detection unit 112 before it was determined that the degree of eye opening was less than the first threshold. The eye state determination unit 116 refers to the eye-related information to acquire facial direction information of the occupant before the degree of eye opening of the occupant became less than the first threshold. Based on the acquired facial direction information of the occupant, the eye state determination unit 116 calculates a trajectory of the occupant's facial direction from the facial direction of the occupant detected by the facial direction detection unit 113 before it was determined that the degree of eye opening was less than the first threshold.

[0038] The eye state determination unit 116 determines whether the calculated trajectory of the occupant's gaze direction is directed downward and whether the calculated trajectory of the occupant's facial direction is directed downward. Specifically, the eye state determination unit 116 determines whether the calculated trajectory of the occupant's gaze direction is directed toward a predetermined downward viewing area and whether the calculated trajectory of the occupant's facial direction is directed toward the downward viewing area. If the trajectory of the occupant's gaze direction is directed toward the downward viewing area, the eye state determination unit 116 determines that the occupant's gaze direction is directed downward. Furthermore, if the trajectory of the occupant's facial direction is directed toward the downward viewing area, the eye state determination unit 116 determines that the occupant's facial direction is directed downward. The downward viewing area is an area set below the image capture device 8 within the vehicle 1. The downward viewing area is set as an area where it is assumed that the occupant may be looking aside if, for example, the occupant's line of sight remains within the area while looking downward for a predetermined period of time. For example, the downward viewing area is set as an area including an area where devices that the occupant can operate below the imaging device 8, such as the display of a car navigation device, are located.

[0039] The eye state determination unit 116 determines that the occupant has their eyes downcast if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are pointing downward. In other words, the eye state determination unit 116 determines that the occupant has their eyes downcast if the occupant's line of sight and the occupant's facial direction tend to be pointing downward and the degree to which the occupant's eyes are open is less than a first threshold.

[0040] The eye state determination unit 116 determines that the occupant has their eyes closed if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are not pointing downward. In other words, the eye state determination unit 116 determines that the occupant has their eyes closed unless the degree of eye opening of the occupant is less than the first threshold while the occupant's line of sight and the occupant's facial direction tend to point downward.

[0041] The eye state determination unit 116 outputs information indicating whether the occupant has their eyes downcast, closed, or open (hereinafter referred to as "eye state information") to the inattentive driving determination unit 117 and the drowsy driving determination unit 118.

[0042] When the eye state determination unit 116 outputs eye state information indicating that the occupant is looking down, in other words, when the eye state determination unit 116 determines that the occupant is looking down, the inattentiveness determination unit 117 determines whether the occupant is looking aside (hereinafter referred to as "inattentiveness determination") based on whether a preset condition (hereinafter referred to as "condition for inattentiveness determination") is satisfied. An example of the condition for inattentiveness determination is that "the occupant's downcast eyes continue for a preset time period (hereinafter referred to as "time period for inattentiveness determination") or more."

[0043] When the eye state determination unit 116 outputs eye state information indicating that the occupant is looking down, the inattentive driving determination unit 117 counts up the time (hereinafter referred to as the "time after downcast eye determination"). When the counted up time after downcast eye determination is equal to or greater than the inattentive driving determination time, the inattentive driving determination unit 117 determines that the occupant's downcast eye state has continued for equal to or greater than the inattentive driving determination time, and determines that the occupant is looking aside. The inattentive driving determination unit 117 stores the counted up time after downcast eye determination. On the other hand, when the inattentive driving determination unit 117 determines that the occupant's downcast eye state has not continued for equal to or greater than the inattentive driving determination time, it determines that the occupant is not looking aside. When the inattentive driving determination unit 117 determines that the occupant is looking aside as a result of the inattentive driving determination, it outputs information indicating that the occupant is looking aside to the suppression determination unit 12 and the memory unit 13.

[0044] When the eye state determination unit 116 outputs eye state information indicating that the occupant has closed eyes, in other words, when the eye state determination unit 116 determines that the occupant has closed eyes, the drowsiness determination unit 118 determines whether the occupant is dozing off (hereinafter also referred to as "drowsiness determination") based on the occupant's drowsiness level detected by the drowsiness detection unit 114. The drowsiness determination unit 118 may identify the occupant's drowsiness level from the drowsiness level information output from the drowsiness detection unit 114. Specifically, the drowsiness determination unit 118 determines that the occupant is dozing off when a state (hereinafter referred to as a "drowsiness state") in which the occupant's drowsiness level is equal to or higher than a preset threshold (hereinafter referred to as a "drowsiness determination threshold") continues for a preset time (hereinafter referred to as a "drowsiness determination time") or longer. The doze determination unit 118 determines that the occupant is in a drowsy state when the eye state determination unit 116 outputs eye state information indicating that the occupant has closed eyes and the occupant's drowsiness level detected by the drowsiness detection unit 114 is equal to or greater than a drowsiness determination threshold. When the doze determination unit 118 determines that the occupant is in a drowsy state, it counts up a time (hereinafter referred to as the "post-drowsiness determination time"). The doze determination unit 118 stores the counted-up post-drowsiness determination time. When the counted-up post-drowsiness determination time is equal to or greater than the doze determination time, the doze determination unit 118 determines that the occupant's drowsy state has continued for equal to or greater than the doze determination time, and determines that the occupant is dozing. When the doze determination unit 118 determines that the occupant is dozing as a result of the doze determination, it outputs information indicating that the occupant is dozing to the suppression determination unit 12 and the storage unit 13.

[0045] The head position detection unit 115 detects the head position of the occupant in real space based on the occupant's face in the captured image. The head position of the occupant in the captured image is indicated, for example, by the center of the occupant's eyebrows. The head position detection unit 115 detects, for example, a point in real space corresponding to the center of the occupant's eyebrows in the captured image as the occupant's head position. Note that this is merely an example, and the position of the occupant's head in the captured image may be indicated, for example, by the center of the occupant's facial area or the center of a line connecting the inner corners of the occupant's eyes. In this case, the head position detection unit 115 detects, for example, a point in real space corresponding to the center of the occupant's facial area in the captured image or the center of a line connecting the inner corners of the occupant's eyes as the occupant's head position in real space. The occupant's facial area is, for example, the smallest rectangle that surrounds the outline of the occupant's face. The occupant's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle in the captured image.

[0046] The head position detection unit 115 may detect the head position of the occupant using, for example, a known coordinate transformation technique that converts points on a captured image into points in real space. The head position of the occupant is expressed, for example, by coordinates in real space. The head position detection unit 115 outputs information related to the detected head position of the occupant (hereinafter referred to as "head position information") to the posture determination unit 119. The head position information includes coordinate information of the occupant's head position.

[0047] The posture determination unit 119 determines which of a plurality of predetermined posture types the posture of the vehicle occupant corresponds to based on the captured image. Occupants may lose their posture due to symptoms such as loss of consciousness or epilepsy. Figure 4 (partially modified from "Basic Design Document for Automatic Occupant Abnormality Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) shows the types of posture loss.

[0048] As shown in Figure 4, there are multiple modes of posture imbalance. Figure 4 shows "head down" in which the occupant leans forward and continues to have their face close to the steering wheel, "head down" in which the occupant continues to have their face facing down, "backward lean" in which the occupant's upper body tilts backward and continues to have their face facing up, "backward lean" in which the occupant's upper body arches up and continues to have their face facing up, "head tilted to the side" in which the occupant's face tilts to the left or right, "sideways lean" in which the occupant's upper body tilts to the left or right and their face tilts in the same direction, and "leaning to the side" in which the occupant's upper body tilts to the left or right. In this disclosure, "leaning face down," "head down," "arching backward," and "arching backward" are classified as posture type 1 (first posture type), and "head tilted to the side only," "head tilted to the side," and "leaning to the side" are classified as posture type 2 (second posture type). Posture type 1 is a posture in which the occupant's head, face, or upper body is tilted forward and along the vehicle, and posture type 2 is a posture in which the occupant's head, face, or upper body is tilted to the left side relative to the fore-and-aft direction of the vehicle.

[0049] The posture determination unit 119 determines whether the occupant is experiencing poor posture using the facial direction input from the facial direction detection unit 113 and the head position input from the head position detection unit 115. For example, the type of poor posture is determined by determining whether the difference between the occupant's head position and facial direction obtained from the current image frame and a reference posture is equal to or greater than a predetermined threshold. As the reference posture, for example, the average value of the occupant's head position and facial direction obtained when the vehicle speed is 25 km / h or higher and the steering angle is within a range of ±20 degrees over a predetermined period of time (e.g., 3 seconds) can be used.

[0050] When the posture determination unit 119 determines that the occupant has lost their posture, it counts up the time (hereinafter referred to as the "time after posture loss determination"). The posture determination unit 119 stores the counted-up time after posture loss determination. When the counted-up time after posture loss determination is equal to or greater than the posture loss determination time, the posture determination unit 119 determines that the state in which the occupant has lost their posture has continued for equal to or greater than the posture loss determination time, and determines that the occupant's posture is abnormal. When the posture determination unit 119 determines whether the occupant's posture is abnormal and determines that the occupant's posture is abnormal, it outputs information that the occupant's posture is abnormal to the suppression determination unit 12 and the memory unit 13.

[0051] The inattention state, which indicates a state of absent-mindedness or low level of alertness, includes the occupant looking away, the occupant falling asleep, and the occupant having an abnormal posture.

[0052] The suppression determination unit 12 uses the detection result of the occupant state detection unit 11 to determine whether or not it is necessary to suppress the driving assistance function of the driving control device 21 that executes braking / driving control of the vehicle 1. The suppression determination unit 12 acquires information that the occupant is in an inattentive state from the inattentiveness determination unit 117. The suppression determination unit 12 also acquires information that the occupant is in an inattentive state from the drowsiness determination unit 118. The suppression determination unit 12 also acquires information that the occupant's posture is abnormal from the posture determination unit 119.

[0053] The suppression determination unit 12 can determine that a notification should be issued to the occupant when it acquires information that the occupant is in an inattentive state, information that the occupant is in a dozing state, or information that the occupant's posture is abnormal from the inattentiveness determination unit 117, the drowsiness determination unit 118, or the posture determination unit 119. Furthermore, when it determines that a notification should be issued to the occupant, the suppression determination unit 12 can output a determination result that a notification should be issued to the command unit 15.

[0054] The suppression determination unit 12 determines that suppression of the driving assistance function is necessary when the occupant state detection unit 11 detects that the occupant is in an inattentive state. More specifically, the suppression determination unit 12 can determine that suppression of the driving assistance function is necessary when the suppression determination unit 12 acquires information indicating that the occupant is in an inattentive state, information indicating that the occupant is in a drowsy state, or information indicating that the occupant's posture is abnormal a predetermined number of times or more from the inattentiveness determination unit 117, the drowsiness determination unit 118, or the posture determination unit 119. In other words, the suppression determination unit 12 can determine that suppression of the driving assistance function is necessary when the number of times it has been determined that a notification should be issued is a predetermined number or more. Note that, in determining whether suppression of the driving assistance function is necessary, it is not necessarily necessary to determine that a notification should be issued and to issue a notification.

[0055] Furthermore, when the suppression determination unit 12 acquires information indicating that the occupant is looking away, that the occupant is dozing, or that the occupant's posture is abnormal a predetermined number of times or more, i.e., when it determines that the driving assistance function needs to be suppressed, the suppression determination unit 12 can output a signal requesting suppression of the driving assistance function to the driving control device 21. The predetermined number of times can be set to any number of times. For example, the predetermined number of times can be set to three times.

[0056] When the suppression determination unit 12 determines that suppression of the driving assistance function is necessary, it outputs a signal to the driving control device 21 requesting that the driving control device 21 suppress the driving assistance function. When the driving control device 21 receives the signal requesting suppression of the driving assistance function, it suppresses the driving assistance function.

[0057] The suppression determination unit 12 outputs factors that determine that suppression of the driving assistance function is necessary to the storage unit 13. Factors that determine that suppression of the driving assistance function is necessary include, for example, the occupant being inattentive, the occupant being dozing, or the occupant having an abnormal posture. In other words, the occupant being in a state of inattention. The suppression determination unit 12 also outputs factors that determine that suppression of the driving assistance function is necessary to the release request unit 14.

[0058] The storage unit 13 acquires information that the occupant is in an inattentive state, information that the occupant is in a drowsy state, and information that the occupant's posture is abnormal from the inattentiveness determination unit 117, the drowsiness determination unit 118, and the posture determination unit 119. The storage unit 13 also acquires factors that determine that suppression of the driving assistance function is necessary from the suppression determination unit 12. The storage unit 13 stores the factors that the suppression determination unit 12 determines that suppression of the driving assistance function is necessary.

[0059] When the suppression determination unit 12 determines that suppression of the driving assist function by the driving control device 21 is necessary, the release request unit 14 outputs a signal to the driving control device 21 requesting release of suppression of the driving assist function by the driving control device 21 based on the detection result of the occupant state detection unit 11. In other words, when it can be determined based on the detection result of the occupant state detection unit 11 that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assist function is necessary has been removed, the release request unit 14 outputs a signal to the driving control device 21 requesting release of suppression of the driving assist function by the driving control device 21.

[0060] The release request unit 14 requests the release of the suppression of the driving assistance function by the driving control device 21 based on the detection results of the occupant state detection unit 11, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed.

[0061] The release request unit 14 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function by the driving control device 21 if the time that the vehicle 1 has been continuously stopped after the driving assistance function has been suppressed is equal to or longer than a predetermined stopping reference time, or if the detection result of the occupant state detection unit 11 when the vehicle 1 is moving after being stopped for equal to or longer than the stopping reference time indicates that the occupant's state is not in an inattentive state, and this state continues for equal to or longer than a predetermined first attention duration time.

[0062] The stopping reference time is, for example, the time required for a rest to recover from fatigue or other factors that may have led to the occupant's inattentive state. The stopping reference time can be set to, for example, 15 minutes. The first attention duration is the time considered necessary to determine that the occupant has recovered from the inattentive state when the vehicle 1 has been stopped continuously for a period of time equal to or longer than the predetermined stopping reference time. The first attention duration can be set to, for example, 5 minutes.

[0063] If the vehicle 1 is continuously stopped for a period of time equal to or longer than a predetermined stopping reference time, it can be assumed that the occupant has had a sufficient rest and recovered from fatigue. Therefore, if the detection result of the occupant state detection unit 11 when the vehicle 1 is stopped for a period of time equal to or longer than the stopping reference time and then driven away indicates that the occupant is not in an inattentive state for a period of time equal to or longer than a predetermined first attention duration, the release request unit 14 can determine that the occupant has recovered from the inattentive state. In other words, it can be determined that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function was necessary has been eliminated.

[0064] The release request unit 14 determines that the occupant has escaped from an inattentive state and outputs a signal to the driving control device 21 requesting that the driving control device 21 release the suppression of the driving assistance function, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed when the vehicle 1 is continuously stopped for a period of time that is equal to or longer than a predetermined stopping reference time.

[0065] In addition, the release request unit 14 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function by the driving control device 21 if the time that the vehicle 1 has been continuously stopped after the driving assistance function has been suppressed is less than a predetermined stopping reference time, or if the detection result of the occupant state detection unit 11 when driving after the driving assistance function has been suppressed indicates that the occupant's state is not in an inattentive state for more than a predetermined second attention duration time.

[0066] The second attention duration is a time considered necessary to determine that the occupant has ceased to be inattentive when the vehicle 1 has been continuously stopped for less than a predetermined stopping reference time. When the vehicle 1 has been continuously stopped for less than the predetermined stopping reference time, it is considered that a longer time is necessary to determine that the occupant has ceased to be inattentive compared to when the vehicle 1 has been continuously stopped for more than the predetermined stopping reference time. Therefore, the second attention duration can be set to, for example, a time longer than the first attention duration. The second attention duration can be set to, for example, 120 minutes.

[0067] If the time that the vehicle 1 is continuously stopped is less than a predetermined stopping reference time, it is possible that the occupant has not been able to take a sufficient rest and recover from fatigue, etc. However, if the detection result of the occupant state detection unit 11 when driving after the driving assistance function is suppressed indicates that the occupant's state is not in an inattentive state for a predetermined second attention duration time or more, the release request unit 14 can determine that the occupant has recovered from the inattentive state. In other words, it can determine that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function was necessary has been removed.

[0068] The release request unit 14 determines that the occupant has escaped from an inattentive state and outputs a signal to the driving control device 21 requesting that the driving control device 21 release the suppression of the driving assistance function, thereby preventing a decrease in user convenience caused by the driving assistance function remaining suppressed when the vehicle 1 is continuously stopped for less than a predetermined stopping reference time.

[0069] In addition, the release request unit 14 can output information to the command unit 15 requesting the release of the suppression of the driving assistance function.

[0070] In addition, when the release request unit 14 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function by the driving control device 21, the release request unit 14 can reset the number of times that the suppression judgment unit 12 has received information that the occupant is looking away, that the occupant is dozing, or that the occupant's posture is abnormal.

[0071] When the driving control device 21 receives a signal requesting cancellation of the suppression of the driving support function from the cancellation request unit 14, the driving control device 21 cancels the suppression of the driving support function.

[0072] The command unit 15 can acquire information requesting the cancellation of suppression of the driving assistance function from the cancellation request unit 14. The command unit 15 can also acquire, from the storage unit 13, factors that are stored in the storage unit 13 and that cause the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary.

[0073] When the release request unit 14 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function, the command unit 15 can issue a command to the output device 9 installed in the vehicle 1 to send a notification indicating the cause based on the cause stored in the memory unit 13.

[0074] The factors that cause the suppression determination unit 12 to determine that suppression of the driving support function is necessary include inattention of the occupant, such as the occupant being inattentive, falling asleep, or having an abnormal posture. When the factor that causes the suppression determination unit 12 to determine that suppression of the driving support function is necessary is inattention, the command unit 15 outputs a command to the output device 9 to make a notification that the driving support function has been suppressed due to inattention of the occupant.

[0075] The command unit 15 outputs a command to the output device 9 to notify the driver that the driving support function has been suppressed due to the driver's inattention, thereby notifying the driver that the driving support function has been suppressed due to the driver's inattention. In other words, the driver can know that the driving support function has been suppressed due to the driver's inattention.

[0076] The output device 9 receives a command from the command unit 15 to issue a notification indicating that the driving assistance function has been suppressed due to an inattentional state of the occupant. The output device 9 outputs a notification indicating that the driving assistance function has been suppressed due to an inattentional state of the occupant.

[0077] Next, the operation of the occupant monitoring device 2 will be described. Figures 5 and 6 are flowcharts showing an example of the operation of the occupant monitoring device 2 according to the first embodiment. The occupant monitoring device 2 starts operation, for example, when the imaging device 8 is activated, and repeats processing operations while the vehicle 1 is traveling. The following description will be given assuming that the occupant state detection unit 11 is detecting an occupant at "Start (A)" in Figure 5.

[0078] In step ST101, the image acquisition unit 10 acquires a captured image from the imaging device 8. The image acquisition unit 10 outputs the acquired captured image to the occupant state detection unit 11.

[0079] In step ST102, the occupant state detection unit 11 detects the state of the occupant.

[0080] In step ST103, the occupant state detection unit 11 determines whether the occupant is in a state of absent-mindedness or a state of low alertness. If the occupant is in an inattentive state (ST103; YES), the process proceeds to step ST104. If the occupant is not in an inattentive state (ST103; NO), the process returns to step ST101. The occupant is detected to be in an inattentive state. Inattentive states that indicate an occupant is in an absent-minded state or a state of low alertness include the occupant being distracted, the occupant being dozing, or the occupant having an abnormal posture.

[0081] In step ST104, the suppression determination unit 12 determines whether the detection result of the occupant state detection unit 11 satisfies a predetermined condition. The predetermined condition may be, for example, that the occupant state detection unit 11 has detected an inattentive state, including the occupant being inattentive, the occupant being dozing, or the occupant having an abnormal posture, a predetermined number of times or more. In other words, the suppression determination unit 12 can determine that the predetermined condition is satisfied when the suppression determination unit 12 has obtained information indicating that the occupant is in an inattentive state a predetermined number of times or more.

[0082] If the detection result of the occupant state detection unit 11 satisfies a predetermined condition (ST104; YES), the suppression determination unit 12 determines that the driving assistance function needs to be suppressed, and proceeds to the processing of ST105. If the detection result of the occupant state detection unit 11 does not satisfy the predetermined condition (ST104; NO), the processing returns to step ST101. Note that when returning from step ST104 to the processing of step ST101, the suppression determination unit 12 may store the number of times that information indicating that the occupant is in an inattentive state in the storage unit 13, and when returning to the processing of step ST104 again, the number of times that information indicating that the occupant is in an inattentive state has been acquired may be added.

[0083] In step ST105, the suppression determination unit 12 outputs a signal to the driving control device 21 requesting the driving control device 21 to suppress the driving assistance function.

[0084] In step ST106, when the driving control device 21 receives a signal requesting suppression of the driving support function, the driving control device 21 suppresses the driving support function.

[0085] In step ST107, the release request unit 14 determines whether the time during which the vehicle 1 has been continuously stopped is equal to or longer than a predetermined stopping reference time. If the time during which the vehicle 1 has been continuously stopped is equal to or longer than the predetermined stopping reference time (ST107; YES), the process proceeds to step ST108. If the time during which the vehicle 1 has been continuously stopped is not equal to or longer than the predetermined stopping reference time (ST107; NO), that is, if the time during which the vehicle 1 has been continuously stopped is shorter than the predetermined stopping reference time, the process proceeds to step ST109.

[0086] The acquisition of the captured image in step ST101, the detection of the occupant's state in step ST102, and the determination of whether the occupant is in an inattentive state in step ST103 are performed continuously, for example, even when the vehicle 1 is traveling after being stopped. Furthermore, the processes of steps ST101 and ST102 may be performed even when the vehicle 1 is stopped.

[0087] Furthermore, when determining whether the occupant state is in an inattentive state, the occupant state detection unit 11 can measure the duration of time during which a state indicating that the occupant state is not in an inattentive state continues when the vehicle 1 is stopped for a stopping reference time or longer and then starts driving. Furthermore, when determining whether the occupant state is in an inattentive state, the occupant state detection unit 11 can measure the duration of time during which a state indicating that the occupant state is not in an inattentive state continues when the vehicle 1 is driven after the driving assistance function is suppressed. Furthermore, the occupant state detection unit 11 may output the duration of time during which a state indicating that the occupant state is not in an inattentive state to the storage unit 13 and store it in the storage unit 13.

[0088] In step ST108, the release request unit 14 determines whether the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration (ST108; YES), the process proceeds to step ST110. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the first attention duration (ST108; NO), the process returns to step ST107.

[0089] In step ST109, the release request unit 14 determines whether the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration (ST109; YES), the process proceeds to step ST110. If the time period during which the state indicating that the occupant is not inattentive has continued for at least the second attention duration (ST109; NO), the process returns to step ST107.

[0090] In step ST110, the release request unit 14 outputs a signal to the driving control device 21 requesting the driving control device 21 to release the suppression of the driving support function.

[0091] In step ST111, the driving control device 21 cancels the suppression of the driving support function. After the processing of step ST111, the processing returns to the processing of step ST101.

[0092] 7 and 8 are diagrams showing an example of the hardware configuration of the occupant monitoring device 2 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the release request unit 14, and the command unit 15 are realized by a processing circuit 1001. That is, the occupant monitoring device 2 includes the processing circuit 1001 for controlling the monitoring of the state of the vehicle occupant based on the captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 7, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 8.

[0093] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0094] When the processing circuit is the processor 1004, the functions of the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the release request unit 14, and the command unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the storage unit 13, the release request unit 14, and the command unit 15 by reading and executing the program stored in the memory 1005. That is, the occupant monitoring device 2 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST10 of FIG. 8 described above. The program stored in the memory 1005 can also be said to cause a computer to execute the processing procedures or methods of the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the release request unit 14, and the command unit 15. Here, memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0095] Note that the functions of the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the release request unit 14, and the command unit 15 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the image acquisition unit 10 may be realized by a processing circuit 1001 as dedicated hardware, and the functions of the occupant state detection unit 11, the suppression determination unit 12, the release request unit 14, and the command unit 15 may be realized by the processor 1004 reading and executing programs stored in the memory 1005.

[0096] The storage unit 13 is configured by, for example, a memory 1005. The occupant monitoring device 2 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 8 or the output device 9.

[0097] The occupant monitoring device 2 according to this embodiment includes an image acquisition unit 10 that acquires captured images of occupants inside the vehicle 1 from an imaging device 8 mounted on the vehicle 1 and capturing images of the interior of the vehicle 1, an occupant state detection unit 11 that uses the captured images acquired by the image acquisition unit 10 to detect an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness, a suppression determination unit 12 that determines whether it is necessary to suppress a driving assistance function by a driving control device 21 that executes braking / driving control of the vehicle 1 when the detection result of the occupant state detection unit 11 indicates an inattentive state, and a release request unit 14 that, when the suppression determination unit 12 determines that it is necessary to suppress the driving assistance function, outputs a signal to the driving control device 21 requesting release of suppression of the driving assistance function by the driving control device 21 based on the detection result of the occupant state detection unit 11. Thus, the occupant monitoring device 2 can suppress a decrease in user convenience.

[0098] Furthermore, the vehicle control system 3 according to this embodiment includes the above-described occupant monitoring device 2 and a driving control device 21 that controls the driving of the vehicle 1. When the occupant monitoring device 2 determines that it is necessary to suppress the driving assistance function, it outputs a signal to the driving control device 21 requesting cancellation of the suppression of the driving assistance function based on the detection result of the occupant state detection unit 11, and when the driving control device 21 acquires the signal requesting cancellation of the suppression of the driving assistance function, it outputs a signal to cancel the suppression of the driving assistance function to the steering mechanism 7 or the braking / driving mechanism 6 mounted on the vehicle 1. Therefore, the vehicle control system 3 can prevent a decrease in user convenience.

[0099] The occupant monitoring method according to the present embodiment includes the steps of: an image acquisition unit 10 acquiring an image of an occupant inside the vehicle 1 from an imaging device 8 mounted on the vehicle 1 and capturing an image of the interior of the vehicle 1; an occupant state detection unit 11 detecting an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness using the image acquired by the image acquisition unit 10; a suppression determination unit 12 determining whether or not it is necessary to suppress a driving assistance function by a driving control device 21 that executes braking / driving control of the vehicle 1 when the detection result of the occupant state detection unit 11 indicates the inattentive state; and a release request unit 14 outputting a signal to the driving control device 21 requesting release of suppression of the driving assistance function by the driving control device 21 based on the detection result of the occupant state detection unit 11 when the suppression determination unit 12 determines that it is necessary to suppress the driving assistance function. Therefore, the occupant monitoring method can suppress a decrease in user convenience.

[0100] The program according to the present embodiment causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle 1 from an imaging device mounted on the vehicle 1 and capturing an image of the interior of the vehicle 1; an occupant state detection unit 11 uses the image acquired by the image acquisition unit 10 to detect an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness; a suppression determination unit 12 determines whether it is necessary to suppress the driving assistance function of the driving control device 21 that executes braking / driving control of the vehicle 1 when the detection result of the occupant state detection unit 11 indicates the inattentive state; and a release request unit 14 outputs a signal to the driving control device 21 requesting release of the suppression of the driving assistance function of the driving control device 21 based on the detection result of the occupant state detection unit 11 when the suppression determination unit 12 determines that it is necessary to suppress the driving assistance function. Therefore, the program can suppress a decrease in user convenience.

[0101] Embodiment 2 An occupant monitoring device 51 according to embodiment 2 will be described with reference to Figures 9 and 10. Description of configurations similar to those of embodiment 1 will be omitted. In addition, in Figures 9 and 10, the same reference numerals as those in Figures 1 to 8 indicate the same or corresponding parts. In addition, steps that are the same as those in the processing of the occupant monitoring device 2 according to embodiment 1 are assigned the same reference numerals as those shown in Figures 5 and 6, and descriptions thereof will be omitted or simplified. The following description will focus on differences from embodiment 1.

[0102] The occupant monitoring device 51 according to this embodiment differs from the occupant monitoring device 2 according to the first embodiment in that an occlusion determining unit 52 is further provided.

[0103] The occupant monitoring device 51 includes an occlusion determination unit 52. The occlusion determination unit 52 is connected to the communication bus 16, and is capable of transmitting and receiving data to and from the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the memory unit 13, the release request unit 14, and the command unit 15 via the communication bus 16.

[0104] 9 is a block diagram showing an occupant monitoring device 51 according to embodiment 2. An occlusion determination unit 52 acquires a captured image from the image acquisition unit 10. Using a known image recognition technique, the occlusion determination unit 52 determines whether or not there is an obstruction between the imaging device and the occupant that obstructs the capture of an image of the occupant.

[0105] The occupant state detection unit 11 outputs the detection result of the occupant state to the suppression determination unit 12 and the storage unit 13 .

[0106] The occlusion determination unit 52 outputs the determination result of whether or not an obstruction exists to the suppression determination unit 12 and the storage unit 13. When the suppression determination unit 12 acquires the determination result that an obstruction exists from the occlusion determination unit 52, the suppression determination unit 12 can determine that the driving assistance function needs to be suppressed. Furthermore, when the suppression determination unit 12 acquires the determination result that an obstruction exists from the occlusion determination unit 52, the suppression determination unit 12 can determine that the driving assistance function needs to be suppressed, and output a signal requesting the suppression of the driving assistance function to the driving control device 21.

[0107] Furthermore, when the suppression determination unit 12 acquires a determination result that an obstruction is present from the obstruction determination unit 52 and determines that the driving assistance function needs to be suppressed, the suppression determination unit 12 can output information that the factor that determined that the driving assistance function needs to be suppressed is the presence of an obstruction to the storage unit 13 and store the information in the storage unit 13. Furthermore, the suppression determination unit 12 outputs information that the factor that determined that the driving assistance function needs to be suppressed is the presence of an obstruction to the release request unit 14.

[0108] Furthermore, the suppression determination unit 12 can determine that a notification should be sent to the occupant when it acquires a determination result that an obstruction exists from the occupancy determination unit 52. Furthermore, the suppression determination unit 12 can output to the command unit 15 information that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is the presence of an obstruction, and a command to the effect that a notification should be sent to the occupant.

[0109] The suppression determination unit 12 can determine that the driving assistance function needs to be suppressed when the information obtained from the occupant state detection unit 11 indicates that the occupant is in an inattentive state, or when the information obtained from the occupant occupancy determination unit 52 indicates that an obstruction is present.

[0110] In other words, if the information obtained from the occupant state detection unit 11 indicates that the occupant is in an inattentive state and the information obtained from the occupancy determination unit 52 indicates that an obstruction is present, the suppression determination unit 12 can determine that the driving assistance function needs to be suppressed.

[0111] In addition, the suppression judgment unit 12 can determine that the driving assistance function needs to be suppressed if the information obtained from the occupant state detection unit 11 indicates that the occupant is in an inattentive state and the information obtained from the occupancy judgment unit 52 indicates that no occupant is present.

[0112] In addition, the suppression judgment unit 12 can determine that the driving assistance function needs to be suppressed if the information obtained from the occupant state detection unit 11 indicates that the occupant's state is not in an inattentive state and the information obtained from the occupancy judgment unit 52 indicates that an obstruction is present.

[0113] In addition, the suppression judgment unit 12 can determine that there is no need to suppress the driving assistance function if the information obtained from the occupant state detection unit 11 indicates that the occupant is not in an inattentive state and the information obtained from the occupancy judgment unit 52 indicates that there is no occupant.

[0114] In addition, the suppression judgment unit 12 may not obtain information indicating the occupant's state from the occupant state detection unit 11, and may determine that the driving assistance function needs to be suppressed based only on the information obtained from the occupancy judgment unit 52 indicating the presence of an obstruction.

[0115] The storage unit 13 can acquire the determination result of whether or not an obstruction exists from the obstruction determination unit 52. The storage unit 13 can also acquire and store information from the suppression determination unit 12 that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is the presence of an obstruction.

[0116] When the command unit 15 acquires from the suppression determination unit 12 or the memory unit 13 information that the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is the presence of an obstruction, and an instruction to notify the occupant, the command unit 15 outputs an instruction to the output device 9 to make a notification indicating that an obstruction is present. That is, when the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is the presence of an obstruction, the command unit 15 outputs an instruction to the output device 9 to make a notification indicating that an obstruction is present.

[0117] When the suppression determination unit 12 determines that suppression of the driving assistance function is necessary, the release request unit 14 outputs a signal requesting the driving control device 21 to release the suppression of the driving assistance function. When the factor acquired from the suppression determination unit 12 that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is the presence of an obstruction, and when the release request unit 14 acquires a determination result from the obstruction determination unit 52 that the obstruction has been removed after the driving assistance function was suppressed, the release request unit 14 outputs a signal requesting the driving control device 21 to release the suppression of the driving assistance function.

[0118] Next, a description will be given of the operation of the occupant monitoring device 51. Fig. 10 is a flowchart showing an example of the operation of the occupant monitoring device 51 according to the second embodiment.

[0119] In step ST201, the image acquisition unit 10 acquires a captured image from the imaging device 8. The image acquisition unit 10 outputs the acquired captured image to the occlusion determination unit 52.

[0120] In step ST202, the occlusion determination unit 52 determines whether or not there is an obstruction between the imaging device and the occupant that blocks the imaging of the occupant. If the occlusion determination unit 52 determines that there is an obstruction (ST202; YES), the process proceeds to step ST105. If the occlusion determination unit 52 determines that there is no obstruction (ST202; NO), the process returns to step ST201.

[0121] In step ST105, the suppression determination unit 12 outputs a signal to the driving control device 21 requesting the driving control device 21 to suppress the driving assistance function.

[0122] In step ST106, when the driving control device 21 receives a signal requesting suppression of the driving support function, the driving control device 21 suppresses the driving support function.

[0123] In step ST203, the command unit 15 commands the suppression determination unit 12 to issue a notification indicating the factor that caused the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary. When the occlusion determination unit 52 determines that an obstruction exists, the command unit 15 outputs, to the output device 9, a command to issue a notification indicating that an obstruction exists.

[0124] In step ST204, the output device 9 outputs a notification indicating that an obstruction is present.

[0125] In step ST205, the image acquisition unit 10 acquires a captured image from the imaging device 8. The image acquisition unit 10 outputs the acquired captured image to the occlusion determination unit 52.

[0126] In step ST206, the occlusion determination unit 52 determines whether or not an obstruction that blocks the capture of an image of the occupant is present between the image capture device and the occupant after the driving assistance function is suppressed. If no obstruction is present (ST206; NO), the occlusion determination unit 52 determines that the obstruction has been removed and proceeds to the processing of step ST110. If an obstruction is present (ST206; YES), the processing returns to step ST205.

[0127] In step ST110, the release request unit 14 outputs a signal to the driving control device 21 requesting the driving control device 21 to release the suppression of the driving support function.

[0128] In step ST111, the driving control device 21 cancels the suppression of the driving support function. After the processing of step ST111, the processing returns to the processing of step ST101.

[0129] The occupant monitoring device 51 in this embodiment is equipped with an occupant obstruction determination unit 52 that determines whether or not there is an obstruction between the imaging device 8 and the occupant that prevents the occupant from being photographed. When the occupancy determination unit 52 determines that an obstruction exists, the suppression determination unit 12 determines that the driving assistance function needs to be suppressed. When the suppression determination unit 12 determines that the driving assistance function needs to be suppressed because of the presence of an obstruction, the command unit 15 outputs a command to the output device 9 to cause a notification indicating the presence of an obstruction. Thus, even when an obstruction exists between the imaging device 8 and the occupant, the occupant monitoring device 51 can prevent a decrease in user convenience.

[0130] Embodiment 3 An occupant monitoring device 53 according to embodiment 3 will be described with reference to Figures 11 and 12. Description of configurations similar to those of embodiment 1 will be omitted. In Figures 11 and 12, the same reference numerals as those in Figures 1 to 10 indicate the same or corresponding parts. In addition, steps that are the same as those in the processing of the occupant monitoring device 2 according to embodiment 1 are assigned the same reference numerals as those shown in Figures 5 and 6, and descriptions thereof will be omitted or simplified. The following description will focus on differences from embodiment 1.

[0131] The occupant monitoring device 53 according to this embodiment differs from the occupant monitoring device 2 according to the first embodiment in that a calibration unit 54 and a reliability calculation unit 55 are further provided.

[0132] 11 is a block diagram showing an occupant monitoring device 53 according to embodiment 3. The occupant monitoring device 53 includes a calibration unit 54 and a reliability calculation unit 55. The calibration unit 54 and the reliability calculation unit 55 are connected to a communication bus 16, and are capable of transmitting and receiving data to and from the image acquisition unit 10, the occupant state detection unit 11, the suppression determination unit 12, the storage unit 13, the release request unit 14, and the command unit 15 via the communication bus 16.

[0133] The calibration unit 54 transmits and receives information to and from the occupant state detection unit 11 and performs calibration processing. The calibration unit 54 executes calibration processing for image processing in the occupant state detection unit 11. The calibration unit 54 executes calibration processing, which is processing for correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit 10. The calibration processing is processing for correcting individual differences between occupants who are the target of image processing. Generally, the detection results by the occupant state detection unit 11 involve detection errors due to individual differences between occupants. Therefore, the calibration unit 54 executes calibration processing to ensure the detection accuracy of the occupant state detection unit 11. Specific examples of calibration processing are shown below. Note that the examples are not limited to the following specific examples, and other known means may also be used.

[0134] First, an example will be described in which the calibration unit 54 corrects the standard model used for gaze direction detection. In this case, the occupant state detection unit 11 holds a standard model for gaze direction detection that associates a captured image of the occupant's face with the occupant's gaze direction. The occupant state detection unit 11 then uses the standard model to detect the occupant's gaze direction from the captured image and determine, for example, whether or not the occupant is looking away from the road. Here, the standard model held in the occupant state detection unit 11 needs to be corrected to accommodate individual differences between occupants.

[0135] Therefore, the calibration unit 54 corrects the standard model based on the actual gaze direction of the occupant and the captured image in that state. For example, the calibration unit 54 detects the gaze direction of the occupant and estimates the vehicle accessory the occupant is gazing at using the result. Then, the calibration unit 54 determines the direction of the vehicle accessory as seen by the occupant as a reference for the gaze direction, and corrects the standard model based on the error between this reference gaze direction and the detected gaze direction. Alternatively, the calibration unit 54 may correct the standard model by correlating the gaze of the occupant when operating each device in the vehicle 1 with the position of the device being operated. This allows the occupant state detection unit 11 to accurately detect the gaze direction regardless of individual differences between occupants.

[0136] The calibration unit 54 may also correct the standard model used for facial direction detection. In this case, the occupant state detection unit 11 holds a 3D model that is the standard model, and uses the 3D model to detect the facial direction of the occupant from the captured image and determine, for example, whether the occupant is looking away from the road. The 3D model is a 3D model that reflects, for example, the size of a standard adult's face and the feature points of the facial features. In other words, the 3D model does not reflect the size of the occupant's face and the feature points of the facial features.

[0137] Therefore, the calibration unit 54 extracts feature points from the occupant in the captured image that correspond to the feature points of the 3D model. The method for extracting the feature points of the occupant from the captured image is not particularly limited, and the feature points may be extracted, for example, by a feature point extraction technique using machine learning, an image recognition technique such as edge extraction, or template matching. The calibration unit 54 then compares the positions of the feature points of the 3D model with the positions of the feature points of the occupant extracted from the captured image on the captured image. Next, the calibration unit 54 calibrates the positions of the feature points of the 3D model to match the positions of the feature points of the occupant extracted from the captured image so that the error between the positions of the feature points of the 3D model and the positions of the feature points of the occupant extracted from the captured image is within a predetermined threshold. This allows the occupant state detection unit 11 to accurately detect the face direction regardless of individual differences between occupants.

[0138] The calibration unit 54 may also correct the standard model used to detect the degree of eye opening. In this case, the occupant state detection unit 11 holds a standard model for detecting the degree of eye opening of the occupant. The occupant state detection unit 11 then detects the occupant's eyes from the captured image and detects the eye opening degree, which is the degree to which the occupant's eyes are open, using the standard model. The standard model for detecting the degree of eye opening includes information on a reference flattening ratio, which is, for example, a value obtained by dividing the width of the eyes when the eyes are open by the height of the eyes, and the reference flattening ratio varies depending on individual occupants.

[0139] Therefore, the calibration unit 54 calculates the occupant's eye width and eye height from the captured image and calculates the occupant's reference flattening.The calibration unit 54 then corrects the standard model based on the error between the standard model's reference flattening and the calculated occupant's reference flattening.This allows the occupant state detection unit 11 to accurately detect the eye opening degree regardless of individual occupant differences.The calibration unit 54 outputs information indicating the corrected standard model, which is the result of the calibration process, to the storage unit 13.

[0140] The suppression determination unit 12 determines that suppression of the driving support function is necessary when the occupant state detection unit 11 detects that the occupant is in an inattentive state. Furthermore, the suppression determination unit 12 outputs information that the factor for determining that suppression of the driving support function is necessary is the occupant's inattentive state to the storage unit 13, and causes the storage unit 13 to store the information.

[0141] When the command unit 15 obtains information from the suppression judgment unit 12 or the memory unit 13 that the factor that causes the suppression judgment unit 12 to determine that the driving assistance function needs to be suppressed is that the occupant is in an inattentive state, the command unit 15 outputs a command to the output device 9 to notify the output device 9 that the driving assistance function has been suppressed due to the occupant's inattentive state.

[0142] In addition, when the suppression determination unit 12 determines that suppression of the driving assistance function is necessary because the occupant is in an inattentive state, the command unit 15 outputs a command to the calibration unit 54 to execute a calibration process.

[0143] The calibration unit 54 executes the calibration process when it receives a command to execute the calibration process from the command unit 15. The calibration unit 54 executes the calibration process based on the captured image in order to determine whether the occupant is looking aside, etc., based on the state in which the occupant is facing forward. However, there is a possibility that the calibration process will be executed by mistake, assuming that the captured image in which the occupant is not facing forward is an image in which the occupant is facing forward. In other words, the result of the calibration process may be incorrect. If the occupant state detection unit 11 detects an inattentive state of the occupant, the result of the calibration process executed by the calibration unit 54 may be incorrect, which may have caused the occupant state detection unit 11 to make an erroneous detection. By executing the calibration process by the calibration unit 54, it is possible to correct the result of the calibration process, which may have been incorrect.

[0144] Furthermore, when the suppression determination unit 12 determines that suppression of the driving assistance function is necessary because of inattention and the command unit 15 acquires a signal indicating that the vehicle 1 is stopped or traveling on a route without congestion from the driving control device 21, the command unit 15 may output a command to the calibration unit 54 to execute a calibration process. When the command unit 15 acquires a signal indicating that the vehicle 1 is stopped or traveling on a route without congestion from the driving control device 21, the command unit 15 outputs a command to the calibration unit 54 to execute a calibration process, so that the calibration unit 54 can perform the calibration process with the occupant facing forward.

[0145] The reliability calculation unit 55 calculates the reliability of the detection of the inattention state by the occupant state detection unit 11. The reliability is expressed, for example, as a value between 0 and 100. The reliability is smaller when a part of the eyes, nose, or mouth is blocked, for example, when the occupant covers their mouth with their hand, rubs their eyes with their hand, or looks back from the driver's seat to look into the backseat. The reliability is also smaller when the occupant is wearing sunglasses or other eyewear that blocks the infrared sensor light, because the contrast ratio around the eyes is reduced. The reliability is also smaller when the eyelids are lowered, because the iris and pupil cannot be captured. The reliability is also smaller when the face angle relative to the image capture device 8 is greater than a certain value, because the contrast ratio of the iris and pupil is reduced due to the shadow of the eyelashes or eyelids. The reliability calculation unit 55 determines whether the reliability is lower than a predetermined threshold.

[0146] When the suppression determination unit 12 determines that the driving assistance function needs to be suppressed because of inattention and the reliability is lower than a predetermined threshold, the command unit 15 outputs a command to the calibration unit 54 to perform a calibration process.

[0147] When the reliability is lower than the threshold, the result of the calibration process performed before it was determined that suppression of the driving assistance function was necessary, i.e., the result previously performed by the calibration unit 54, is more likely to be incorrect than when the reliability is equal to or higher than the threshold. In other words, there is a high possibility that an inattentive state has been erroneously detected. When the reliability is lower than the predetermined threshold, the calibration unit 54 can be made to perform the calibration process again in the case where an inattentive state has been erroneously detected, thereby correcting the result of the calibration process that may have been erroneous.

[0148] If the suppression determination unit 12 determines that the suppression of the driving assistance function is necessary because the driver is inattentive and the reliability is equal to or greater than a predetermined threshold, the command unit 15 outputs a command to the occupant state detection unit 11 to detect the driver's inattentive state using the results of previous execution by the calibration unit 54.

[0149] When the reliability is equal to or higher than the threshold, the result previously performed by the calibration unit 54 is more likely to be correct than when the reliability is lower than the threshold. When the reliability is equal to or higher than the predetermined threshold, the calibration unit 54 outputs a command to the occupant state detection unit 11 to detect the state of inattention of the occupant using the result previously performed by the calibration unit 54, so that the occupant state detection unit 11 can detect the state of the occupant using the result of the calibration process that is more likely to be correct.

[0150] When it is detected that the occupant is in an inattentive state, the calibration process may be performed even if the reliability is not low.

[0151] In addition, when the release request unit 14 outputs a signal to the driving control device 21 requesting the release of suppression of the driving assistance function, the command unit 15 can issue a command to the occupant status detection unit 11 to eliminate the factor based on the factor stored in the memory unit 13.

[0152] When the occupant state detection unit 11 receives a command from the command unit 15 to execute elimination of the cause, it can output a signal to the calibration unit 54 to execute calibration processing. When the calibration unit 54 receives a signal to execute calibration processing from the occupant state detection unit 11, it executes calibration processing. Execution of calibration processing by the calibration unit 54 leads to elimination of the cause.

[0153] When the release request unit 14 outputs a signal to the driving control device 21 requesting the release of the suppression of the driving assistance function, the command unit 15 issues at least one of a command to the output device 9 mounted on the vehicle 1 to notify the output device 9 of the cause based on the cause stored in the memory unit 13, and a command to the occupant status detection unit 11 to resolve the cause, thereby further preventing a decrease in user convenience.

[0154] Next, a description will be given of the operation of the occupant monitoring device 53. Fig. 12 is a flowchart showing an example of the operation of the occupant monitoring device 51 according to the second embodiment.

[0155] In step ST301, the reliability calculation unit 55 determines whether the reliability is lower than a predetermined threshold. If the reliability is lower than the predetermined threshold (ST301; YES), the process proceeds to step ST302. If the reliability is equal to or higher than the predetermined threshold (ST301; NO), the process proceeds to step ST304.

[0156] In step ST302, if the factor that causes the suppression determination unit 12 to determine that suppression of the driving assistance function is necessary is that the occupant is in an inattentive state, the command unit 15 outputs a command to the calibration unit 54 to execute configuration processing.

[0157] In step ST303, the calibration unit 54 executes a calibration process. After the process of step ST303, the process proceeds to step ST107, and the process of step ST108 or step ST109 is executed. That is, steps ST108 and ST109 can be executed using the determination result of the occupant state executed after step ST303. That is, the occupant state can be detected based on the result of the calibration process, which is likely to be correct.

[0158] In step ST304, if the factor that causes the suppression judgment unit 12 to determine that the driving assistance function needs to be suppressed is that the occupant is in an inattentive state, the command unit 15 outputs a command to the occupant state detection unit 11 to detect the occupant's inattentive state using the results previously executed by the calibration unit 54.

[0159] Note that even when the reliability is equal to or greater than a predetermined threshold (ST301; NO), the command unit 15 may output a command to the calibration unit 54 to cause the calibration unit 54 to execute the configuration process.

[0160] In the above-described embodiments, the materials, materials, dimensions, shapes, relative positional relationships, and implementation conditions of each component may be described. However, these are merely examples in all respects and are not intended to limit the scope of each embodiment. Therefore, countless variations not exemplified are contemplated within the scope of each embodiment. For example, these include cases where any component is modified, added, or omitted, and even cases where at least one component in at least one embodiment is extracted and combined with a component in another embodiment.

[0161] It goes without saying that various design modifications are possible within the scope of the present invention, as long as the object of the present invention can be achieved and the gist of the present invention is not deviated from.

[0162] REFERENCE SIGNS LIST 1 Vehicle, 2, 51, 53 Occupant monitoring device, 3 Vehicle control system, 4 Occupant monitoring system, 5 Driving control system, 6 Braking / driving mechanism, 7 Steering mechanism, 8 Imaging device, 9 Output device, 10 Image acquisition unit, 11 Occupant state detection unit, 12 Suppression determination unit, 13 Memory unit, 14 Release request unit, 15 Command unit, 21 Driving control device, 52 Obstruction determination unit, 54 Calibration unit, 55 Reliability calculation unit

Claims

1. An occupant monitoring device comprising: an image acquisition unit that acquires captured images of occupants inside the vehicle from an imaging device mounted on the vehicle that captures images of the interior of the vehicle; an occupant state detection unit that uses the captured images acquired by the image acquisition unit to detect an inattentive state indicating that the occupant is in a state of absent-mindedness or low alertness; a suppression determination unit that determines, when the occupant state detection unit detects that the occupant is in an inattentive state, that it is necessary to suppress a driving assistance function by a driving control device that executes braking and driving control of the vehicle; and a release request unit that, when the suppression determination unit determines that it is necessary to suppress the driving assistance function by the driving control device, outputs a signal to the driving control device to request that suppression of the driving assistance function by the driving control device be released based on the detection result of the occupant state detection unit.

2. The occupant monitoring device of claim 1, wherein the release request unit outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device when the time the vehicle has been continuously stopped after the driving assistance function by the driving control device has been suppressed is equal to or longer than a predetermined stopping reference time, and when the detection result of the occupant state detection unit when the vehicle is moving after having been stopped for equal to or longer than the stopping reference time indicates that the occupant is not in an inattentive state for equal to or longer than a predetermined first attention duration.

3. The occupant monitoring device of claim 1, wherein the release request unit outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device when the time the vehicle has been continuously stopped after the driving assistance function by the driving control device has been suppressed is less than a predetermined stopping reference time, and when the detection result of the occupant state detection unit when driving after the driving assistance function by the driving control device has been suppressed indicates that the occupant is not in an inattentive state for more than a predetermined second attention duration time.

4. An occupant monitoring device as described in claim 1, comprising: a memory unit that stores factors that cause the suppression determination unit to determine that suppression of the driving assistance function by the driving control device is necessary; and a command unit that, when the release request unit outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device, issues a command to an output device mounted on the vehicle to notify the output device of the factor based on the factor stored in the memory unit.

5. An occupant monitoring device as described in claim 4, further comprising an occupant obstruction determination unit that determines whether or not there is an obstruction between the imaging device and the occupant that obstructs imaging of the occupant, wherein the suppression determination unit determines that suppression of the driving assistance function is necessary when the occupancy determination unit determines that the obstruction exists, and wherein the command unit outputs a command to the output device to issue a notification indicating the presence of the obstruction when the suppression determination unit determines that suppression of the driving assistance function by the driving control device is necessary because of the presence of the obstruction.

6. The occupant monitoring device according to claim 5, characterized in that the release request unit outputs a signal to the driving control device requesting the release of the suppression of the driving assistance function by the driving control device when the occlusion determination unit determines that the occlusion has been removed.

7. The occupant monitoring device of claim 4, wherein the command unit outputs a command to the output device to notify the driver that the driving assistance function of the driving control device has been suppressed due to the driver's inattention when the suppression determination unit determines that the suppression of the driving assistance function of the driving control device is necessary due to the driver's inattention.

8. An occupant monitoring device as described in claim 4, further comprising a calibration unit that executes a calibration process which is a process of correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit, wherein the suppression determination unit determines that it is necessary to suppress the driving assistance function by the driving control device when the occupant state detection unit detects that the occupant is in an inattentive state, and the command unit outputs a command to the calibration unit to execute the calibration process when the inattentive state is the cause of the suppression determination unit's determination that it is necessary to suppress the driving assistance function by the driving control device.

9. The occupant monitoring device according to claim 8, wherein the command unit outputs a command to the calibration unit to execute a calibration process when the suppression determination unit determines that the suppression of the driving assistance function by the driving control device is necessary because of the inattention state and receives a signal from the driving control device indicating that the vehicle is stopped or is traveling on a route without congestion.

10. The occupant monitoring device according to claim 8, wherein the command unit outputs a command to the output device to notify the driver that the driving support function by the driving control device has been suppressed due to the inattention of the occupant when the suppression determination unit determines that the suppression of the driving support function by the driving control device is necessary because of the inattention of the occupant, and outputs a command to the calibration unit to execute a calibration process when the suppression determination unit determines that the suppression of the driving support function by the driving control device is necessary because of the inattention of the occupant and receives from the driving control device a signal indicating that the vehicle is stopped or is traveling on a route without congestion.

11. An occupant monitoring device as described in claim 4, comprising: a calibration unit that executes a calibration process, which is a process of correcting the position of the occupant's body parts in the captured image acquired by the image acquisition unit; and a reliability calculation unit that calculates the reliability of the detection of an inattentive state by the occupant state detection unit, wherein the command unit outputs a command to the calibration unit to execute the calibration process when the factor that has caused the suppression determination unit to determine that the driving assistance function by the driving control device needs to be suppressed is the inattentive state and the reliability is lower than a predetermined threshold value.

12. The occupant monitoring device according to claim 11, wherein the command unit outputs a command to the calibration unit to execute a calibration process when the factor for which the suppression determination unit has determined that the driving assistance function by the driving control device needs to be suppressed is the inattention state and the reliability is lower than a predetermined threshold, and outputs a command to the occupant state detection unit to detect the occupant's inattention state using results of previous executions by the calibration unit when the suppression determination unit has determined that the factor for which the driving assistance function by the driving control device needs to be suppressed is the inattention state and the reliability is higher than a predetermined threshold.

13. A vehicle control system comprising: an occupant monitoring device according to any one of claims 1 to 12; and a driving control device that controls the driving of the vehicle, wherein when the occupant monitoring device determines that it is necessary to suppress a driving assistance function by the driving control device, it outputs a signal to the driving control device requesting release of suppression of the driving assistance function by the driving control device based on the detection result of the occupant state detection unit, and when the driving control device acquires a signal requesting release of suppression of the driving assistance function by the driving control device, it outputs a signal to release the suppression of the driving assistance function to a steering mechanism or braking / driving mechanism mounted on the vehicle.

14. An occupant monitoring method comprising the steps of: an image acquisition unit acquiring an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing images of the interior of the vehicle; an occupant state detection unit detecting an inattentive state indicating that the occupant is in a state of absent-mindedness or low alertness using the image acquired by the image acquisition unit; a suppression determination unit determining whether it is necessary to suppress a driving assistance function by a driving control device that executes braking and driving control of the vehicle, when the detection result of the occupant state detection unit determines that it is necessary to suppress the driving assistance function by the driving control device; and a release request unit outputting a signal to the driving control device, based on the detection result of the occupant state detection unit, to request the driving control device to release the suppression of the driving assistance function by the driving control device, when the suppression determination unit determines that it is necessary to suppress the driving assistance function by the driving control device.

15. A program that causes a computer to execute the following steps: an image acquisition unit acquires an image of an occupant inside the vehicle from an imaging device mounted on the vehicle and capturing images of the interior of the vehicle; an occupant state detection unit uses the image acquired by the image acquisition unit to detect an inattentive state indicating that the occupant is in a distracted state or has a low level of alertness; a suppression determination unit determines whether it is necessary to suppress a driving assistance function by a driving control device that executes braking and driving control of the vehicle, when the detection result of the occupant state detection unit determines that it is necessary to suppress the driving assistance function by the driving control device; and a release request unit outputs a signal to the driving control device, based on the detection result of the occupant state detection unit, to request release of suppression of the driving assistance function by the driving control device, when the suppression determination unit determines that it is necessary to suppress the driving assistance function by the driving control device.

Images