Occupant monitoring device, occupant monitoring method, occupant monitoring program, and vehicle control system

Abstract

An occupant monitoring device (2) comprises: an image acquisition unit (10) that acquires a captured image in which an occupant in a vehicle (1) is captured; an occupant-state detection unit (11) that uses the captured image acquired by the image acquisition unit (10) to detect a distraction state or an awakening state of the occupant; a warning determination unit (12) that uses the result detected by the occupant-state detection unit (11) to determine whether or not a warning to the occupant is necessary; and a suppression determination unit (14) that uses the result detected by the occupant-state detection unit (11) to determine the necessity of suppression of a driving support function in the vehicle (1), and the suppression determination unit (14) determines whether or not suppression of the driving support function is necessary on the basis of the cumulative number of times of notification, and a determination criterion of a notification determination unit (13) is independent of a determination criterion of the warning determination unit (12). As a result, since the necessity of suppression of the driving support function is determined independently of the warning, the driving support function is not suppressed even if a warning is issued by erroneous detection.

Description

Occupant monitoring device, occupant monitoring method, occupant monitoring program, and vehicle control system

[0001] The present disclosure relates to an occupant monitoring device, an occupant monitoring method, an occupant monitoring program, and a vehicle control system.

[0002] There is an occupant monitoring device that detects a distracted state or an alert state of an occupant in a vehicle, and when it is determined that the occupant is in a distracted state or a low level of alertness, suppresses a driving assistance function in the vehicle to alert the occupant. For example, Patent Document 1 discloses a vehicle driving control device that includes a control means for automatically driving the vehicle by controlling the vehicle speed, a recognition means for recognizing the alertness level of the driver, an alarm means for alerting the driver when the recognized alertness level falls below a predetermined level, a counting means for counting the number of times the alarm means has been activated, a prohibition means for prohibiting the activation of the control means when the number of times exceeds a predetermined value, and a recovery means for detecting that the driver has performed a predetermined recovery operation and canceling the prohibition of the activation of the control device.

[0003] Japanese Patent Application Publication No. 6-171391

[0004] In the device of Patent Document 1, even if the driver is not in a distracted state or a low level of alertness, an alarm may be issued due to erroneous recognition by the recognition means. Even if there is erroneous recognition by the recognition means, if the number of alarms exceeds a predetermined value, the driving assistance function is suppressed, thereby reducing convenience for the driver.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an occupant monitoring device, an occupant monitoring method, an occupant monitoring program, and a vehicle control system that can improve user convenience.

[0006] The occupant monitoring device of the present disclosure includes an image acquisition unit that acquires captured images of occupants in a vehicle, an occupant state detection unit that detects a distracted or awake state of the occupant using the captured images acquired by the image acquisition unit, a warning determination unit that uses the results detected by the occupant state detection unit to determine whether a warning is needed to the occupant, a suppression determination unit that uses the results detected by the occupant state detection unit to determine whether a driving assistance function in the vehicle needs to be suppressed, and a notification determination unit that uses the results detected by the occupant state detection unit to determine whether an alert is needed to notify the occupant in advance that the driving assistance function will be suppressed, and the suppression determination unit determines whether an alert is needed to notify the occupant in advance that the driving assistance function will be suppressed based on the cumulative number of alerts, and the determination criteria of the notification determination unit are independent of the determination criteria of the warning determination unit.

[0007] The occupant monitoring method according to the present disclosure includes the steps of: acquiring an image of an occupant in a vehicle; detecting the occupant's distracted or awake state using the image; determining whether or not a warning is needed for the occupant using the result of detecting the occupant's distracted or awake state; determining whether or not a driving assistance function in the vehicle needs to be suppressed using the result of detecting the occupant's distracted or awake state; and determining whether or not an alert is needed to notify the occupant in advance that the driving assistance function will be suppressed using the result of detecting the occupant's distracted or awake state, wherein in the step of determining whether or not suppression is needed, a determination is made as to whether or not the driving assistance function needs to be suppressed based on the cumulative number of alerts, and the determination criteria in the step of determining whether or not an alert is needed are independent of the determination criteria in the step of determining whether or not a warning is needed.

[0008] The occupant monitoring program of the present disclosure causes a computer to function as an image acquisition unit that acquires images of occupants in a vehicle, an occupant state detection unit that detects the occupant's distracted or awake state using the images acquired by the image acquisition unit, a warning determination unit that uses the results detected by the occupant state detection unit to determine whether a warning is needed to the occupant, a suppression determination unit that uses the results detected by the occupant state detection unit to determine whether a driving assistance function in the vehicle needs to be suppressed, and a notification determination unit that uses the results detected by the occupant state detection unit to determine whether an alert is needed to notify the occupant in advance that the driving assistance function will be suppressed, and the determination criteria of the suppression determination unit are independent of the determination criteria of the warning determination unit.

[0009] The vehicle control system according to the present disclosure includes an occupant monitoring device according to the present disclosure, a vehicle control unit that controls the braking and driving of the vehicle to perform driving assistance, a warning unit that warns of an occupant's distracted state or low alertness, and a notification unit that notifies of the suppression of the driving assistance function.

[0010] According to the present disclosure, even if a warning is issued due to erroneous detection, the driving assistance function is not suppressed, thereby improving convenience for the user.

[0011] Fig. 1 is a block diagram showing a part of a vehicle equipped with an occupant monitoring device according to embodiment 1. Fig. 2 is a block diagram showing a driving control system equipped with an occupant monitoring device according to embodiment 1. Fig. 3 is a block diagram showing an occupant state detection unit according to embodiment 1. Fig. 4 is a block diagram showing an example of a hardware configuration of an occupant monitoring device according to embodiment 1. Fig. 5 is a block diagram showing an example of a hardware configuration of an occupant monitoring device according to embodiment 1. Fig. 6 is a flowchart showing a processing operation of the occupant monitoring device according to embodiment 1. Fig. 7 is a block diagram showing an occupant state detection unit according to embodiment 2.

[0012] Embodiment 1. Fig. 1 is a block diagram showing a portion of a vehicle 1 equipped with an occupant monitoring device 2 according to this embodiment. Fig. 2 is a block diagram showing a vehicle control system 3 of the vehicle 1. The occupant monitoring device 2 constitutes a portion of the vehicle control system 3. The vehicle control system 3 includes an occupant monitoring system 4, a driving control system 5, a braking / driving mechanism 6, and a steering mechanism 7. The occupant monitoring system 4 is made up of the occupant monitoring device 2, an imaging device 8, and an output device 9. The occupant monitoring device 2 constitutes a portion of the occupant monitoring system 4.

[0013] The imaging device 8 is a camera mounted on a movable part within the vehicle 1, and captures images of the interior of the vehicle 1. The imaging device 8 captures an image of an imaging area that extends from the installation position at a preset camera angle of view. The imaging device 8 is installed so that the imaging area includes the faces of the occupants in the vehicle 1, who are the image targets. The imaging device 8 captures images of the occupants in the vehicle 1 at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured images to the occupant monitoring device 2. The captured images may be still images, moving images, or both still images and moving images.

[0014] The output device 9 includes at least one of a display, a speaker, and a vibrator. The display is configured, for example, by a liquid crystal display, an organic EL (Electro-Luminescence) display, or a HUD (Head-Up Display). The display is provided, for example, on the dashboard of the vehicle 1. The speaker is provided, for example, on the dashboard of the vehicle 1. The vibrator is provided, for example, on the steering wheel of the vehicle 1 or on the driver's seat of the vehicle 1.

[0015] The occupant monitoring device 2 monitors the state of the occupant, and when the occupant is in a distracted state or has a low level of alertness, outputs a signal commanding an inattention warning or advance notification of the suppression of the driving support function to the output device 9. As shown in Fig. 1 , the occupant monitoring device 2 includes an image acquisition unit 10, an occupant state detection unit 11, a warning determination unit 12, a notification determination unit 13, and a suppression determination unit 14, each of which is connected to a communication bus 15, and is capable of transmitting and receiving data via the communication bus 15.

[0016] The image acquisition unit 10 acquires the captured image from the imaging device 8 and outputs the acquired image to the occupant state detection unit 11 .

[0017] 3 is a block diagram showing the occupant state detection unit 11. The occupant state detection unit 11 detects the occupant's distracted state or awake state using the captured image acquired by the image acquisition unit 10. The occupant state detection unit 11 includes an eye opening degree determination unit 111, a gaze detection unit 112, a face direction detection unit 113, a drowsiness detection unit 114, an inattentiveness determination unit 115, an eye state determination unit 116, and a dozing determination unit 117.

[0018] The image acquisition unit 10 outputs the captured image acquired from the imaging device 8 to the eye opening degree determination unit 111 , the gaze detection unit 112 , the face direction detection unit 113 and the drowsiness detection unit 114 .

[0019] The eye-opening degree determination unit 111 determines the degree of eye opening of the occupant based on the captured image acquired by the image acquisition unit 10. Specifically, the eye-opening degree determination unit 111 detects the occupant's upper eyelid and lower eyelid using a known image recognition technique on the captured image. For example, an area in the captured image where the occupant's face may be present (hereinafter referred to as a "face detection area") is set in advance, and the eye-opening degree determination unit 111 performs known edge detection on the face detection area of ​​the captured image to extract feature points indicating the occupant's upper eyelid and lower eyelid. Next, the eye-opening degree determination unit 111 calculates, for example, the vertical distance between the occupant's upper eyelid and lower eyelid (hereinafter referred to as "eyelid distance") in the captured image, starting from left to right, and calculates the maximum eyelid distance. Then, the eye-opening degree determination unit 111 determines the degree of eye opening based on the ratio between the calculated maximum eyelid distance and a reference value for the eye-opening degree. The reference value of the eye openness is set to, for example, the average value of the eye openness of an adult of average build who is awake. The reference value of the eye openness may be, for example, the average value of the eye openness of the occupant for a predetermined time after the occupant gets into the vehicle, or the average value of all the eye openness determined by the eye openness determination unit up to now. The above-described method of determining the eye openness is merely an example. The eye openness determination unit 111 may determine the eye openness using various known algorithms. The eye openness determination unit 111 outputs information related to the determination result of the eye openness of the occupant (hereinafter referred to as "eye openness information") to the eye state determination unit 116. The eye openness information is, for example, information that associates the eye openness of the occupant with the captured image acquired by the image acquisition unit.

[0020] The gaze detection unit 112 detects the gaze of the occupant based on the captured image acquired by the image acquisition unit 10. The gaze detection unit 112 may detect the gaze direction of the occupant using a known image recognition technique on the captured image. For example, if the imaging device 8 is a visible light camera, the gaze detection unit 112 detects the inner corners of the occupant's eyes and irises in the captured image and detects the gaze direction of the occupant from the positional relationship between the inner corners of the eyes and the irises. If the imaging device 8 is an infrared camera, the gaze detection unit 112 detects the gaze direction of the occupant from the positional relationship between the pupil and a Purkinje image obtained by reflecting a near-infrared point light source from the infrared camera. Note that these are merely examples, and the gaze detection unit 112 may detect the gaze direction of the occupant using various known algorithms. The gaze direction of the occupant is expressed as an angle with respect to a predetermined reference direction. For example, the gaze direction of the occupant is expressed as a gaze angle that indicates how many degrees the gaze direction has rotated horizontally and vertically from the state in which the occupant is looking straight ahead while sitting in the seat. The gaze detection unit 112 outputs information relating to the detected gaze of the occupant (hereinafter referred to as "gaze information") to the eye state determination unit 116. The gaze information is, for example, information in which information indicating the gaze direction of the occupant is associated with a captured image. If the gaze detection unit 112 is unable to detect the gaze of the occupant, it outputs gaze information indicating that the gaze of the occupant could not be detected to the eye state determination unit 116.

[0021] The face direction detection unit 113 detects the face direction of the occupant based on the captured image acquired by the image acquisition unit 10. The face direction detection unit 113 may detect the face direction of the occupant using a known image recognition technique on the captured image. For example, the face direction detection unit 113 may detect the face direction of the occupant using general pattern recognition based on the captured image. This is merely one example, and the face direction detection unit 113 may detect the face direction of the occupant using various known algorithms. The face direction of the occupant is expressed as an angle with respect to a predetermined reference direction. For example, the face direction of the occupant is expressed as a face direction angle that indicates how many degrees the face direction has rotated horizontally and vertically from a state in which the occupant is looking straight ahead in the seat. The face direction detection unit 113 outputs information regarding the detected face direction of the occupant (hereinafter referred to as "face direction information") to the eye state determination unit 116. The face direction information is, for example, information that associates the face direction of the occupant with the captured image.

[0022] The drowsiness detection unit 114 detects the drowsiness level of the occupant based on the captured image acquired by the image acquisition unit 10. The drowsiness detection unit 114 may detect the drowsiness level of the occupant using a known image recognition technique on the captured image. For example, the drowsiness detection unit 114 may determine the number of blinks of the occupant per a predetermined unit time based on the captured image. The drowsiness detection unit 114 detects the drowsiness level of the occupant based on the number of blinks per unit time. The drowsiness level corresponding to the number of blinks per unit time is determined in advance. For example, the drowsiness level may be set to a value such that a higher drowsiness level indicates greater drowsiness of the occupant, and a lower drowsiness level indicates a more awake occupant. The drowsiness detection unit 114 may detect the drowsiness level of the occupant using various known algorithms. Note that, although the drowsiness detection unit 114 detects the drowsiness level of the occupant based on the captured image in this example, this is merely an example. The drowsiness detection unit 114 may, for example, acquire biological information of the occupant, such as body temperature, from a biological sensor provided in the vehicle 1 and detect the drowsiness level of the occupant from the biological information. The drowsiness detection unit 114 outputs information related to the detected drowsiness level of the occupant (hereinafter referred to as "drowsiness level information") to the dozing determination unit 117. The drowsiness level information is, for example, information in which the drowsiness level of the occupant is associated with a captured image.

[0023] When the eye opening degree of the occupant determined by the eye opening degree determination unit 111 is less than a first threshold, the eye state determination unit 116 determines whether the occupant has their eyes downcast or closed, based on the gaze direction of the occupant detected by the gaze detection unit 112 before it was determined that the eye opening degree was less than the first threshold and the facial direction of the occupant detected by the facial direction detection unit. Specifically, when the eye opening degree determined by the eye opening degree determination unit 111 is less than the first threshold, if the gaze direction of the occupant detected by the gaze detection unit 112 is directed downward and the degree of eye opening becomes less than the first threshold after the facial direction of the occupant detected by the facial direction detection unit 113 is directed downward, the eye state determination unit 116 determines that the occupant has their eyes downcast. On the other hand, the eye state determination unit 116 determines that the occupant has their eyes closed if the gaze direction of the occupant detected by the gaze detection unit 112 is directed downward and the degree of eye opening does not become less than the first threshold after the occupant's facial direction detected by the face direction detection unit 113 has turned downward. The determination of whether the occupant has their eyes down or closed, performed by the eye state determination unit 116, is also referred to as a "downcast eyes determination." The eye state determination unit 116 determines whether the degree of eye opening of the occupant has become less than the first threshold based on the eye opening degree information output from the eye opening degree determination unit 111. Note that if the degree of eye opening of the occupant determined by the eye opening degree determination unit 111 is equal to or greater than the first threshold, the eye state determination unit 116 does not perform a downcast eyes determination and determines that the occupant has their eyes open. The downcast eyes determination process performed by the eye state determination unit 116 is described below.

[0024] Eye state determination unit 116 stores, in chronological order, the eye opening degree information output from eye opening degree determination unit 111, the gaze information output from gaze detection unit 112, and the facial direction information output from facial direction detection unit 113. The information stored in chronological order by eye state determination unit 116, in which the eye opening degree information, gaze information, and facial direction information are associated with each other, is referred to as "eye-related information."

[0025] When the eye state determination unit 116 determines that the degree of eye opening of the occupant has become less than the first threshold, it refers to the eye-related information and obtains the gaze information of the occupant before the degree of eye opening of the occupant became less than the first threshold.

[0026] Next, based on the acquired gaze information of the occupant, the eye state determination unit 116 calculates a trajectory of the occupant's gaze direction from the gaze direction of the occupant detected by the gaze detection unit 112 before it was determined that the degree of eye opening was less than the first threshold. The eye state determination unit 116 refers to the eye-related information to acquire facial direction information of the occupant before the degree of eye opening of the occupant became less than the first threshold. Based on the acquired facial direction information of the occupant, the eye state determination unit 116 calculates a trajectory of the occupant's facial direction from the facial direction of the occupant detected by the facial direction detection unit 113 before it was determined that the degree of eye opening was less than the first threshold.

[0027] The eye state determination unit 116 determines whether the calculated trajectory of the occupant's gaze direction is directed downward and whether the calculated trajectory of the occupant's facial direction is directed downward. Specifically, the eye state determination unit 116 determines whether the calculated trajectory of the occupant's gaze direction is directed toward a predetermined downward viewing area and whether the calculated trajectory of the occupant's facial direction is directed toward the downward viewing area. If the trajectory of the occupant's gaze direction is directed toward the downward viewing area, the eye state determination unit 116 determines that the occupant's gaze direction is directed downward. Furthermore, if the trajectory of the occupant's facial direction is directed toward the downward viewing area, the eye state determination unit 116 determines that the occupant's facial direction is directed downward. The downward viewing area is an area set below the image capture device 8 within the vehicle 1. The downward viewing area is set as an area where it is assumed that the occupant may be looking aside if, for example, the occupant's line of sight remains within the area while looking downward for a predetermined period of time. For example, the downward viewing area is set as an area including an area where devices that the occupant can operate below the imaging device 8, such as the display of a car navigation device, are located.

[0028] The eye state determination unit 116 determines that the occupant has their eyes downcast if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are pointing downward. In other words, the eye state determination unit 116 determines that the occupant has their eyes downcast if the occupant's line of sight and the occupant's facial direction tend to be pointing downward and the degree to which the occupant's eyes are open is less than a first threshold.

[0029] The eye state determination unit 116 determines that the occupant has their eyes closed if the trajectory of the occupant's line of sight and the trajectory of the occupant's facial direction are not pointing downward. In other words, the eye state determination unit 116 determines that the occupant has their eyes closed unless the degree of eye opening of the occupant is less than the first threshold while the occupant's line of sight and the occupant's facial direction tend to point downward.

[0030] The eye state determination unit 116 outputs information indicating whether the occupant has their eyes downcast, closed, or open (hereinafter referred to as "eye state information") to the inattentive driving determination unit 115 and the drowsy driving determination unit 117.

[0031] When the eye state determination unit 116 outputs eye state information indicating that the occupant is looking down, in other words, when the eye state determination unit 116 determines that the occupant is looking down, the inattentiveness determination unit 115 determines whether the occupant is looking aside (hereinafter referred to as "inattentiveness determination") based on whether a preset condition (hereinafter referred to as "condition for inattentiveness determination") is satisfied. An example of the condition for inattentiveness determination is that "the occupant's downcast eyes continue for a preset time period (hereinafter referred to as "time period for inattentiveness determination") or more."

[0032] When the eye state determination unit 116 outputs eye state information indicating that the occupant is looking down, the inattentive driving determination unit 115 counts up the time (hereinafter referred to as the "time since downcast eye determination"). If the counted up time since downcast eye determination is equal to or greater than the inattentive driving determination time, the inattentive driving determination unit 115 determines that the occupant's downcast eye state has continued for equal to or greater than the inattentive driving determination time, and determines that the occupant is looking aside. The inattentive driving determination unit 115 stores the counted up time since downcast eye determination. On the other hand, if the inattentive driving determination unit 115 determines that the occupant's downcast eye state has not continued for equal to or greater than the inattentive driving determination time, it determines that the occupant is not looking aside. If the inattentive driving determination unit 115 determines that the occupant is looking aside as a result of its inattentive driving determination, it outputs information that the occupant is looking aside and information indicating the time since downcast eye determination to the warning determination unit 12 and the notification determination unit 13.

[0033] When the eye state determination unit 116 outputs eye state information indicating that the occupant has closed eyes, in other words, when the eye state determination unit 116 determines that the occupant has closed eyes, the drowsiness determination unit 117 determines whether the occupant is dozing off (hereinafter also referred to as "drowsiness determination") based on the drowsiness level of the occupant detected by the drowsiness detection unit 114. The drowsiness determination unit 117 may identify the drowsiness level of the occupant from the drowsiness level information output from the drowsiness detection unit 114. Specifically, the drowsiness determination unit 117 determines that the occupant is dozing off when a state (hereinafter referred to as a "drowsiness state") in which the occupant's drowsiness level is equal to or higher than a preset threshold (hereinafter referred to as a "drowsiness determination threshold") continues for a preset time (hereinafter referred to as a "drowsiness determination time") or more. The doze determination unit 117 determines that the occupant is in a drowsy state when the eye state determination unit 116 outputs eye state information indicating that the occupant has closed eyes and the occupant's drowsiness level detected by the drowsiness detection unit 114 is equal to or greater than a drowsiness determination threshold. When the doze determination unit 117 determines that the occupant is in a drowsy state, it counts up a time (hereinafter referred to as the "post-drowsiness determination time"). The doze determination unit 117 stores the counted post-drowsiness determination time. When the counted post-drowsiness determination time is equal to or greater than the doze determination time, the doze determination unit 117 determines that the occupant's drowsy state has continued for equal to or greater than the doze determination time, and determines that the occupant is dozing. When the doze determination unit 117 determines that the occupant is dozing as a result of the doze determination, it outputs information indicating that the occupant is dozing to the notification determination unit 13. The above is the configuration of the occupant state detection unit 11.

[0034] When the inattentiveness determination unit 115 outputs information indicating that the occupant is in an inattentive state, in other words, when the inattentiveness determination unit 115 determines that the occupant is in an inattentive state, the warning determination unit 12 determines whether or not a warning is necessary for the occupant based on the level of inattentiveness of the occupant determined by the inattentiveness determination unit 115 (hereinafter also referred to as a "warning determination"). Specifically, when the time after downcast eye determination output from the inattentiveness determination unit 115 is equal to or longer than the warning determination time, the warning determination unit 12 determines that a warning is necessary for the occupant. On the other hand, when the time after downcast eye determination output from the inattentiveness determination unit 115 is shorter than the warning determination time, the warning determination unit 12 determines that a warning is not necessary for the occupant. When the warning determination unit 12 determines that a warning is necessary for the occupant as a result of the warning determination, it outputs a warning command to the output device 9.

[0035] When the inattentive driving determination unit 115 outputs information indicating that the occupant is in an inattentive state and the drowsiness determination unit 117 outputs information indicating that the occupant is in a drowsy state, in other words, when the inattentive driving determination unit 115 determines that the occupant is in an inattentive state and the drowsiness determination unit 117 determines that the occupant is in a drowsy state, the notification determination unit 13 determines whether or not to notify the occupant in advance that the driving assistance function being executed by the driving control device 16 will be suppressed (hereinafter also referred to as "notification determination"). Specifically, when the time after the downcast eye determination output from the inattentive driving determination unit 115 is equal to or longer than the warning determination time and the drowsiness determination unit 117 outputs information indicating that the occupant is in a drowsy state, the notification determination unit 13 determines that an advance notification to the occupant is necessary. On the other hand, when the time after the downcast eye determination output from the inattentive driving determination unit 115 is shorter than the warning determination time, the notification determination unit 13 determines that an advance notification to the occupant is not necessary. When the notification determination unit 13 determines, as a result of the notification determination, that advance notification to the occupants is required, it outputs information to the suppression determination unit 14 to the effect that advance notification to the occupants will be executed, and outputs a notification command to the output device 9. On the other hand, when the notification determination unit 13 determines, as a result of the notification determination, that advance notification to the occupants is not required, it outputs information to the suppression determination unit 14 to the effect that advance notification to the occupants is not required.

[0036] The determination criteria of the notification determination unit 13 are set by adding a different type of determination criterion, "the occupant is dozing," to the determination criteria of the warning determination unit 12. That is, the determination criteria of the warning determination unit 12 are more relaxed than the determination criteria of the notification determination unit 13. Therefore, even if the warning determination unit 12 determines that a warning is necessary based on an erroneous detection by the occupant state detection unit 11, the notification determination unit 13 does not necessarily determine that a notification is necessary. This reduces the suppression of the driving assistance function based on the erroneous detection by the occupant state detection unit 11. Furthermore, the determination criteria of the notification determination unit 13 are set by adding a different type of determination criterion to the determination criteria of the warning determination unit 12. Therefore, even if the determination criterion of the warning determination unit 12 is satisfied based on an erroneous detection by the occupant state detection unit 11, a determination that a notification is necessary is not made unless the different additional criterion is satisfied. This reduces the need for a notification, thereby suppressing notification due to an erroneous detection by the occupant state detection unit 11.

[0037] When the notification determination unit 13 outputs information indicating that advance notification to the occupants is required, in other words, when the notification determination unit 13 determines that advance notification to the occupants is required, the suppression determination unit 14 counts up the number of notifications. Then, when the cumulative number of notifications reaches or exceeds the number of times that it is determined that suppression of the driving support function is required (hereinafter also referred to as the "number of suppression determinations"), the suppression determination unit 14 determines that suppression of the driving support function is required, and outputs information indicating that suppression of the driving support function is required to the driving control device 16. Furthermore, when the notification determination unit 13 outputs information indicating that advance notification to the occupants is not required, the suppression determination unit 14 resets the cumulative number of notifications that had been counted up up to that point.

[0038] The determination criterion of the suppression determination unit 14 is "the cumulative number of notifications is equal to or greater than the suppression determination number," and the determination criterion of the notification determination unit 13 is independent of the determination criterion of the warning determination unit 12. Therefore, since the necessity of suppressing the driving assistance function is determined independently of the warning, even if a warning is issued due to a false detection, the driving assistance function is not suppressed. Furthermore, since the driving assistance function is suppressed based on the cumulative number of notifications, the possibility of suppressing the driving assistance function due to a false detection can be reduced.

[0039] When a warning command is output from the warning determination unit 12, in other words, when the inattentiveness determination unit 115 determines that the occupant is looking aside, the output device 9 outputs information (hereinafter referred to as "warning information") to warn the occupant that they are not paying attention to the road ahead. Also, when a notification command is output from the notification determination unit 13, in other words, when the notification determination unit 13 determines that the driving assistance function needs to be suppressed, the output device 9 outputs information (hereinafter referred to as "driving assistance function suppression information") to the occupant that the driving assistance function will be suppressed.

[0040] When the output device 9 is, for example, a display, the display displays a message corresponding to the warning information or the driving assistance function suppression information. As a display example, when the warning information is output, the output device 9 displays a message saying "Watch out ahead." Furthermore, when the driving assistance function is, for example, a lane departure prevention function (LKA, Lane Keeping Assist) and the output device 9 outputs information to suppress the lane departure prevention function, the output device 9 displays a message saying, for example, "Assist function will be stopped. Please stay within the lane." The output device 9 may also display an icon encouraging the driver to stay within the lane.

[0041] When the output device 9 is an audio output device, the audio output device outputs audio corresponding to the warning information or the driving assistance function suppression information. When outputting warning information, the output device outputs audio such as "Watch out ahead." When the driving assistance function is a lane departure prevention function, the output device 9 outputs audio such as "The assist function will be stopped. Please stay within your lane."

[0042] As described above, the warning method and the method for notifying the driver in advance of the suppression of the driving support function are different, so the driver can distinguish between the two. Therefore, the driver can clearly understand in advance that the driving support function will be suppressed, which reduces the psychological burden on the driver.

[0043] Next, we will explain the configuration of the driving control system 5 that provides driving assistance using the detection results of the occupant monitoring system 4. As shown in Figure 2, the driving control system 5 is configured to be able to communicate with the occupant monitoring system 4 wirelessly or via a wire.

[0044] The driving control system 5 includes a driving control device 16 that controls the steering mechanism 7 and braking / driving mechanism 6 of the vehicle 1 by outputting control signals based on the detection results of the occupant monitoring system 4. The driving control system 5 also includes a map information storage device (not shown) that stores map information, a surrounding condition monitoring device 17 that monitors the surrounding conditions of the vehicle, and a vehicle state acquisition device 18 that acquires information indicating the state of the vehicle 1. The occupant monitoring system 4, the driving control device 16, the surrounding condition monitoring device 17, and the vehicle state acquisition device 18 included in the driving control system 5 are each connected to a communication bus 15, and are capable of transmitting and receiving data via the communication bus 15.

[0045] The braking / driving mechanism 6 is a mechanism for controlling the traveling speed of the vehicle 1 and for switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 61. The steering mechanism 7 is a mechanism for determining the traveling direction of the vehicle 1, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 71. The braking / driving actuator 61 that controls the braking / driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, and the steering actuator 71 that controls the steering mechanism 7 is composed of, for example, an EPS (Electric Power Steering) motor, and the like.

[0046] The surrounding condition monitoring device 17 monitors the conditions around the vehicle 1 and includes a GPS (Global Positioning System) receiver 171, an on-board communication device 172, an external sensor 173, and a navigation system 174. The GPS receiver 171 receives signals transmitted from GPS positioning satellites to detect the current location of the vehicle 1. The on-board communication device 172 communicates with communication devices of other vehicles or communication devices installed on the road to acquire information on the locations of other vehicles or pedestrians, traffic information, etc. Here, traffic information includes, for example, congestion information, traffic regulation information, construction zone information, etc. The on-board communication device 172 can also perform vehicle-to-vehicle communication via wireless communication with on-board communication devices of other vehicles around the vehicle 1. In addition, the on-board communication device 172 may perform mobile communication with a base station external to the vehicle 1. The external sensor 173 is, for example, composed of at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and detects the positions of vehicles, pedestrians, obstacles, etc. that are present around the vehicle 1 or the distance from the vehicle 1. The navigation system 174 calculates a route from the current position of the vehicle 1 to a destination and provides guidance along the calculated route. The navigation system 174 has a display unit composed of, for example, a liquid crystal display, and is housed in the instrument panel. The navigation system 174 also has an operation unit such as a touch panel or physical buttons, and is configured to be able to accept operations by the occupant. Note that the operation unit of the navigation system 174 may be composed of a microphone or the like to be able to accept operations via voice uttered by the occupant.

[0047] The vehicle state acquisition device 18 acquires information indicating the state of the vehicle 1 and includes a steering angle sensor 181, a vehicle speed sensor 182, a steering torque sensor 183, an accelerator position sensor 184, and a brake position sensor 185. The steering angle sensor 181 is provided, for example, on an EPS (Electric Power Steering) motor or a steering wheel and detects the steering angle of the vehicle 1. The vehicle speed sensor 182 is provided, for example, on a wheel and detects the traveling speed of the vehicle 1. The steering torque sensor 183 is provided, for example, on the steering wheel and detects the magnitude of the steering wheel operation force applied by the driver. The accelerator position sensor 184 detects the amount of accelerator pedal depression by the driver. The brake position sensor 185 detects the amount of brake pedal depression by the driver.

[0048] The driving control device 16 of the driving control system 5 controls a braking / driving actuator 61 or a steering actuator 71 mounted on the vehicle 1 to assist the driver in driving. The driving control device 16 performs automatic driving control of the vehicle 1, for example, by outputting signals to the braking / driving actuator 61 or the steering actuator 71. The driving control device 16 performs braking / driving control of the vehicle 1, such as by controlling the braking / driving actuator 61, which is composed of an electronically controlled throttle and a brake actuator, to activate the brakes and decelerate or stop the vehicle 1. The driving control device 16 performs steering control of the vehicle 1, such as by controlling the steering actuator 71, which is composed of an EPS motor, to maintain the lane in which the vehicle 1 is traveling. Furthermore, the driving control device 16 has multiple driving assistance functions that assist or substitute for the driver's driving operation by controlling the driving force, braking force, steering force, etc. of the vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function is referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function is referred to as LKA (Lane Keeping Assist).

[0049] When the driving control device 16 executes ACC, it controls the traveling speed of the vehicle 1 by adjusting the driving force and braking force based on monitoring information of the vehicle ahead obtained from the surrounding conditions monitoring device 17. If no vehicle ahead is detected, the driving control device 16 causes the vehicle 1 to travel at a constant speed at a target speed set in advance by the driver or the like. On the other hand, if a vehicle ahead is detected, the ACC causes the vehicle 1 to follow the vehicle ahead while maintaining a distance from the vehicle ahead.

[0050] Furthermore, when the driving control device 16 executes LKA, the driving control device 16 controls the steering force and steering holding force based on shape information of the lane markings in the traveling direction acquired from the surrounding conditions monitoring device 17. LKA causes the vehicle 1 to travel along the lane by applying a steering force to the steering wheel in a direction that prevents the vehicle from approaching the lane markings. Note that road information output by the surrounding conditions monitoring device 17 may be used for vehicle control by ACC and LKA.

[0051] Furthermore, when the driving control device 16 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) to automatically stop the vehicle 1. When the automatic evacuation control is started, the driving control device 16 causes the surrounding condition monitoring device 17 to search for an evacuation location where the vehicle 1 should be stopped. Then, the driving control device 16 moves the vehicle 1 to the evacuation location set by the surrounding condition monitoring device 17 and stops the vehicle 1. Note that, on expressways, the above-mentioned evacuation location may be a shoulder outside the lane in which the vehicle 1 is traveling, and on ordinary roads, it may be a location that avoids a shoulder outside the lane in which the vehicle 1 is traveling, as well as intersections, railroad crossings, sidewalks, and locations where there is a high possibility of the presence of moving objects such as other vehicles, trains, and pedestrians.

[0052] 4 and 5 are diagrams showing the hardware configurations of the occupant monitoring device 2 and the operation control device 16. The hardware configurations of the occupant monitoring device 2 and the operation control device 16 will be described with reference to FIGS. 4 and 5. As an example, as shown in FIG. 4, the occupant monitoring device 2 is implemented by a processing circuit 19. The processing circuit 19 implements the functions of the functional units of the occupant monitoring device 2. The processing circuit 19 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of each functional unit of the occupant monitoring device 2 may be implemented by separate processing circuits, or these functions may be implemented collectively by a single processing circuit. The operation control device 16 can also be implemented by a similar configuration.

[0053] 5, the occupant monitoring device 2 is implemented by a processor 20 and a memory 21. The functions of the functional units of the occupant monitoring device 2 are implemented by reading and executing a program stored in the memory 21 by the processor 20. The program is implemented as software, firmware, or a combination of software and firmware. Examples of the memory 21 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

[0054] Next, an occupant monitoring method by the occupant monitoring device 2 will be described. Fig. 6 is a flowchart showing the processing operation of the occupant monitoring device 2 according to the first embodiment. The occupant monitoring device 2 starts operation, for example, when the imaging device 8 is activated, and repeats the processing operation while the vehicle 1 is traveling. The following description will be given assuming that the occupant state detection unit 11 is detecting an occupant at "Start (A)" in Fig. 6.

[0055] In step S1 , the image acquisition unit 10 acquires a captured image from the imaging device 8 . The image acquisition unit 10 outputs the acquired captured image to the occupant state detection unit 11 .

[0056] In step S2, the eye state determination unit 116 determines whether the occupant has their eyes downcast, whether the occupant has their eyes closed, or whether the occupant has their eyes open, and outputs the eye state information to the inattentive determination unit 115 and the drowsiness determination unit 117. If the inattentive determination unit 115 determines that the occupant is looking aside, it outputs information that the occupant is looking aside and information indicating the time since the downcast eyes determination to the warning determination unit 12 and the notification determination unit 13. If the drowsiness determination unit 117 determines that the occupant is dozing, it outputs information that the occupant is dozing and information indicating the time since the drowsiness determination to the notification determination unit 13.

[0057] In step S3, the warning determination unit 12 determines whether the time after the downcast eye determination output from the inattentive eye determination unit 115 is equal to or greater than the warning determination time. If the time after the downcast eye determination is equal to or greater than the warning determination time (YES in step S3), the warning determination unit 12 outputs a warning command to the output device 9 and also outputs information to the image acquisition unit 10 instructing it to acquire a new captured image (return to start "A"). If the time after the downcast eye determination is less than the warning determination time (NO in step S3), the warning determination unit 12 outputs information to the image acquisition unit 10 instructing it to acquire a new captured image (return to start "A").

[0058] In step S4, the output device 9 outputs a warning to the occupant when the warning command is output from the warning determination unit 12. Thereafter, the image acquisition unit 10 acquires a new captured image.

[0059] In steps S5 and S6, processing is carried out in parallel with steps S3 and S4. In step S5, if the notification determination unit 13 determines that the occupant is inattentive and that the occupant is dozing, it outputs a notification command to the output device 9 and the suppression determination unit 14. When the notification command is output from the notification determination unit 13, the output device 9 notifies the occupant in advance. If the notification command is output multiple times and the suppression determination unit 14 determines that the driving assistance function needs to be suppressed (YES in step S5), it outputs information to the driving control device 16 that the driving assistance function needs to be suppressed and outputs information to the image acquisition unit 10 that instructs the image acquisition unit 10 to acquire a new captured image (return to start "A"). If the notification determination unit 13 outputs information indicating that a notification is not required (NO in step S5), the suppression determination unit 14 resets the cumulative number of notifications that had been counted up to that point to zero and outputs information to instruct the image acquisition unit 10 to acquire a new captured image (return to start "A").

[0060] In step S6, when the information indicating that the driving assistance function needs to be suppressed is output, the driving control device 16 suppresses the driving assistance function by controlling the steering mechanism 7 and the braking / driving mechanism 6. After that, the image acquisition unit 10 acquires a new captured image.

[0061] As described above, in the occupant monitoring device 2 according to the first embodiment, whether or not the driving assistance function needs to be suppressed is determined regardless of the warning, so even if a warning is issued due to a false detection, the driving assistance function is not suppressed. Furthermore, since the driving assistance function is suppressed based on the cumulative number of warnings, it is possible to reduce the possibility of the driving assistance function being suppressed due to a false detection.

[0062] Furthermore, in the occupant monitoring device 2 according to the first embodiment, even if the warning determination unit 12 determines that a warning is necessary based on an erroneous detection by the occupant state detection unit 11, the notification determination unit 13 does not necessarily determine that a notification is necessary, thereby reducing suppression of the driving assistance function based on an erroneous detection by the occupant state detection unit 11. Furthermore, even if the determination criterion of the warning determination unit 12 is satisfied based on an erroneous detection by the occupant state detection unit 11, a determination that a notification is necessary is not made unless a different additional criterion is satisfied, thereby making it possible to suppress notification due to an erroneous detection by the occupant state detection unit 11.

[0063] Furthermore, in the occupant monitoring device 2 according to embodiment 1, the method of warning and the method of notifying in advance of the suppression of the driving assistance function are different, so that the occupant can clearly understand in advance that the driving assistance function will be suppressed, thereby reducing the psychological burden on the occupant.

[0064] Embodiment 2. An occupant monitoring device 2 according to embodiment 2 will now be described. Figure 7 is a block diagram showing a portion of a vehicle 1 equipped with an occupant monitoring device 2 according to embodiment 2. The occupant monitoring device 2 according to embodiment 2 uses only the inattentive state of an occupant as a criterion for determining whether to issue a warning or notification. The occupant monitoring device 2 according to embodiment 2 differs from the occupant monitoring device 2 according to embodiment 1 in that it does not include a drowsiness detection unit or a dozing-off determination unit, and in that the warning determination unit 12 and the notification determination unit 13 make determinations based on the same type of determination criterion (the time since the downcast eye determination).

[0065] The inattentiveness determination unit 115 performs an inattentiveness determination when the eye state information is output from the eye state determination unit 116. If the inattentiveness determination unit 115 determines that the occupant is looking aside as a result of the inattentiveness determination, the inattentiveness determination unit 115 outputs information that the occupant is in an inattentive state and information indicating the time since the occupant was determined to be looking down to the warning determination unit 12 and the notification determination unit 13.

[0066] When the inattentive driving determination unit 115 outputs information indicating an inattentive state and information indicating the time since the downcast eye determination, the warning determination unit 12 performs a warning determination, and if the time since the downcast eye determination is equal to or longer than the warning determination time, the warning determination unit 12 determines that a warning to the occupant is necessary. On the other hand, if the time since the downcast eye determination output from the inattentive driving determination unit 115 is shorter than the warning determination time, the warning determination unit 12 determines that a warning to the occupant is not necessary. If the warning determination unit 12 determines that a warning to the occupant is necessary as a result of the warning determination, it outputs a warning command to the output device 9.

[0067] The notification determination unit 13 performs a notification determination when the inattentive driving determination unit 115 outputs information indicating an inattentive driving state and information indicating the time since the downcast eye determination. In the second embodiment, the notification determination determines whether the time since the downcast eye determination is equal to or longer than the notification determination time, which is longer than the warning determination time. If the time since the downcast eye determination is equal to or longer than the notification determination time, the notification determination unit 13 determines that an advance notification to the occupant is required. On the other hand, if the time since the downcast eye determination output from the inattentive driving determination unit 115 is shorter than the notification determination time, the notification determination unit 13 determines that an advance notification to the occupant is not required. If the notification determination unit 13 determines that an advance notification to the occupant is required as a result of the notification determination, the notification determination unit 13 outputs information to the suppression determination unit 14 indicating that an advance notification to the occupant will be executed, and outputs a notification command to the output device 9. On the other hand, if the notification determination unit 13 determines, as a result of the notification determination, that advance notification to the occupants is not required, it outputs information to the suppression determination unit 14 that advance notification to the occupants is not required.

[0068] Next, an occupant monitoring method by the occupant monitoring device 2 will be described with reference to Fig. 6. The occupant monitoring device 2 starts operation, for example, when the imaging device 8 is activated, and repeats the operation while the vehicle 1 is traveling. The following description will be given assuming that the occupant state detection unit 11 is detecting an occupant at "Start (A)" in Fig. 6.

[0069] In step S1 , the image acquisition unit 10 acquires a captured image from the imaging device 8 . The image acquisition unit 10 outputs the acquired captured image to the occupant state detection unit 11 .

[0070] In step S2, the eye state determination unit 116 determines whether the occupant has their eyes downcast, their eyes closed, or their eyes open, and outputs the eye state information to the inattentive driving determination unit 115. If the inattentive driving determination unit 115 determines that the occupant is inattentive, it outputs information that the occupant is inattentive and information indicating the time since the inattentive driving determination was made to the warning determination unit 12 and the notification determination unit 13.

[0071] In step S3, the warning determination unit 12 determines whether the time after the downcast eye determination output from the inattentive eye determination unit 115 is equal to or greater than the warning determination time. If the time after the downcast eye determination is equal to or greater than the warning determination time (YES in step S3), the warning determination unit 12 outputs a warning command to the output device 9 and also outputs information to the image acquisition unit 10 instructing it to acquire a new captured image (return to start "A"). If the time after the downcast eye determination is less than the warning determination time (NO in step S3), the warning determination unit 12 outputs information to the image acquisition unit 10 instructing it to acquire a new captured image (return to start "A").

[0072] In step S4, the output device 9 outputs a warning to the occupant when the warning command is output from the warning determination unit 12. Thereafter, the image acquisition unit 10 acquires a new captured image.

[0073] In steps S5 and S6, processing is performed in parallel with steps S3 and S4. In step S5, if the notification determination unit 13 determines that the occupant is inattentive, it outputs a notification command to the output device 9 and the suppression determination unit 14. When the notification command is output from the notification determination unit 13, the output device 9 notifies the occupant in advance. If the notification command is output multiple times and the suppression determination unit 14 determines that suppression of the driving assistance function is required (YES in step S5), it outputs information indicating that suppression of the driving assistance function is required to the driving control device 16 and outputs information indicating that the image acquisition unit 10 should acquire a new captured image (return to start "A"). If the notification determination unit 13 outputs information indicating that a notification is not required (NO in step S5), the suppression determination unit 14 resets the cumulative number of notifications that had been counted up to that point to zero and outputs information indicating that the image acquisition unit 10 should acquire a new captured image (return to start "A").

[0074] In step S6, when the information indicating that the driving assistance function needs to be suppressed is output, the driving control device 16 suppresses the driving assistance function by controlling the steering mechanism 7 and the braking / driving mechanism 6. After that, the image acquisition unit 10 acquires a new captured image.

[0075] As described above, in the occupant monitoring device 2 according to the second embodiment, the judgment criterion of the warning judgment unit 12 and the judgment criterion of the notification judgment unit 13 are the same type of judgment criterion, namely, "time after downcast eye judgment," and both judgment criteria can be set by a relatively easy method in which the warning judgment time is shorter than the notification judgment time.

[0076] The above describes in detail preferred embodiments, but the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the claims.

[0077] In the first embodiment, the determination criteria of the warning determination unit 12 are relaxed compared to the determination criteria of the notification determination unit 13, but this is not limiting. For example, the determination criteria of the notification determination unit 13 may be relaxed compared to the determination criteria of the warning determination unit 12. That is, since the determination criteria of the warning determination unit 12 are stricter than the determination criteria of the notification determination unit 13, a warning due to an erroneous detection by the occupant state detection unit 11 can be suppressed. On the other hand, a notification of suppression of the driving assistance function due to an erroneous detection by the occupant state detection unit 11 may be issued. However, by issuing the notification before the warning, there is a grace period between the warning and the suppression of the driving assistance function, which can suppress the occupant from becoming careless.

[0078] In the first and second embodiments, the occupant monitoring device 2 determines whether a warning and a notification are necessary depending on the occupant's inattentive state and drowsy state (only the inattentive state in the second embodiment), but the criteria for the determination are not limited to the inattentive state or drowsy state. For example, the occupant's line of sight when looking aside, the occupant's driving posture, the occupant's grip on the steering wheel, etc. may also be used as the criteria for the determination.

[0079] REFERENCE SIGNS LIST 1 Vehicle, 2 Occupant monitoring device, 3 Vehicle control system, 4 Occupant monitoring system, 5 Driving control system, 6 Braking / driving mechanism, 7 Steering mechanism, 8 Imaging device, 9 Output device, 10 Image acquisition unit, 11 Occupant state detection unit, 12 Warning determination unit, 13 Notification determination unit, 14 Suppression determination unit, 15 Communication bus, 16 Driving control device, 17 Surrounding condition monitoring device, 18 Vehicle state acquisition device, 19 Processing circuit, 20 Processor, 21 Memory

Claims

1. An occupant monitoring device comprising: an image acquisition unit that acquires images of occupants in a vehicle; an occupant state detection unit that uses the images acquired by the image acquisition unit to detect whether the occupant is in a distracted state or an awake state; a warning determination unit that uses the results detected by the occupant state detection unit to determine whether a warning is needed to the occupant; a suppression determination unit that uses the results detected by the occupant state detection unit to determine whether a driving assistance function in the vehicle needs to be suppressed; and a notification determination unit that uses the results detected by the occupant state detection unit to determine whether an alert is needed to notify the occupant in advance that the driving assistance function will be suppressed, wherein the suppression determination unit determines whether the driving assistance function needs to be suppressed based on the cumulative number of times the alert has been issued, and the determination criteria of the notification determination unit are independent of the determination criteria of the warning determination unit.

2. The passenger monitoring device according to claim 1, wherein the judgment criteria of the warning judgment section are relaxed compared to the judgment criteria of the notification judgment section.

3. The passenger monitoring device according to claim 1, wherein the judgment criteria of the notification judgment section are relaxed compared to the judgment criteria of the warning judgment section.

4. An occupant monitoring device as described in claim 2 or 3, characterized in that the judgment criteria of the warning judgment unit and the judgment criteria of the notification judgment unit are the same type of judgment criteria, and the range in which the warning judgment unit judges that a warning is necessary and the range in which the notification judgment unit judges that a notification is necessary are different from each other.

5. An occupant monitoring device according to claim 2 or 3, characterized in that the judgment criteria of the warning judgment unit and the judgment criteria of the notification judgment unit are different, and one judgment criterion is a different type of judgment criterion added to the other judgment criterion.

6. An occupant monitoring device according to any one of claims 1 to 5, characterized in that the notification method and the warning method are different from each other.

7. An occupant monitoring method comprising the steps of: acquiring an image of an occupant in a vehicle; detecting a distracted or awake state of the occupant using the captured image; determining whether or not a warning is required for the occupant using the result of detecting the distracted or awake state of the occupant; determining whether or not a driving assistance function in the vehicle needs to be suppressed using the result of detecting the distracted or awake state of the occupant; and determining whether or not an alert is required to notify the occupant in advance that the driving assistance function will be suppressed using the result of detecting the distracted or awake state of the occupant; wherein in the step of determining whether or not suppression is required, a determination is made as to whether or not the driving assistance function needs to be suppressed based on a cumulative number of alerts, and the determination criterion in the step of determining whether or not the alert is required is independent of the determination criterion in the step of determining whether or not a warning is required.

8. An occupant monitoring program that causes a computer to function as an image acquisition unit that acquires images of occupants in a vehicle; an occupant state detection unit that uses the images acquired by the image acquisition unit to detect whether the occupant is distracted or awake; a warning determination unit that uses the results detected by the occupant state detection unit to determine whether a warning needs to be issued to the occupant; a suppression determination unit that uses the results detected by the occupant state detection unit to determine whether a driving assistance function in the vehicle needs to be suppressed; and a notification determination unit that uses the results detected by the occupant state detection unit to determine whether an alert needs to be issued to the occupant in advance to notify them that the driving assistance function will be suppressed, wherein the suppression determination unit determines whether the driving assistance function needs to be suppressed based on the cumulative number of times the alert has been issued, and the determination criteria of the suppression determination unit are independent of the determination criteria of the warning determination unit.

9. A vehicle control system comprising: an occupant monitoring device according to any one of claims 1 to 6; a vehicle control unit that controls braking and driving of the vehicle to execute the driving assistance function; a warning unit that warns of the occupant's distracted state or low alertness; and a notification unit that notifies of the suppression of the driving assistance function.

Images