Occupant monitoring device, occupant monitoring method, occupant monitoring program, and vehicle control system

Abstract

In the present invention, when vehicle information acquired from a vehicle control device (10) that controls a vehicle (1) indicates that the vehicle is being manually operated by an occupant, a control unit (19) of an occupant monitoring device (2) causes an image acquisition unit (13) to acquire a captured image and causes a calibration processing unit (16) to perform a calibration process, regardless of whether or not an occupant state detection process is being carried out by an occupant state detection unit (15) and a notification determination process is being carried out by a notification determination unit (18). When the vehicle information indicates that a driving assistance function has been carried out by the vehicle control device (10), the control unit (19) uses the captured image newly acquired by the image acquisition unit (13) and the result of the calibration process carried out by the calibration processing unit (16) during manual operation, to cause the occupant state detection unit (15) to perform the occupant state detection process and to cause the notification determination unit (18) to perform the notification determination process. As a result, unnecessary notifications that would occur due to appropriate calibration not being performed are suppressed.

Description

Occupant monitoring device, occupant monitoring method, occupant monitoring program, and vehicle control system

[0001] The present disclosure relates to an occupant monitoring device, an occupant monitoring method, an occupant monitoring program, and a vehicle control system.

[0002] Patent Literature 1 discloses a driving assistance device that includes a data acquisition unit that acquires a driver's biological activity data measured by a sensor that detects the state or activity of a part of the body, a reference data management unit that sets reference data for each driver's state based on the driver's biological activity data measured in the past, an estimation unit that estimates the driver's state by comparing the acquired driver's biological activity data with the reference data, a determination unit that determines the type to which the driver belongs based on driving operation characteristics, and a presentation unit that presents driving assistance information corresponding to the estimated driver's state and the type to which the driver belongs. For example, if the driver is absentminded, the driving assistance device emits an alarm to the driver. Furthermore, the reference data is repeatedly updated from the start of driving to the end of driving.

[0003] JP 2018-97485 A

[0004] In the driving assistance device of Patent Document 1, depending on the scene at which the reference data is to be updated, the update may not be performed appropriately, leading to an erroneous estimation of the driver's condition and unnecessary notifications being issued.

[0005] On the other hand, in vehicles equipped with driving assistance functions such as a lane departure prevention function, in order to prevent the driver from overconfident in the driving assistance function and becoming inattentive, attempts have been made to notify the driver when he or she is in an inattentive state and to suppress the driving assistance function when the number of notifications has accumulated. Therefore, it is preferable to activate the monitoring function at the same time as activating the driving assistance function.

[0006] However, depending on the activation timing of the driving assistance function, there is a risk that appropriate calibration will not be performed, as with the driving assistance device of Patent Document 1. As a result, the driver's condition may be erroneously detected, unnecessary notifications may be issued, and the driving assistance function may be unnecessarily suppressed.

[0007] The present disclosure has been made to solve the above-mentioned problems, and aims to suppress unnecessary notifications that occur when appropriate calibration is not performed.

[0008] The occupant monitoring device according to the present disclosure includes an image acquisition unit that acquires captured images of occupants in a vehicle, a calibration processing unit that uses the captured images acquired by the image acquisition unit to perform calibration processing to correct individual differences among occupants targeted in image processing, an occupant state detection unit that uses the captured images acquired by the image acquisition unit and the results of the calibration processing by the calibration processing unit to perform occupant state detection processing to detect an inattentive state of an occupant, a notification determination unit that uses the results of the occupant state detection processing by the occupant state detection unit to perform notification determination processing to determine whether or not an alert is required for the occupant, and a control unit that controls the execution of the processes of the image acquisition unit, the calibration processing unit, the occupant state detection unit, and the notification determination unit. and a control unit, wherein when vehicle information acquired from a vehicle control unit that controls the vehicle indicates that the vehicle is being manually driven by an occupant, the control unit acquires an image by the image acquisition unit and executes a calibration process by the calibration processing unit, regardless of whether the occupant state detection process by the occupant state detection unit and the notification determination process by the notification determination unit are being executed, and when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the control unit executes an occupant state detection process by the occupant state detection unit and a notification determination process by the notification determination unit, using the newly acquired image by the image acquisition unit and the result of the calibration process executed by the calibration processing unit during manual driving.

[0009] The occupant monitoring method according to the present disclosure includes: an image acquisition unit acquiring an image of an occupant in a vehicle; a calibration processing unit using the image acquired by the image acquisition unit to perform a calibration process for correcting individual differences among occupants targeted in image processing; an occupant state detection unit using the image acquired by the image acquisition unit and a result of the calibration process by the calibration processing unit to perform an occupant state detection process for detecting an inattentive state of the occupant; a notification determination unit using a result of the occupant state detection process by the occupant state detection unit to perform a notification determination process for determining whether or not a notification is required for the occupant; and a control unit controlling the vehicle. When the vehicle information acquired from the unit indicates that the vehicle is being manually driven by an occupant, the image acquisition unit acquires an image and the calibration processing unit executes a calibration processing, regardless of whether the occupant state detection unit executes the occupant state detection processing and the notification determination unit executes the notification determination processing, and when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the occupant state detection unit executes the occupant state detection processing and the notification determination processing are executed by the notification determination unit using the image acquisition unit's newly acquired image and the result of the calibration processing executed by the calibration processing unit during manual driving.

[0010] An occupant monitoring program according to the present disclosure includes a computer including an image acquisition unit that acquires an image of an occupant in a vehicle, a calibration processing unit that uses the image acquired by the image acquisition unit to perform a calibration process to correct individual differences among occupants who are the target of image processing, an occupant state detection unit that uses the image acquired by the image acquisition unit and a result of the calibration process by the calibration processing unit to perform an occupant state detection process to detect an inattentive state of an occupant, and a notification determination unit that uses a result of the occupant state detection process by the occupant state detection unit to perform a notification determination process to determine whether or not an alert is required for the occupant, and a program that controls the execution of the processes of the image acquisition unit, the calibration processing unit, the occupant state detection unit, and the notification determination unit. and a control unit that controls the vehicle, and when the vehicle information acquired by the control unit from the vehicle control unit that controls the vehicle indicates that the vehicle is being manually driven by an occupant, the control unit acquires an image by the image acquisition unit and executes a calibration process by the calibration processing unit regardless of whether the occupant state detection process by the occupant state detection unit and the notification determination process by the notification determination unit are being executed, and when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the control unit executes an occupant state detection process by the occupant state detection unit and a notification determination process by the notification determination unit using the newly acquired image by the image acquisition unit and the result of the calibration process executed by the calibration processing unit during manual driving.

[0011] The vehicle control system according to the present disclosure includes an occupant monitoring device according to the present disclosure, a vehicle control unit that controls the braking and driving of the vehicle to perform driving assistance functions, and an alarm unit that alerts the driver of an inattentive state.

[0012] According to the occupant monitoring device, occupant monitoring method, occupant monitoring program, and vehicle control system disclosed herein, unnecessary notifications that occur due to inappropriate calibration can be suppressed.

[0013] It is a block diagram showing a part of a vehicle equipped with an occupant monitoring device according to embodiment 1. It is a block diagram showing an example of a hardware configuration of the occupant monitoring device according to embodiment 1. It is a block diagram showing an example of a hardware configuration of the occupant monitoring device according to embodiment 1. It is a flowchart showing a processing operation of the occupant monitoring device according to embodiment 1.

[0014] Embodiment 1. Fig. 1 is a block diagram showing a portion of a vehicle 1 equipped with an occupant monitoring device 2 according to embodiment 1. Fig. 2 is a block diagram showing a vehicle control system 3 of the vehicle 1. The occupant monitoring device 2 constitutes part of the vehicle control system 3. The vehicle control system 3 comprises an occupant monitoring system 4, a driving control system 5, a braking / driving mechanism 6, and a steering mechanism 7. The occupant monitoring system 4 is made up of the occupant monitoring device 2, an imaging device 8, and an output device 9. As shown in Fig. 1, the driving control system 5 is made up of a vehicle control device 10, a surrounding condition monitoring device 11, and a vehicle state acquisition device 12, and the braking / driving mechanism 6 and steering mechanism 7 are controlled by the vehicle control device 10.

[0015] The imaging device 8 is a camera mounted on a movable part within the vehicle 1, and captures images of the interior of the vehicle 1. The imaging device 8 captures an image of an imaging area that extends from the installation position at a preset camera angle of view. The imaging device 8 is installed so that the imaging area includes the faces of the occupants in the vehicle 1, who are the image targets. The imaging device 8 captures images of the occupants in the vehicle 1 at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured images to the occupant monitoring device 2. The captured images may be still images, moving images, or both still images and moving images.

[0016] The output device 9 includes at least one of a display, a speaker, and a vibrator. The display is configured, for example, by a liquid crystal display, an organic EL (Electro-Luminescence) display, or a HUD (Head-Up Display). The display is provided, for example, on the dashboard of the vehicle 1. The speaker is provided, for example, on the dashboard of the vehicle 1. The vibrator is provided, for example, on the steering wheel of the vehicle 1 or in the driver's seat of the vehicle 1.

[0017] The occupant monitoring device 2 monitors the state of the occupant, and when the occupant is in an inattentive state, outputs a signal commanding an inattention notification to the output device 9. Here, "inattention state" refers to a state in which the occupant's attention is reduced and the occupant is distracted, or a state in which the occupant's level of alertness is reduced. As shown in Fig. 1, the occupant monitoring device 2 includes an image acquisition unit 13, a vehicle information acquisition unit 14, an occupant state detection unit 15, a calibration processing unit 16, a memory unit 17, a notification determination unit 18, a control unit 19, and a suppression determination unit 20, each of which is connected to a communication bus 21, and data can be transmitted and received via the communication bus 21.

[0018] The image acquisition unit 13 acquires the captured image from the imaging device 8 and outputs the acquired image to the occupant state detection unit 15 and the calibration processing unit 16 .

[0019] The vehicle information acquisition unit 14 acquires vehicle information related to the vehicle 1. The vehicle information includes position information of the vehicle 1, information on the positions of other vehicles or pedestrians, and traffic information output from the surrounding situation monitoring device 11 of the driving control system 5. The vehicle information also includes information on the traveling speed of the vehicle 1, information on the steering of the vehicle, information indicating that the vehicle is being manually driven by an occupant, information indicating that a driving assistance function is activated, and the like.

[0020] The occupant state detection unit 15 performs an occupant state detection process to detect an inattentive state of an occupant, such as drowsy driving or inattentive driving, using the captured image acquired by the image acquisition unit 13 and the result of the calibration process executed by the calibration processing unit 16. The image processing of the captured image in the occupant state detection unit 15 includes, for example, detection of the facial direction, the degree of eye opening, and the direction of gaze.

[0021] The calibration processing unit 16 performs calibration processing for image processing in the occupant state detection unit 15. The calibration processing is processing to correct individual differences between occupants who are the target of image processing. Generally, the detection results by the occupant state detection unit 15 involve detection errors due to individual differences between occupants. Therefore, the calibration processing unit 16 performs calibration processing to ensure the detection accuracy of the occupant state detection unit 15. Specific examples of the calibration processing are shown below. Note that the examples are not limited to the following specific examples, and other known means may also be used.

[0022] First, an example will be described in which the calibration processing unit 16 corrects the standard model used for gaze direction detection. In this case, the occupant state detection unit 15 holds a standard model for gaze direction detection that associates a captured image of the occupant's face with the occupant's gaze direction. The occupant state detection unit 15 then uses the standard model to detect the occupant's gaze direction from the captured image and determine, for example, whether or not the occupant is looking away from the road. Here, the standard model held in the occupant state detection unit 15 needs to be corrected to accommodate individual differences between occupants.

[0023] Therefore, the calibration processing unit 16 corrects the standard model based on the actual gaze direction of the occupant and the captured image in that state. For example, the calibration processing unit 16 detects the gaze direction of the occupant and estimates the vehicle equipment the occupant is gazing at using the result. Then, the calibration processing unit 16 determines the direction of the vehicle equipment as seen by the occupant as a reference for the gaze direction, and corrects the standard model based on the error between this reference gaze direction and the detected gaze direction. Alternatively, the calibration processing unit 16 may correct the standard model by correlating the gaze of the occupant when operating each device in the vehicle 1 with the position of the device being operated. This allows the occupant state detection unit 15 to accurately detect the gaze direction regardless of individual differences between occupants.

[0024] The calibration processing unit 16 may also correct the standard model used for facial direction detection. In this case, the occupant state detection unit 15 holds a 3D model that is the standard model, and uses the 3D model to detect the facial direction of the occupant from the captured image and determine, for example, whether the occupant is looking away from the road. The 3D model is a 3D model that reflects, for example, the size of a standard adult's face and the feature points of the facial features. In other words, the 3D model does not reflect the size of the occupant's face and the feature points of the facial features.

[0025] Therefore, the calibration processing unit 16 extracts feature points from the occupant in the captured image that correspond to the feature points of the 3D model. The method for extracting the feature points of the occupant from the captured image is not particularly limited, and the feature points may be extracted, for example, by a feature point extraction technique using machine learning, an image recognition technique such as edge extraction, or template matching. The calibration processing unit 16 then compares the positions of the feature points of the 3D model with the positions of the feature points of the occupant extracted from the captured image on the captured image. Next, the calibration processing unit 16 calibrates the positions of the feature points of the 3D model to match the positions of the feature points of the occupant extracted from the captured image so that the error between the positions of the feature points of the 3D model and the positions of the feature points of the occupant extracted from the captured image is within a predetermined threshold. This allows the occupant state detection unit 15 to accurately detect the face direction regardless of individual differences between occupants.

[0026] The calibration processing unit 16 may also correct the standard model used to detect the degree of eye opening. In this case, the occupant state detection unit 15 holds a standard model for detecting the degree of eye opening of the occupant. The occupant state detection unit 15 then detects the occupant's eyes from the captured image and detects the eye opening degree, which is the degree to which the occupant's eyes are open, using the standard model. The standard model for detecting the degree of eye opening includes information on a reference flattening ratio, which is, for example, a value obtained by dividing the width of the eyes when the eyes are open by the height of the eyes, and the reference flattening ratio differs depending on the individual occupant.

[0027] Therefore, the calibration processing unit 16 calculates the occupant's eye width and eye height from the captured image and calculates the occupant's reference flattening.The calibration processing unit 16 then corrects the standard model based on the error between the standard model's reference flattening and the calculated occupant's reference flattening.This allows the occupant state detection unit 15 to accurately detect the eye opening degree regardless of individual occupant differences.The calibration processing unit 16 outputs information indicating the corrected standard model, which is the result of the calibration process, to the storage unit 17.

[0028] The storage unit 17 stores the captured images acquired by the image acquisition unit 13, the vehicle information acquired by the vehicle information acquisition unit 14, etc. The storage unit 17 also stores various standard models corrected by the calibration processing unit 16. The corrected standard models stored in the storage unit 17 are stored in association with the time when the corrected standard models were generated.

[0029] The notification determination unit 18 executes a notification determination process to determine whether or not a notification to the occupant is necessary, using the detection result from the occupant state detection unit 15. When the notification determination unit 18 determines that the occupant is in an inattentive state and that a notification to the occupant is necessary, for example, based on the occupant's facial orientation, eye opening degree, and gaze direction, the notification determination unit 18 outputs a notification command to the output device 9 and also outputs information indicating that a notification to the occupant is necessary to the suppression determination unit 20. On the other hand, when the notification determination unit 18 determines that the occupant is not in an inattentive state and that a notification to the occupant is unnecessary, the notification determination unit 18 outputs information indicating that a notification to the occupant is unnecessary to the suppression determination unit 20 and also outputs information indicating that a captured image will be acquired to the image acquisition unit 13.

[0030] The control unit 19 controls the execution of each process by the image acquisition unit 13, the calibration processing unit 16, the occupant state detection unit 15, and the notification determination unit 18, based on the vehicle information acquired by the vehicle information acquisition unit 14. Specifically, when the vehicle information indicates that the vehicle is being manually driven by an occupant, the control unit 19 executes the acquisition of a captured image by the image acquisition unit 13 and the calibration processing by the calibration processing unit 16, regardless of whether the occupant state detection processing by the occupant state detection unit 15 and the notification determination processing by the notification determination unit 18 are being executed. On the other hand, when the vehicle information indicates that a driving assistance function by the vehicle control device 10 has been executed, the control unit 19 executes the occupant state detection processing by the occupant state detection unit 15 and the notification determination processing by the notification determination unit 18, using a newly acquired captured image by the image acquisition unit 13 and the result of the calibration processing executed by the calibration processing unit 16 during manual driving. Here, there may be cases where the monitoring function and the notification function for the occupant are turned off due to the occupant's intention or the like during manual driving. Even in this case, the control unit 19 forcibly activates the monitoring function at the timing when the driving assistance function is activated as described above.

[0031] When the vehicle is being manually driven, the occupant is highly attentive, and the occupant's facial orientation, posture, etc. are likely to be appropriate for the calibration process. The control unit 19 outputs information to the image acquisition unit 13 to acquire a captured image, and also outputs information to the calibration processing unit 16 to execute the calibration process. This allows the correction of the standard model in the calibration process to be executed in an appropriate scene, thereby reducing erroneous detection by the occupant state detection unit 15.

[0032] On the other hand, when the vehicle information indicates that the driving assistance function by the vehicle control device 10 has been executed, the control unit 19 outputs information to the image acquisition unit 13 indicating that a new captured image will be acquired in order to start monitoring the occupant, outputs information to the occupant state detection unit 15 indicating that an occupant state detection process will be executed, and further outputs information to the notification determination unit 18 indicating that a notification determination process will be executed. The occupant state detection unit 15 acquires a new captured image from the image acquisition unit 13, acquires from the storage unit 17 the result of the calibration process executed by the calibration processing unit 16 during manual driving, and executes the occupant state detection process. Here, the occupant state detection unit 15 acquires the latest calibration process result stored in the storage unit 17. The notification determination unit 18 executes the notification determination process using the result of the occupant state detection process output from the occupant state detection unit 15. As a result, the occupant state is detected using the latest result calibrated for an appropriate scene at the timing when the driving assistance function is started, thereby suppressing notifications due to erroneous detection.

[0033] When the notification determination unit 18 outputs information indicating that a notification to the occupant is required, in other words, when the notification determination unit 18 determines that a notification to the occupant is required, the suppression determination unit 20 counts up the number of notifications. Then, the suppression determination unit 20 executes a determination process (hereinafter referred to as the “suppression determination process”) as to whether or not to suppress the driving assistance function. That is, when the cumulative number of notifications is equal to or greater than a threshold for determining that suppression of the driving assistance function is required, the suppression determination unit 20 determines that suppression of the driving assistance function is required and outputs information indicating that suppression of the driving assistance function is required to the vehicle control device 10. On the other hand, when the cumulative number of notifications is less than the threshold, the suppression determination unit 20 determines that suppression of the driving assistance function is not required and outputs information indicating that a captured image will be acquired to the image acquisition unit 13. When the notification determination unit 18 outputs information indicating that a notification to the occupant is not required, the suppression determination unit 20 resets the cumulative number of notifications that it has been counting up and outputs information indicating that a captured image will be acquired to the image acquisition unit 13.

[0034] When a notification command is output from the notification determination unit 18, the output device 9 outputs information (hereinafter referred to as "notification information") to notify the occupant that they are not paying attention. If the output device 9 is, for example, a display, the display displays the notification information. As an example of the display, the output device 9 displays the message "Watch out ahead." The output device 9 may also display an icon urging the occupant to look ahead. Furthermore, if the output device 9 is an audio output device, the audio output device outputs audio corresponding to the notification information. The output device 9 outputs, for example, the audio "Watch out ahead."

[0035] Next, we will explain the configuration of the driving control system 5 that provides driving assistance using the detection results of the occupant monitoring system 4. As shown in Fig. 1, the driving control system 5 is configured to be able to communicate with the occupant monitoring system 4 wirelessly or via a cable. The occupant monitoring system 4, the vehicle control device 10, the surrounding condition monitoring device 11, and the vehicle state acquisition device 12 provided in the driving control system 5 are each connected to a communication bus 21, and can send and receive data via the communication bus 21.

[0036] The braking / driving mechanism 6 is a mechanism for controlling the traveling speed of the vehicle 1 and for switching between forward and reverse driving, and includes, for example, an accelerator, a brake, a shift, and a braking / driving actuator 61. The steering mechanism 7 is a mechanism for determining the traveling direction of the vehicle 1, and includes, for example, a steering column, a steering shaft, a rack, a pinion, and a steering actuator 71. The braking / driving actuator 61 that controls the braking / driving mechanism 6 is composed of, for example, an electronically controlled throttle, a brake actuator, and the steering actuator 71 that controls the steering mechanism 7 is composed of, for example, an EPS (Electric Power Steering) motor, and the like.

[0037] The surrounding situation monitoring device 11 monitors the situation around the vehicle 1 and includes a GPS (Global Positioning System) receiver, an on-board communication device, an external sensor, and a navigation system (none of which are shown), and acquires information indicating the current position of the vehicle 1, information on the positions of other vehicles or pedestrians, traffic information, etc. Here, traffic information includes, for example, congestion information, traffic regulation information, construction zone information, etc. The surrounding situation monitoring device 11 also includes, for example, at least one of a camera that captures images outside the vehicle, a millimeter-wave radar, a LiDAR, and an ultrasonic sensor, and acquires information indicating the positions of vehicles, pedestrians, obstacles, etc. present around the vehicle 1 or their distances from the vehicle 1. Furthermore, the surrounding situation monitoring device 11 calculates a route from the current position of the vehicle 1 to a destination using the navigation system and provides guidance along the calculated route.

[0038] The vehicle state acquisition device 12 acquires information indicating the state of the vehicle 1, and is equipped with, for example, a steering angle sensor, a vehicle speed sensor, an accelerator position sensor, a brake position sensor (none of which are shown), etc., to detect the steering angle, traveling speed, accelerator pedal depression amount, brake pedal depression amount, etc. The vehicle state acquisition device 12 also detects whether the driving assistance function executed by the vehicle control device 10 of the driving control system 5 is ON or OFF.

[0039] The vehicle control device 10 controls a braking / driving actuator 61 or a steering actuator 71 mounted on the vehicle 1 to assist the driver in driving. The vehicle control device 10 performs automatic driving control of the vehicle 1, for example, by outputting a signal to the braking / driving actuator 61 or the steering actuator 71. The vehicle control device 10 performs braking / driving control of the vehicle 1, such as by controlling the braking / driving actuator 61, which is composed of an electronically controlled throttle and a brake actuator, to activate the brakes and slow down or stop the vehicle 1. The vehicle control device 10 performs steering control of the vehicle 1, such as by controlling the steering actuator 71 to maintain the lane in which the vehicle 1 is traveling. Furthermore, the vehicle control device 10 has multiple driving assistance functions that assist or substitute for the driver in driving operations by controlling the driving force, braking force, steering force, etc. of the vehicle. For example, the driving assistance functions include a cruise control function and a lane departure prevention function.

[0040] 2 and 3 are diagrams showing the hardware configurations of the occupant monitoring device 2 and the vehicle control device 10. The hardware configurations of the occupant monitoring device 2 and the vehicle control device 10 will be described with reference to FIGS. 2 and 3. As an example, as shown in FIG. 2, the occupant monitoring device 2 is implemented by a processing circuit 22. The processing circuit 22 implements the functions of the functional units of the occupant monitoring device 2. The processing circuit 22 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of each functional unit of the occupant monitoring device 2 may be implemented by separate processing circuits, or these functions may be implemented together by a single processing circuit. The vehicle control device 10 can also be implemented by a similar configuration.

[0041] 3, the occupant monitoring device 2 is implemented by a processor 23 and a memory 24. The functions of the functional units of the occupant monitoring device 2 are implemented by reading and executing programs stored in the memory 24 by the processor 23. The programs are implemented as software, firmware, or a combination of software and firmware. Examples of the memory 24 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

[0042] Next, an occupant monitoring method using the occupant monitoring device 2 will be described. Fig. 4 is a flowchart showing the processing operation of the occupant monitoring device 2 according to embodiment 1. The occupant monitoring device 2 starts operation, for example, when the vehicle information acquisition unit 14 is activated, and repeats the processing operation while the vehicle 1 is traveling.

[0043] In step S1 , the vehicle information acquisition unit 14 acquires vehicle information from the vehicle state acquisition device 12 of the driving control system 5 .

[0044] In step S2, the control unit 19 determines whether to execute the notification determination process based on the vehicle information acquired by the vehicle information acquisition unit 14. If the vehicle information indicates that the vehicle is being manually driven (NO in step S2), the control unit 19 executes the image acquisition and calibration process regardless of whether the occupant state detection process and the notification determination process are being executed.

[0045] Specifically, in step S3 , the control unit 19 causes the image acquisition unit 13 to acquire a captured image, and outputs the acquired captured image to the calibration processing unit 16 .

[0046] In step S4 , the control unit 19 executes calibration processing by the calibration processing unit 16 using the captured image output from the image acquisition unit 13 , and outputs the corrected standard model, which is the calibration result, to the storage unit 17 .

[0047] In step S5, the control unit 19 stores the standard model output from the calibration processing unit 16 in the storage unit 17. When the storage in the storage unit 17 is completed, the control unit 19 returns the process to step S1.

[0048] In step S2, if the vehicle information indicates that the driving assistance function has been executed by the vehicle control device 10 (YES in step S2), the control unit 19 performs occupant status detection processing, notification determination processing, and suppression determination processing using the results of the calibration processing during manual driving and the newly acquired captured image.

[0049] Specifically, in step S6 , the control unit 19 executes acquisition of a captured image by the image acquisition unit 13 , and outputs the acquired captured image to the occupant state detection unit 15 .

[0050] In step S7, the control unit 19 performs occupant status detection processing using the captured image acquired by the image acquisition unit 13 and the latest calibration processing result stored in the memory unit 17, and outputs the detection result to the notification determination unit 18.

[0051] In step S8, the control unit 19 executes a notification determination process by the notification determination unit 18 using the detection result of the occupant state detection unit 15. If it is determined that the occupant is not in an inattentive state (NO in step S8), the control unit 19 outputs information indicating that it is not necessary to notify the occupant from the notification determination unit 18 to the suppression determination unit 20, and returns the process to step S6. When the information indicating that it is not necessary to notify the occupant is output from the notification determination unit 18, the suppression determination unit 20 resets the cumulative number of notifications.

[0052] When it is determined that the occupant is in an inattentive state (YES in step S8), the control unit 19 outputs information indicating that an occupant needs to be notified from the notification determination unit 18 to the output device 9 and the suppression determination unit 20.

[0053] In step S9, the control unit 19 causes the output device 9 to notify the occupant.

[0054] In step S10, when the notification determination unit 18 outputs information indicating that a notification to the occupant is required, the control unit 19 executes the suppression determination process by the suppression determination unit 20.

[0055] If it is determined that the cumulative number of notifications is less than the threshold value (NO in step S10), the suppression determination unit 20 determines that suppression of the driving assistance function is unnecessary, and the control unit 19 returns the process to step S6.

[0056] If it is determined that the cumulative number of notifications is greater than or equal to the threshold value (YES in step S10), the suppression judgment unit 20 determines that it is necessary to suppress the driving assistance function, and the control unit 19 outputs information from the suppression judgment unit 20 to the vehicle control device 10 of the driving control system 5 that the driving assistance function will be suppressed.

[0057] In step S11, the vehicle control device 10 suppresses the driving assistance function that is currently being executed. After that, the control unit 19 returns the process to the start (A).

[0058] As described above, in the occupant monitoring device 2 according to the first embodiment, the occupant monitoring function and the notification function may be turned off by the occupant's will or the like during manual driving. Even in such a case, by forcibly activating the monitoring function and the notification function at the timing of activating the driving assistance function, it is possible to notify the occupant of inattention and suppress the driving assistance function while the driving assistance function is being executed. Furthermore, the occupant monitoring device 2 acquires captured images and executes calibration processing regardless of whether the occupant status monitoring function and the notification function are being executed during manual driving. When the driving assistance function is activated during manual driving to start monitoring of the occupant status, monitoring is executed using the results of the calibration processing executed during manual driving. As a result, the calibration processing is executed during an appropriate scenario, i.e., during manual driving, and the occupant status is detected using the latest results calibrated in an appropriate scenario when the driving assistance function is started, thereby suppressing notifications due to erroneous detection.

[0059] Embodiment 2. An occupant monitoring device 2 according to embodiment 2 will be described. In the occupant monitoring device 2 according to embodiment 2, the control unit 19 executes a calibration process by the calibration processing unit 16 when the vehicle information indicates that the vehicle 1 is being manually driven and that the road on which the vehicle 1 is traveling is a non-sloped road, a straight road, or a non-congested road, and executes an occupant state detection process and a notification determination process using the results of the calibration process when the driving assistance function is activated.

[0060] In this way, in the occupant monitoring device 2 according to the second embodiment, when the road on which the vehicle 1 is traveling is a non-sloped road, a straight road, or a road without congestion, the calibration process is performed in a situation where the calibration process can be performed more appropriately, such as when the occupant of the vehicle 1 is looking ahead, and the calibration process result is used to monitor the occupant. Therefore, it is possible to further reduce notifications due to erroneous detection.

[0061] The above describes in detail preferred embodiments, but the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the claims.

[0062] In the first and second embodiments, when manual driving and the execution of the driving assistance function are repeated, it is preferable that the results of the latest calibration process stored in the memory unit 17 be used when the driving assistance function is started.

[0063] In embodiments 1 and 2, sensors that detect the state of the occupants are mounted on vehicle 1, while at least one of the processing using the detection results of the sensors, the processing for making judgments using the detection results, and the function for controlling vehicle 1 may be realized by a server or the like located outside vehicle 1.

[0064] REFERENCE SIGNS LIST 1 Vehicle, 2 Occupant monitoring device, 3 Vehicle control system, 4 Occupant monitoring system, 5 Driving control system, 6 Braking / driving mechanism, 7 Steering mechanism, 8 Imaging device, 9 Output device, 10 Vehicle control device, 11 Surrounding condition monitoring device, 12 Vehicle state acquisition device, 13 Image acquisition unit, 14 Vehicle information acquisition unit, 15 Occupant state detection unit, 16 Calibration processing unit, 17 Storage unit, 18 Notification determination unit, 19 Control unit, 20 Suppression determination unit, 22 Processing circuit, 23 Processor, 24 Memory

Claims

1. A vehicle vehicle information acquisition system comprising: an image acquisition unit that acquires an image of an occupant in a vehicle; a calibration processing unit that uses the image acquired by the image acquisition unit to perform a calibration process to correct individual differences among the occupants who are the target of image processing; an occupant state detection unit that uses the image acquired by the image acquisition unit and a result of the calibration processing by the calibration processing unit to perform an occupant state detection process to detect an inattentive state of the occupant; a notification determination unit that uses a result of the occupant state detection processing by the occupant state detection unit to perform a notification determination process to determine whether or not an alert is required for the occupant; and a control unit that controls the execution of each process of the image acquisition unit, the calibration processing unit, the occupant state detection unit, and the notification determination unit, wherein when vehicle information acquired from a vehicle control unit that controls the vehicle indicates that the occupant is manually driving the vehicle, the control unit executes the acquisition of the image by the image acquisition unit and the calibration processing by the calibration processing unit, regardless of whether the occupant state detection process by the occupant state detection unit and the notification determination process by the notification determination unit are being executed, an occupant monitoring device characterized in that, when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the occupant state detection process is executed by the occupant state detection unit and the notification determination process is executed by the notification determination unit using the newly acquired captured image by the image acquisition unit and the result of the calibration process executed by the calibration processing unit during the manual driving.

2. The occupant monitoring device of claim 1, characterized in that the control unit, when the vehicle information indicates that the vehicle is being manually driven and traveling on a non-sloping road, a straight road, or a non-congested road, acquires the captured image by the image acquisition unit and executes the calibration process by the calibration processing unit, regardless of whether the occupant state detection process by the occupant state detection unit and the notification determination process by the notification determination unit are being executed, and when the vehicle information indicates that the driving assistance function by the vehicle control unit has been executed, executes the occupant state detection process by the occupant state detection unit and the notification determination process by the notification determination unit using the captured image newly acquired by the image acquisition unit and the result of the calibration process by the calibration processing unit executed while the vehicle is being manually driven and traveling on a non-sloping road, a straight road, or a non-congested road.

3. An image acquisition unit acquires an image of an occupant in a vehicle; a calibration processing unit uses the image acquired by the image acquisition unit to perform a calibration process to correct individual differences among the occupants targeted in image processing; an occupant state detection unit uses the image acquired by the image acquisition unit and the result of the calibration process by the calibration processing unit to perform an occupant state detection process to detect an inattentive state of the occupant; and a notification determination unit uses the result of the occupant state detection process by the occupant state detection unit to perform a notification determination process to determine whether or not an alert is required for the occupant. an occupant monitoring method, characterized in that, when vehicle information acquired by a control unit from a vehicle control unit that controls the vehicle indicates that the occupant is manually driving the vehicle, the control unit acquires the captured image by the image acquisition unit and executes the calibration processing by the calibration processing unit, regardless of whether the occupant state detection processing by the occupant state detection unit and the notification determination processing by the notification determination unit are being executed, and when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the control unit executes the occupant state detection processing by the occupant state detection unit and the notification determination processing by the notification determination unit using the captured image newly acquired by the image acquisition unit and the result of the calibration processing executed by the calibration processing unit during the manual driving.

4. A computer is made to function as: an image acquisition unit that acquires an image of an occupant in a vehicle; a calibration processing unit that uses the image acquired by the image acquisition unit to perform calibration processing to correct individual differences among the occupants who are the target of image processing; an occupant state detection unit that uses the image acquired by the image acquisition unit and the result of the calibration processing by the calibration processing unit to perform occupant state detection processing to detect an inattentive state of the occupant; a notification determination unit that uses the result of the occupant state detection processing by the occupant state detection unit to perform notification determination processing to determine whether or not an alert is required for the occupant; and a control unit that controls the execution of each process of the image acquisition unit, the calibration processing unit, the occupant state detection unit, and the notification determination unit, wherein when vehicle information acquired from a vehicle control unit that controls the vehicle indicates that the occupant is manually driving the vehicle, the control unit executes the acquisition of the image by the image acquisition unit and the calibration processing by the calibration processing unit regardless of whether the occupant state detection processing by the occupant state detection unit and the notification determination processing by the notification determination unit are being executed, an occupant monitoring program characterized in that, when the vehicle information indicates that a driving assistance function has been executed by the vehicle control unit, the occupant status detection process is executed by the occupant status detection unit and the notification determination process is executed by the notification determination unit using the newly acquired captured image by the image acquisition unit and the result of the calibration process executed by the calibration processing unit during the manual driving.

5. A vehicle control system comprising: an occupant monitoring device according to claim 1 or 2; a vehicle control unit that controls braking and driving of the vehicle to execute the driving assistance function; and a notification unit that notifies the occupant of an inattentive state.

Images