Vehicle driver assistance systems

Abstract

The present invention relates to a driving assistance system for a vehicle, and aims to improve the vehicle. The driving assistance system for a vehicle comprises: a photographing device capable of outputting a first photographed image of an occupant, an alarm unit capable of outputting an alarm to the occupant, and a control unit. The control unit functions as the following components: a facial recognition unit that performs recognition processing of at least a portion of the occupant's face on the first photographed image; an occupant determination unit that determines the driving state of the occupant based on a second photographed image in which at least a portion of the face can be recognized by the facial recognition unit in the first photographed image, and generates a first alarm request when the occupant is not in a drivable state; and a loss determination unit that generates a second alarm request when a plurality of third photographed images in which at least a portion of the face cannot be recognized by the facial recognition unit in the images that can be output by the photographing device are generated. The alarm unit outputs an alarm to the occupant based on the first alarm request and the second alarm request.

Description

Technical Field

[0001] The invention relates to a driving assistance system for a vehicle. Background Art

[0002] In automobiles, research is being conducted on automated driving that does not rely on the driving of passengers such as drivers, and on driving that assists passengers. Automated driving is a high level of driving assistance.

[0003] On the other hand, there are also studies on occupant monitoring devices for occupants in vehicles such as automobiles. Patent Document 1 discloses a technology for monitoring occupants and outputting an alarm or the like.

[0004] It is expected that by organically combining these technologies, the convenience and safety of vehicles such as automobiles will be improved.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 11-043009 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, in an apparatus for monitoring passengers such as a driver riding in a car, a camera that captures the passengers can be used appropriately to monitor the passengers, and the state of the passengers can be determined based on the images captured by the camera.

[0010] However, when monitoring the state of an occupant by photographing the occupant with a photographing device, the determination is based on the image taken by the photographing device, and therefore, there are problems such as the following.

[0011] For example, in a vehicle, depending on the driving environment, a strong light may shine on the face of the occupant, or the interior of the vehicle may become dark. In this case, the occupant monitoring device may not correctly determine the state of the occupant's face based on the captured image. Furthermore, the vehicle's driving assistance system that performs control based on the determination result of the occupant monitoring device may also fail to perform control based on the correct determination.

[0012] Therefore, in the vehicle occupant monitoring device, it is required to be able to judge the state of the occupant's face as much as possible in the above-mentioned captured image. In order to make such image judgment possible, for example, the judgment criterion for the state of the occupant's face is adjusted in a manner that the face of the occupant that is not clearly mapped in the captured image can be judged as the face of the occupant. However, it is predicted that the judgment criterion for the state of the occupant's face in the case of such adjustment becomes looser than the original judgment criterion. As a result, for example, although the occupant is in a correct driving state to drive, for example, a car, the possibility of being mistakenly judged as an inappropriate driving state due to the quality of the captured image and the loose judgment criterion is increased. In addition, for example, although the occupant is not in a correct driving state to drive, for example, a car, the possibility of being mistakenly judged as an appropriate driving state due to the quality of the captured image and the loose judgment criterion is increased.

[0013] Thus, the vehicle requires improvement.

[0014] Solutions to Solve Problems

[0015] One embodiment of the present invention provides a driving assistance system for a vehicle, comprising: a camera that can output a first captured image of an occupant of the vehicle; an alarm unit that outputs an alarm to the occupant; and a control unit that can execute control using the camera and the alarm unit, wherein the control unit functions as: a face recognition unit that performs recognition processing of at least a portion of the face of the occupant on the first captured image acquired from the camera; an occupant determination unit that determines the driving state of the occupant based on a second captured image of the first captured image that can be output by the camera, in which at least a portion of the face can be recognized by the face recognition unit, and generates a first alarm request when the occupant is not in a state where he or she can drive; and a loss determination unit that generates a second alarm request when a plurality of third captured images of the first captured image that can be output by the camera, in which at least a portion of the face cannot be recognized by the face recognition unit, generates a second alarm request, wherein the alarm unit outputs an alarm to the occupant based on the first alarm request of the occupant determination unit and the second alarm request of the loss determination unit.

[0016] Preferably, the occupant determination unit may determine whether the occupant is looking sideways or dozing off based on a state of eyes of the occupant's face in the first captured image.

[0017] Preferably, the loss judgment unit can set at least the first captured image in which the occupant judgment unit cannot judge the driving status of the occupant as the third captured image in which the face cannot be recognized, and judge whether multiple third captured images in which the face cannot be recognized are generated.

[0018] Preferably, the loss judgment unit can be set to generate the third captured image in which the face cannot be recognized by the facial recognition unit when the first captured image cannot be obtained from the shooting device, and judge whether multiple third captured images in which the face cannot be recognized are generated.

[0019] Preferably, the loss determination unit may generate the second alarm request when the face recognition unit fails to recognize the face in the first captured image that can be output by the capturing device.

[0020] Preferably, the loss judgment unit may include: a first loss judgment unit which generates a third alarm request as the second alarm request when the state in which the face cannot be recognized by the facial recognition unit continues to occur continuously; and a second loss judgment unit which generates a fourth alarm request as the second alarm request when the state in which the face cannot be recognized by the facial recognition unit continues to occur in an intermittently repeated manner, wherein the alarm unit outputs an alarm to the occupant based on the third alarm request of the first loss judgment unit as the second alarm request and the fourth alarm request of the second loss judgment unit.

[0021] Preferably, the second loss determination unit may determine that the repetition occurs intermittently over a period longer than a period during which the first loss determination unit determines that the repetition occurs continuously.

[0022] Preferably, there may be a detection unit that detects whether an operating component operated by an occupant when driving the vehicle is in a state in which the occupant can operate it. When the alarm unit generates the first alarm request for the occupant judgment unit or the second alarm request for the loss judgment unit, the alarm unit determines whether the occupant is in a state in which the operating component can be operated based on the detection result of the detection unit. When it cannot be determined that the occupant is in a state in which the operating component can be operated, an alarm requesting the occupant to operate the operating component is output to the occupant.

[0023] Preferably, there may be a driving control device that controls the driving of the vehicle, and the driving control device decelerates or stops the vehicle after the alarm unit outputs an alarm if the detection result of the detection unit has not changed to indicate that the occupant is in a state where the operating component can be operated.

[0024] Effects of the Invention

[0025] In the present invention, based on the first captured image of the camera device that captures the occupant of the vehicle, the facial recognition unit first performs recognition processing of at least a portion of the occupant's face. Then, the occupant judgment unit judges the driving state of the occupant for the second captured image in the first captured image that can be output by the camera device, in which at least a portion of the face can be recognized by the facial recognition unit. As a result, the occupant judgment unit does not need to judge the driving state of the occupant for the first captured image of poor quality that cannot recognize the face, for example. The occupant judgment unit only needs to judge the driving state of the occupant based on the second captured image of good quality, which is taken in the first captured image that can be output by the camera device so that the occupant's face can be correctly recognized. The judgment of the driving state of the occupant by the occupant judgment unit can be optimized for its judgment. It can be expected that the judgment accuracy of the occupant judgment unit will be increased by the optimized judgment. Moreover, the alarm unit can output an alarm to the occupant based on the first alarm request of the occupant judgment unit.

[0026] Moreover, in the present invention, a loss judgment unit is provided in parallel with the passenger judgment unit. Moreover, in the case where a plurality of third captured images in which at least a part of the face cannot be recognized by the face recognition unit are generated in the first captured images output from the camera, the loss judgment unit generates a second alarm request in the same manner as the passenger judgment unit. Thus, for the third captured images in the first captured images output from the camera that are not involved in the judgment of the passenger judgment unit, the loss judgment unit can generate a second alarm request. In the case where there is a possibility that the passenger is in an inappropriate driving state such as squinting or drowsiness in the third captured images that are not involved in the judgment of the passenger judgment unit, the loss judgment unit does not directly judge the state of the passenger based on the captured images, but can generate a second alarm request based on the inherent possibility. By the parallel cooperative processing of the passenger judgment unit and the loss judgment unit, in the case where there is a possibility that the passenger is in an inappropriate driving state such as squinting or drowsiness, an alarm about the situation can be output roughly and reliably. In the present invention, regardless of the imaging environment and state of the imaging device, when there is a possibility that the occupant is in an inappropriate driving state such as squinting or falling asleep, a warning about this situation can be outputted almost reliably.

[0027] In this way, the present invention can achieve both correct judgment of the inappropriate driving state of the occupant as processing mutually contradictory requests, and outputting a warning about the situation widely when there is a possibility that the occupant is in an inappropriate driving state. The present invention can correctly judge the driving state of the occupant as a vehicle occupant monitoring, and can output a warning widely including the situation with the possibility of an inappropriate driving state to give priority to safety, and can be appropriately used in a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic explanatory diagram of a vehicle according to an embodiment of the present invention.

[0029] Figure 2 yes Figure 1 A schematic illustration of a control system of an automobile.

[0030] Figure 3 yes Figure 2 An illustration of a driving assistance system for an automobile implemented in a control system.

[0031] Figure 4 yes Figure 3 Flowchart of the control process of the occupant monitoring device.

[0032] Figure 5 yes Figure 3 Flow chart of the control process of the alarm device.

[0033] Figure 6 yes Figure 3 Flowchart of the control process of the travel control device.

[0034] Figure 7 This is an explanatory diagram of an example of a process for generating an alarm request when captured images in which a face cannot be recognized are repeatedly generated.

[0035] Explanation of symbols

[0036] 1… Automobile (vehicle), 14… Steering ECU (automatic driving / driving assistance ECU), 17… Occupant monitoring ECU, 18… Alarm ECU, 31… Steering wheel, 44… Hand sensor, 61… Interior camera, 63… Timer, 64… Memory, 80… Driving assistance system, 81… Occupant monitoring device, 82… Face recognition unit, 83… Squint judgment unit (Occupant judgment unit), 84… Drowsiness judgment unit (Occupant judgment unit), 85… Continuous loss judgment unit (first loss judgment unit), 86… Frequent loss judgment unit (second loss judgment unit), 87… Alarm device, 88… Alarm unit, 89… Driving control device, 90… Driving control unit DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0038] Figure 1 It is a schematic explanatory diagram of a car 1 according to an embodiment of the present invention.

[0039] Figure 1The automobile 1 is an example of a vehicle. The automobile 1 has a vehicle cabin 3 provided with seats 4 for passengers to sit in the center of the vehicle body 2. Passengers such as a driver riding in the automobile 1 sit in the seats 4 and operate the driving of the automobile 1. Thus, the automobile 1 can be driven by automatic driving or manual driving. The automobile 1 can also perform driving assistance when driving based on the driving operation performed by the passengers. This automobile 1 assists both driving related to automatic driving and driving manually.

[0040] Figure 2 yes Figure 1 Schematic illustration of a control system 10 of a vehicle 1 . Figure 2 1 shows a control system 10 of the automobile 1 and a traffic information system 100 that communicates with the control system 10 .

[0041] In the automobile 1, it is required to assist the automatic driving that does not depend on the driving of the passengers such as the driver and the driving by the passengers by using the driving control device. In addition, in the automobile 1, it is also required to identify and monitor the passengers in the vehicle by using the passenger monitoring device 81. It is considered that by organically combining these technologies, for example, the driver can be identified and a high level of driving assistance corresponding to the driver can be realized. It is expected that the convenience and safety of the automobile 1 will be improved.

[0042] Figure 2 In the figure, a plurality of control devices constituting a control system 10 of the automobile 1 are represented by respective built-in control ECUs (Electronic Control Units).

[0043] Figure 2Specifically, in the control system 10 of the automobile 1, there are represented: a driving ECU 11, a steering ECU 12, a brake ECU 13, a control ECU (automatic driving / driving assistance ECU) 14 as a device for controlling the driving of the automobile 1 through automatic driving or driving assistance during manual driving, a driving operation ECU 15, a detection ECU 16, an occupant monitoring ECU 17 of an occupant monitoring device, an alarm ECU 18 of an alarm device, an external communication ECU 19, and a UIECU 20. These multiple control ECUs are directly connected to a cable 26, and are connected to a central gateway (CGW) 27 as a relay device through a vehicle network 25 such as CAN (Controller Area Network) and LIN (Local Interconnect Network) used in the automobile 1. Each control ECU can communicate with each other through encrypted data to which its own ID as a sender (sender) and the ID of the device of the receiver (sender) are attached. Each control ECU can also send encrypted data without specifying the receiver to multiple other control ECUs at the same time as broadcasting. In addition, each control ECU receives encrypted data with its own ID as the receiver and uses it for its own control. In addition, the control ECU can also receive encrypted data with a specific ID other than itself as the recipient and use it for its own control. When the encrypted data received from each of the multiple cables 26 reaches the control ECU connected to other cables 26, the central gateway 27 outputs the received encrypted data to the other cables 26. Figure 2 The multiple control ECUs shown are capable of sending and receiving encrypted data to each other.

[0044] The drive ECU 11 receives the encrypted data through the vehicle network 25, and controls a driving source such as an engine or a motor (not shown) and a transmission provided in the automobile 1. As a result, the automobile 1 can accelerate and travel.

[0045] The steering ECU 12 receives the encrypted data through the vehicle network 25, and controls a steering device (not shown) provided in the automobile 1. Thus, the automobile 1 can change the traveling direction.

[0046] The brake ECU 13 receives the encrypted data through the vehicle network 25 and controls the brake device (not shown) provided on the automobile 1. Thus, the traveling automobile 1 can be decelerated and stopped. In addition, the brake ECU 13 can also communicate with the drive ECU 11 through the vehicle network 25 to suppress the rotation of the drive source or change the gear ratio of the transmission to decelerate the traveling automobile 1.

[0047] The driving operation ECU 15 is connected to operating components such as a steering wheel 31, a brake pedal 32, an accelerator pedal 33, and a shift lever 34 for operating the vehicle 1. The driving operation ECU 15 outputs encrypted data including operation information on the operated operating components to the steering ECU 14 and the like through the vehicle network 25.

[0048] The detection ECU 16 is connected to, for example, a speed sensor 41, an acceleration sensor 42, an external camera 43, and a hand sensor 44. The speed sensor 41 detects the speed of the car 1. The acceleration sensor 42 detects the acceleration of the car 1. The external camera 43 captures the surroundings of the outside of the car 1. The external camera 43 can also capture, for example, the front, side, and rear of the car 1. The hand sensor 44 is provided on, for example, the steering wheel 31, and detects changes in electrostatic capacitance and pressure caused by the hand of the occupant operating the steering wheel 31. The detection ECU 16 acquires the physical quantities of the connected sensors as physical quantities detected in the car 1 regarding the driving of the car 1, and outputs them to the operation ECU 14, etc., through the vehicle network 25. In addition, the detection ECU 16 can also analyze the external image captured by the external camera 43, detect other moving bodies around the car 1, and output the detected information about other moving bodies to the operation ECU 14, etc., through the vehicle network 25.

[0049] The external communication ECU 19 communicates with the outside of the automobile 1. The external communication ECU 19 communicates with, for example, a communication relay station 101 of the traffic information system 100 and sends and receives communication data with the server device 102. In addition, the external communication ECU 19 sends and receives communication data with, for example, another automobile 110 traveling near the automobile 1 or a pedestrian's mobile terminal (not shown) through, for example, V2X communication. The external communication ECU 19 can also receive navigation data to the destination, traffic information, information on the surrounding conditions of the automobile 1, etc. from the automobile 1 through these communications, and output them to the control ECU 14, etc. through the vehicle network 25.

[0050] The UIECU 20 is connected to, for example, a display device 51 and an operating device 52. The operating device 52 may also include a touch panel or a plurality of keys that overlap with the display screen of the display device 51. When the UIECU 20 receives encrypted data related to display, it causes the display device 51 to display an image. The UIECU 20 may also generate, for example, navigation data based on the operation of the operating device 52 corresponding to the display of the display device 51, and output it to the operation ECU 14 through the vehicle network 25.

[0051] The operation ECU 14 switches between automatic driving and driving assistance and controls the driving of the automobile 1. The operation ECU 14 controls the driving of the automobile 1 based on various information obtained through the vehicle network 25. For example, in the case of automatic driving, the operation ECU 14 outputs control information to the drive ECU 11, the steering ECU 12, and the brake ECU 13 in a manner that the vehicle drives to the destination while confirming the safety of the surroundings based on the navigation data. In the case of driving assistance, the operation ECU 14 outputs control information whose operation amount is adjusted according to the safety of the surroundings, etc. to the drive ECU 11, the steering ECU 12, and the brake ECU 13 based on the operation information.

[0052] The occupant monitoring ECU 17 is connected to an interior camera 61 , a microphone 62 , a timer 63 , and a memory 64 .

[0053] The interior camera 61 is installed in the vehicle cabin 3 to capture the interior of the vehicle 1. The interior camera 61 is installed on the inner surface of the vehicle cabin 3, for example Figure 1 As shown in FIG. 1 , the interior camera 61 is located on the dashboard in front of the occupants. As a result, the driver and other passengers in the car 1 are mapped in the image captured by the interior camera 61. When capturing the entire interior of the car 1, the interior camera 61 may also be a wide-angle camera or a 360-degree camera. The interior camera 61 repeatedly captures the passengers in the cabin 3 at a specified period.

[0054] The microphone 62 records sounds inside the car 1 .

[0055] The timer 63 measures time.

[0056] The memory 64 records programs, setting data, and the like for occupant monitoring.

[0057] The occupant monitoring ECU 17 reads and executes the program from the memory 64 . Thus, the occupant monitoring ECU 17 functions as a control unit of the occupant monitoring device 81 .

[0058] The occupant monitoring ECU 17 as a control unit of the occupant monitoring device 81 monitors occupants including the driver riding in the automobile 1 based on images captured by the interior camera 61 and sounds recorded by the microphone 62 .

[0059] The occupant monitoring ECU 17 recognizes an occupant riding in the vehicle 1 based on an image captured by, for example, the interior camera 61 .

[0060] The occupant monitoring ECU 17 may output setting information of the automobile 1 related to the recognized occupant to the operation ECU 14 or the like through the vehicle network 25 .

[0061] The occupant monitoring ECU 17 monitors the status of the identified occupant or the driver of the vehicle, etc., regarding the person responsible for operation, generates an alarm request as needed, and outputs it through the vehicle network 25. The occupant monitoring ECU 17 may also determine whether the occupant is drowsy based on, for example, the degree of eye opening of the occupant, or whether the occupant is squinting based on, for example, the direction of the occupant's face and eyes, and generates and outputs an alarm request when it is determined that the occupant is in a drowsy or squinting state.

[0062] The alarm ECU 18 is connected to a meter panel 71 and a speaker 72 .

[0063] The instrument panel 71 is provided on the inner surface of the vehicle interior 3, for example, on a dashboard located in front of the occupant. The instrument panel 71 includes a liquid crystal panel for displaying warnings and a light emitting element.

[0064] The speaker 72 is provided on the inner surface of the vehicle interior 3 , for example, at a door, an instrument panel, a pillar, a roof, or the like.

[0065] When the alarm ECU 18 outputs a warning request to the vehicle network 25, for example, the warning request is received and a warning sound, a warning display, and a warning message corresponding to the warning request are output from the instrument panel 71 and the speaker 72. Thus, the occupants in the vehicle cabin 3 can recognize the warning issued by the vehicle 1 and respond to the warning.

[0066] However, in the occupant monitoring device 81 for monitoring the driver and other passengers in the automobile 1, the interior camera 61 for photographing the passengers is often used to monitor the passengers' riding status. In this case, the occupant monitoring ECU 17 of the occupant monitoring device 81 determines the status of the passengers based on the images taken by the interior camera 61.

[0067] However, when judging the state of the occupant based on the image of the occupant captured by the interior camera 61 , the judgment is based on the image captured by the interior camera 61 , and therefore, there are problems such as the following.

[0068] For example, in the automobile 1, depending on the driving environment, strong light may be irradiated on the face of the passenger, or the interior of the vehicle may become dark. In this case, the passenger monitoring ECU 17 may not be able to correctly judge the state of the passenger based on the captured image. In particular, when judging the direction and degree of opening of the eyes, which are relatively small in size in the captured image, it is likely that it will be difficult to judge them in the captured image when strong light is irradiated on the face of the passenger or the interior of the vehicle becomes dark. In addition, the automobile 1 that executes warnings or controls based on the judgment results of the passenger monitoring ECU 17 may not be able to execute appropriate warnings or controls based on correct judgments.

[0069] Therefore, the occupant monitoring ECU 17 is required to judge the state of the occupant as much as possible even for the inappropriate captured images as described above. In order to make such image judgment possible, it is considered to judge the face and eyes of the occupant that are not clearly mapped in the captured image as the face and eyes of the occupant, and judge the state of the occupant. The judgment criterion of the occupant state performed by the occupant monitoring ECU 17 is adjusted more loosely than the case where the face and eyes of the occupant that are clearly mapped in the captured image are judged. Therefore, in the judgment after such adjustment, as a result of the adjustment, the anxiety about the possibility of erroneously judging that the occupant is in an inappropriate driving state even though the occupant is in a correct driving state for driving the car 1, for example, is aggravated. In addition, the anxiety about the possibility of erroneously judging that the occupant is in an appropriate driving state even though the occupant is not in a correct driving state for driving the car 1, for example, is aggravated.

[0070] In this way, the occupant monitoring ECU17 of the occupant monitoring device 81 is required to appropriately judge the occupant's riding status based on the image captured by the internal camera 61, so that the car 1 can appropriately perform warnings or controls even when the quality of the image captured is poor.

[0071] Furthermore, the driving assistance system 80 of the automobile 1 is required not to output too many warnings so as not to frequently or excessively request the occupant to perform a specific action or operation such as holding the hand due to, for example, an erroneous judgment or sensitive judgment in automatic driving.

[0072] As described above, further improvement is required in the driving assistance system 80 of the automobile 1 .

[0073] Figure 3 yes Figure 2 1 is an explanatory diagram of a driving assistance system 80 of an automobile 1 implemented in a control system 10.

[0074] Figure 3 The driving assistance system 80 includes an occupant monitoring device 81 , an alarm device 87 , a travel control device 89 , and a hand-grip sensor 44 . The occupant monitoring device 81 , the alarm device 87 , and the travel control device 89 are connected via the vehicle network 25 .

[0075] The hand sensor 44 detects changes in electrostatic capacitance or pressure caused by the hand of the occupant operating the steering wheel 31, thereby being able to detect whether the occupant has placed his hand on the occupant-operated steering wheel 31 to manually drive the car 1 and is in an operable state.

[0076] The occupant monitoring device 81 includes, as a function of the occupant monitoring ECU 17 executing a program, a face recognition unit 82, a side-gaze determination unit 83, a dozing determination unit 84, a continuous loss determination unit 85, and a frequent loss determination unit 86. Figure 3Also shown is an internal camera 61 and a timer 63.

[0077] The interior camera 61 repeatedly captures images of passengers in the vehicle interior 3 at predetermined intervals and outputs captured images.

[0078] The face recognition unit 82 performs recognition processing of the face of the occupant or the eyes, nose, mouth, etc., which are part of the face, on the captured image of the interior camera 61. The face recognition unit 82 may confirm whether the captured image can be used in the judgment of the squint judgment unit 83 and the sleepiness judgment unit 84 in the following paragraph by simple processing, but it is preferable to recognize the face, eyes, nose, and mouth of the occupant in the captured image by a stricter judgment standard than that of the squint judgment unit 83 and the sleepiness judgment unit 84 in the following paragraph. Thus, the face recognition unit 82 can set the captured image as a captured image that cannot judge the driving state of the occupant when at least the white eyeball and the black eyeball are not captured in the image part of the eyes involved in the judgment of the squint judgment unit 83 and the sleepiness judgment unit 84 in the following paragraph. Or, it can set the captured image to have the possibility of confusing the squint judgment unit 83 and the sleepiness judgment unit 84 in the following paragraph in their judgment. Moreover, when the face recognition unit 82 can appropriately recognize the face, eyes, nose, and mouth of the occupant in the captured image, it outputs a permission signal indicating that the occupant can be recognized. In the event that recognition is not possible, a disapproval signal may be output.

[0079] When a permission signal is input, the squint judgment unit 83 judges whether the passenger is squinting based on the captured image of the internal camera 61. The squint judgment unit 83 extracts, for example, the portion of the captured passenger's face and eyes in the captured image, and judges the direction of the passenger's face and eyes based on the extracted partial image. At this time, the squint judgment unit 83 may also use the recognition result of the face recognition unit 82 in the previous paragraph to extract or judge, or may extract and judge from one by independent processing. Moreover, when the direction of the passenger's face and eyes is not facing, for example, the front, which is the direction of travel of the automobile 1, a warning request about squint is output. In this way, the squint judgment unit 83 judges squint based on the state of the face or the state of the eyes in the captured image.

[0080] When a permission signal is input, the drowsiness determination unit 84 determines whether the occupant is drowsy based on the image captured by the internal camera 61. The drowsiness determination unit 84 extracts, for example, a portion of the captured occupant's eyes from the captured image, and determines the degree of opening of the occupant's eyes based on the extracted partial image. At this time, the drowsiness determination unit 84 may also use the recognition result of the face recognition unit 82 in the previous paragraph to extract or determine, or may extract and determine from one by independent processing. Moreover, when the degree of opening of the occupant's eyes is below a specified threshold, a warning request regarding drowsiness is output. In this way, the drowsiness determination unit 84 determines drowsiness based on the state of the eyes in the captured image.

[0081] In this way, the side-looking judgment unit 83 and the drowsiness judgment unit 84 judge whether the occupant is in an inappropriate driving state such as side-looking or drowsiness based on the captured image that can recognize the occupant's face, etc., and generate and output an alarm request in the case of an inappropriate driving state. Here, the inappropriate driving state refers to a state in which the occupant is not in the driving posture that is taken when the occupant operates the vehicle 1, for example, or is not in a state defined as being immediately operable.

[0082] The continuous loss judgment unit 85 counts the number of images captured by the internal camera 61 in a state where, for example, a permission signal is not input. The count value may be recorded in the memory 64. The continuous loss judgment unit 85 may also reset the count value when a captured image of the internal camera 61 is generated in a state where a permission signal is input. Thus, the continuous loss judgment unit 85 can count the number of multiple consecutive captured images in a state where a permission signal is not input. In addition, the state where a permission signal is not input includes a situation where the internal camera 61 does not output a captured image at the timing of outputting the captured image. This is because, in this case, the face recognition unit 82 cannot output a permission signal either. Moreover, when the count value representing the continuous number becomes a threshold number of times corresponding to a specified continuous loss judgment period, since the captured images become continuously inappropriate, the continuous loss judgment unit 85 outputs a warning request about the inability to correctly judge strabismus and drowsiness. The continuous loss judgment period may also be measured using the timer 63. In this way, the continuous loss judgment unit 85 generates and outputs an alarm request when the state in which the face recognition unit 82 cannot continue to recognize the face for a certain number of captured images to be output from the internal camera 61 continues to occur repeatedly for more than a certain number of times or for a certain period of time.

[0083] The frequent loss judgment unit 86 counts the number of images captured by the internal camera 61 in a state where, for example, the permission signal is not input. The count value may be recorded in the memory 64. Even if the frequent loss judgment unit 86 generates images captured by the internal camera 61 in a state where the permission signal is input, the count value is not reset, but the count value continues to be counted from the original value. Thus, even if the state where the permission signal is not input is generated intermittently, the frequent loss judgment unit 86 can count the number of multiple captured images in the state where the permission signal is not input through the entire period. In addition, the state where the permission signal is not input includes the case where the captured image is not output at the time when the internal camera 61 outputs the captured image. Moreover, when the count value of the most recent frequent loss judgment period becomes more than the threshold number of times, because the captured image becomes intermittently inappropriate, the frequent loss judgment unit 86 outputs a warning request about the inability to correctly judge strabismus and drowsiness. The most recent frequent loss judgment period may also be continued using the timer 63. In this way, the frequent loss judgment unit 86 generates and outputs an alarm request when the state in which the face recognition unit 82 cannot continue to recognize the face for multiple captured images that should be output from the internal camera 61 occurs intermittently and repeatedly for more than a certain number of recent times or a certain cumulative period of recent times.

[0084] In this way, the continuous loss determination unit 85 and the frequent loss determination unit 86 generate and output an alarm request when the face recognition unit 82 cannot repeatedly recognize the face in a plurality of captured images continuously output from the internal camera 61 .

[0085] The alarm device 87 includes an alarm unit 88 as a function of the alarm ECU 18 executing a program.

[0086] When an alarm request is output from the occupant monitoring device 81, the alarm unit 88 determines whether the occupant is in a state where the operating component can be operated based on the detection of the hand-held sensor 44. Furthermore, in the case where it cannot be determined that the occupant is in a state where the operating component can be operated, the alarm unit 88 selects an alarm corresponding to the alarm request and outputs the alarm. For example, based on the alarm request, the alarm unit 88 outputs an alarm to the occupant for correcting an inappropriate driving state such as squinting or drowsiness. The alarm is output to the occupant from the speaker 72 and the instrument panel 71. The alarm here may also be, for example, a hand-held alarm for correcting an inappropriate driving state such as squinting or drowsiness. In addition, in the case where it can be determined that the occupant is in a state where the operating component can be operated, the alarm unit 88 does not output an alarm. In this way, unnecessary alarm output can be reduced.

[0087] The travel control device 89 includes a travel control unit 90 as a function of operating the ECU 14 to execute a program.

[0088] The travel control unit 90 controls the travel of the automobile 1 during automatic driving and manual driving of the automobile 1. For example, during automatic driving of the automobile 1, the travel control unit 90 controls the travel of the automobile 1 regardless of the operation of the occupant.

[0089] After the alarm device 87 outputs an alarm or the occupant monitoring device 81 outputs an alarm request, the travel control unit 90 determines whether the occupant is in a state where the operating member can be operated based on the detection of the hand-holding sensor 44. If it cannot be determined that the occupant is in a state where the operating member can be operated, the travel control unit 90 performs a predetermined travel control of the automobile 1 according to the contents of the alarm. For example, the travel control unit 90 decelerates the traveling automobile 1 and stops the automobile 1 at a safe roadside or the like.

[0090] Figure 4 yes Figure 3 Flowchart of the control process of the occupant monitoring device 81.

[0091] The passenger monitoring ECU 17 repeatedly executes Figure 4 processing.

[0092] In step ST1, the occupant monitoring ECU 17 determines whether it is the captured image acquisition timing for the interior camera 61 to output the captured image. The occupant monitoring ECU 17 may also determine whether it is the captured image acquisition timing based on the time measured by the timer 63 or the elapsed time from the last acquisition timing. If it is not the captured image acquisition timing, the occupant monitoring ECU 17 repeats this process. If it is the captured image acquisition timing, the occupant monitoring ECU 17 advances the process to step ST2.

[0093] In step ST2 , the occupant monitoring ECU 17 acquires a new captured image output by the interior camera 61 .

[0094] In step ST3, the passenger monitoring ECU 17, as the face recognition unit 82, performs recognition processing on the passenger's face or the eyes, nose, and mouth that are part of the face on the acquired new captured image. The passenger monitoring ECU 17 performs recognition processing on the white and black of the eyes, especially for the eyes. In the captured image, the passenger's face or the eyes, nose, and mouth that are part of the face may not be captured well due to, for example, the shooting environment. In addition, the passenger monitoring ECU 17 may not be able to obtain the captured image at the time of capturing the captured image due to, for example, a malfunction of the internal camera 61. The passenger monitoring ECU 17 performs recognition processing on the passenger including these conditions. When the passenger monitoring ECU 17 can recognize the passenger's face or the eyes, nose, and mouth that are part of the face in the captured image, it outputs a permission signal indicating that the passenger can be recognized to the squint judgment unit 83, the drowsiness judgment unit 84, the continuous loss judgment unit 85, and the frequent loss judgment unit 86 involved in the processing of the following later paragraphs. In other cases, the passenger monitoring ECU 17 does not output a permission signal. Alternatively, the passenger monitoring ECU 17 may output a non-permission signal. These permission signals may be output by, for example, recording in the memory 64 or directly transmitting through inter-program communication.

[0095] In step ST4, the occupant monitoring ECU 17 as the face recognition unit 82 determines whether the occupant's face or the eyes, nose, and mouth as part of the face can be recognized from the captured image. If the occupant's face or the eyes, nose, and mouth as part of the face can be recognized from the captured image, the occupant monitoring ECU 17 advances the processing to step ST5.

[0096] If the face of the occupant or the like cannot be recognized, the occupant monitoring ECU 17 advances the process to step ST7 .

[0097] In step ST5, the occupant monitoring ECU 17, as the side-looking judgment unit 83 to which the permission signal is input, judges whether the occupant is side-looking based on the image captured by the interior camera 61. If it is judged that the occupant is side-looking, the occupant monitoring ECU 17 advances the processing to step ST10. If it is not judged that the occupant is side-looking, the occupant monitoring ECU 17 advances the processing to step ST6.

[0098] In step ST6, the occupant monitoring ECU 17, as the drowsiness determination unit 84 to which the permission signal is input, determines whether the occupant is drowsy based on the image captured by the interior camera 61. If it is determined that the occupant is drowsy, the occupant monitoring ECU 17 advances the processing to step ST10. If it is not determined that the occupant is drowsy, the occupant monitoring ECU 17 ends the process. Figure 4 processing.

[0099] Note that the occupant monitoring ECU 17 may perform the dozing determination in step ST6 before the side-looking determination in step ST5 .

[0100] In step ST7, the occupant monitoring ECU 17 records the loss of the image that cannot be recognized by the face recognition unit 82 in the memory 64. The occupant monitoring ECU 17 may count, for example, the number of consecutive captured images that are loss images that cannot be recognized by the face, and record the count value in the memory 64. In addition, for example, the occupant monitoring ECU 17 may cumulatively record the acquisition execution time of the captured images that are loss images that cannot be recognized by the face recognition unit 82 in the memory 64.

[0101] In step ST8, the occupant monitoring ECU 17, as the continuous loss determination unit 85 to which the permission signal is not input, determines whether loss images that cannot be recognized by the face recognition unit 82 are continuously generated for a certain number or more corresponding to a preset continuous loss determination period.

[0102] The occupant monitoring ECU 17 compares the number of images of the loss images that have been continuously generated recently with the judgment threshold value corresponding to the continuous loss judgment period. If the number of images of the loss images that have been continuously generated recently is greater than the judgment threshold value, the occupant monitoring ECU 17 advances the processing to step ST10. In other cases, the occupant monitoring ECU 17 advances the processing to step ST9.

[0103] In step ST9, the occupant monitoring ECU 17, as the frequent loss judgment unit 86 to which the permission signal is not input, judges whether a loss image that cannot be recognized by the face recognition unit 82 is intermittently generated at a frequency above a certain frequency during the most recent frequent loss judgment period that is preset. The occupant monitoring ECU 17 compares the count value of the loss image in the most recent frequent loss judgment period with the threshold value of the number of loss images generated that is preset for the frequent loss judgment period. If the count value of the loss image in the most recent frequent loss judgment period is above the threshold value of the number of generation, the occupant monitoring ECU 17 causes the processing to enter step ST10. In other cases, the occupant monitoring ECU 17 ends. Figure 4 processing.

[0104] In step ST10, the occupant monitoring ECU 17 outputs a hand-held alarm request to the alarm ECU 18 and the operation ECU 14 via the vehicle network 25. The occupant monitoring ECU 17 may output the alarm request in a manner that allows the cause of the alarm request to be distinguished.

[0105] Figure 5 yes Figure 3 Flow chart of the control process of the alarm device 87.

[0106] The alarm ECU 18 serves as the alarm unit 88 of the alarm device 87 and repeatedly executes Figure 5 processing.

[0107] In step ST21, the alarm ECU 18 determines whether a handrail alarm request is received from the occupant monitoring device 81. If an alarm request is not received, the alarm ECU 18 repeats this process. If an alarm request is received, the alarm ECU 18 advances the process to step ST22.

[0108] In step ST22, the alarm ECU 18 acquires the detection value of the hand-holding sensor 44. When the steering wheel 31 is held by the passenger's hand, the detection ECU 16 outputs "hand-holding" as the detection value of the hand-holding sensor 44. When the steering wheel 31 is not held by the passenger's hand, the detection ECU 16 outputs "hands-off" as the detection value of the hand-holding sensor 44. The alarm ECU 18 acquires the detection value of any of these sensors.

[0109] In step ST23, the alarm ECU 18 determines whether the occupant is in the hand-holding state where manual operation can be started immediately based on the sensor detection value obtained in step ST22. If the occupant is in the hand-holding state, the alarm ECU 18 ends. Figure 5 processing.

[0110] The alarm ECU 18 does not output the hand-held alarm based on the alarm request, but ends the Figure 5 If the occupant is not in the hand-holding state, the alarm ECU 18 advances the process to step ST24.

[0111] In step ST24 , the alarm ECU 18 outputs a hand-held alarm. The alarm ECU 18 may output a hand-held alarm to the operation ECU 14 via the vehicle network 25 .

[0112] Figure 6 yes Figure 3 Flow chart of the control process of the travel control device 89.

[0113] The ECU 14 is operated as the travel control unit 90 of the travel control device 89 to repeatedly execute Figure 6 processing.

[0114] In step ST31, the operation ECU 14 determines whether the automobile 1 has started running. If the automobile 1 has not started running, the operation ECU 14 repeats this process. If the automobile 1 has started running, the operation ECU 14 advances the process to step ST32.

[0115] In step ST32, the ECU 14 is operated to obtain driving information. The driving information may include, for example, a hand-holding alarm request of the occupant monitoring device 81, the presence or absence of the hand-holding alarm output of the alarm device 87, the detection value of the hand-holding sensor 44, and information for determining whether automatic driving is possible. The information for determining whether automatic driving is possible may include, for example, information on the state of the occupant, images captured by the external camera 43, information on other surrounding moving objects, navigation data, and information on the route based thereon.

[0116] In step ST33, the operation ECU 14 determines whether the hand-holding alarm is performed. When the alarm device 87 outputs the hand-holding alarm, or when the occupant monitoring device 81 outputs the hand-holding alarm request, the operation ECU 14 determines that the hand-holding alarm is performed and advances the processing to step ST34. In other cases, the operation ECU 14 advances the processing to step ST36.

[0117] In step ST34, the ECU 14 determines whether the occupant is in the hand-holding state where manual operation can be started immediately based on the detection value of the hand-holding sensor 44. If the occupant is in the hand-holding state, the ECU 14 causes the process to proceed to step ST36. If the occupant is not in the hand-holding state even after the alarm, the ECU 14 causes the process to proceed to step ST35.

[0118] In step ST35, the ECU 14 is operated to execute the automatic deceleration and stop control for safely stopping the automobile 1. The ECU 14 is operated to decelerate the traveling automobile 1 while confirming the safety of the surroundings, and stops the automobile 1 at a safe roadside, etc. Thus, the automobile 1 can be decelerated and stopped even if the occupant does not hold the vehicle even after the alarm.

[0119] In step ST36, the ECU 14 determines whether the vehicle can be driven automatically by automatic driving. For example, if the ECU 14 has navigation data and information on the route ahead based thereon and information on other surrounding moving objects, and if autonomous driving is possible under a state where reliable safety including passenger support is ensured, the ECU 14 determines that the vehicle can be driven automatically by automatic driving, and the process proceeds to step ST37. If not, the process proceeds from step ST36 to step ST39.

[0120] In step ST37 , the ECU 14 is operated to execute autonomous automatic driving control.

[0121] In step ST38, since the automobile 1 is in the automatic driving state, the ECU 14 is operated to output a hands-off request to the vehicle network 25. The alarm unit 88 receives and outputs the hands-off request through the vehicle network 25. This allows the occupant to release his hands from the steering wheel 31, for example.

[0122] In step ST39 , since the automobile 1 cannot automatically travel by automatic driving, the manipulation ECU 14 outputs a hand-holding request to the vehicle network 25 .

[0123] The alarm unit 88 acquires and outputs the hand-holding request via the vehicle network 25. As a result, the occupant can hold the steering wheel 31 with his hands.

[0124] In step ST40 , the operation ECU 14 controls the travel of the automobile 1 based on the driving operation of the occupant so as to assist the driving operation.

[0125] Figure 7 This is an explanatory diagram of an example of a process for generating and outputting an alarm request when captured images in which a face cannot be recognized are repeatedly generated.

[0126] For captured images in which the face cannot be recognized, the face recognition unit 82 does not output a permission signal, so the side-gaze determination unit 83 and the sleepiness determination unit 84 do not perform their respective processes. In contrast, the continuous loss determination unit 85 and the frequent loss determination unit 86 perform their respective processes based on the continued generation of loss images.

[0127] Figure 7 In the video, time flows from left to right.

[0128] Figure 7 Middle (A)~ Figure 7 The middle (C) is a timing chart showing an example of a situation where image loss, which makes face recognition impossible by the face recognition unit 82, continues to occur.

[0129] Figure 7 (A) in the figure indicates a cycle for acquiring a captured image. Each arrow corresponds to one cycle for outputting a captured image by the interior camera 61. The occupant monitoring ECU 17 executes a process for acquiring a captured image from the interior camera 61 at each capturing cycle indicated by an arrow.

[0130] Figure 7 (B) is an output signal of the recognition result generated by the face recognition unit 82 with respect to the acquired captured images. In this example, the face recognition unit 82 outputs a recognition OK (good) permission signal as an output signal for the first two captured images. For the next three captured images, the face recognition unit 82 outputs a recognition NG (bad) disapproval signal as an output signal. In addition, for the next three captured images, a recognition OK permission signal is output as an output signal.

[0131] Figure 7 The middle (C) is an output signal of an alarm request output by the continuous loss determination unit 85 based on continuous image loss.

[0132] In this example, the continuous loss determination unit 85 outputs an alarm request when three or more image losses occur continuously. The period during which three or more captured images are continuously output is the period for determining the continuous loss. The determination threshold value of the count value is 3.

[0133] Figure 7 Middle (D)~ Figure 7 The middle (F) is a timing chart showing an example of a situation in which image loss, which makes face recognition impossible by the face recognition unit 82, occurs intermittently.

[0134] Figure 7 Middle (D) and Figure 7 Same as (A) in the figure, it indicates the acquisition cycle of the captured image.

[0135] Figure 7 (E) is an output signal of the recognition result generated by the face recognition unit 82 with respect to the acquired captured image. In this example, the face recognition unit 82 outputs a recognition OK permission signal as an output signal for the first captured image. For the next captured image, a recognition NG disapproval signal is output as an output signal. Then, as an output signal for each captured image, a recognition OK permission signal and a recognition NG disapproval signal are alternately output.

[0136] Figure 7 The middle (F) is an output signal of an alarm request output by the frequent loss determination unit 86 based on intermittent image loss.

[0137] In this example, the frequent loss determination unit 86 outputs a warning request when four or more image losses occur intermittently during the determination period of the most recent seven or more frequent losses. The determination threshold value of the count value is four.

[0138] In addition, the judgment period of frequent loss is longer than the judgment period of continuous loss. The judgment threshold of the count value in the judgment period of frequent loss is greater than the judgment threshold of the count value in the judgment period of continuous loss. Thus, when continuous image loss occurs, for example, more than 4, the frequent loss judgment unit 86 can not output an alarm request before the continuous loss judgment unit 85. The continuous loss judgment unit 85 can output an intentional alarm request by making its own judgment without affecting the output of the alarm request of the frequent loss judgment unit 86.

[0139] As described above, this embodiment is based on the captured images of the interior camera 61 that can periodically and repeatedly capture the occupants of the automobile 1. First, as a preliminary process, the face recognition unit 82 performs recognition processing on the face of the occupant or the eyes, nose, and mouth that are part of the face. In addition, the occupant judgment unit such as the squint judgment unit 83 and the drowsiness judgment unit 84 judges that the occupant is not in an inappropriate driving state such as squint or drowsiness for the captured images of the interior camera 61 in which the face can be recognized by the face recognition unit 82. Therefore, the occupant judgment unit does not need to judge that the occupant is not in an inappropriate driving state such as squint or drowsiness for the captured images of the interior camera 61 with poor quality. The occupant judgment unit only needs to judge that the occupant is not in an inappropriate driving state such as squint or drowsiness based on a part of the captured images with good quality, on the premise that the captured images of the interior camera 61 are captured in a manner that can correctly recognize the face of the occupant. The judgment of whether the occupant is in an inappropriate driving state such as squint or drowsiness by the occupant judgment unit can be adapted to the content that becomes appropriate for the judgment. The judgment accuracy of the occupant judgment unit is increased in a manner appropriate for the judgment. In particular, when judging strabismus or drowsiness based on the eyes in the captured image, the size and resolution of the eye image in the image are often not sufficiently high, and the occupant judgment unit is required to judge the state of the eye with high accuracy based on the information of the eye with less information in the image. For example, compared with simply judging the presence or absence of eye detection based on the state of the eye opening degree, the size of the pupil, etc., a very high-precision judgment is required. In the present invention, the occupant judgment unit can be set as a component that can make such a high-precision judgment.

[0140] In addition, in the present embodiment, a loss determination unit such as a continuous loss determination unit 85 and a frequent loss determination unit 86 that generate an alarm request are provided in parallel with the occupant determination unit that can perform such a high-precision determination. In addition, these loss determination units generate an alarm request when a plurality of captured images that can be output from the internal camera 61 and the captured images of the eyes, nose, and mouth that cannot be recognized by the face recognition unit 82 or are part of the face are repeatedly generated. Thus, the loss determination unit can generate an alarm request for the captured images that are not involved in the determination of the occupant determination unit among the captured images of the internal camera 61. When there is a possibility that the occupant is in an inappropriate driving state such as squinting or drowsing in the captured images of the internal camera 61, for example, the captured images that are not involved in the determination of the occupant determination unit, the loss determination unit can generate an alarm request based on the inherent possibility without directly determining the state of the occupant.

[0141] For example, for a captured image of an eye in a state where the occupant judgment unit cannot make a high-precision judgment, it is determined that the eye as a part of the occupant's face cannot be recognized by the face recognition unit 82. Thus, in the case of an image or situation where the state of the eye cannot be directly determined from the image, an alarm request can be generated and output by the judgment of the loss judgment unit. For a low-quality captured image in the captured image of the internal camera 61 that is difficult to judge in the occupant judgment unit, an alarm request can be generated by the judgment of the loss judgment unit. The occupant can respond to the alarm based on the judgment of the loss judgment unit in the same way as the case of the alarm based on the judgment of the occupant judgment unit. By performing multi-layered direct and indirect judgments of the occupant's driving state by the occupant judgment unit and the loss judgment unit, an alarm can be output in a situation where an alarm request should be generated. In a situation where an alarm request should be generated, the occurrence of a situation where an alarm request cannot be generated can be effectively suppressed.

[0142] In addition, for example, when the interior camera 61 fails, the captured image cannot be obtained from the interior camera 61. In this case, the face recognition unit 82 does not output a permission signal, as in the case where the face cannot be recognized in the captured image. As a result, the loss determination unit such as the continuous loss determination unit 85 and the frequent loss determination unit 86 can generate an alarm request based on the continuation of the situation in which the captured image cannot be obtained from the interior camera 61. In this embodiment, not only the state of the occupant, but also the situation that the occupant monitoring function based on the image of the interior camera 61 cannot function correctly can be notified to the occupant through an alarm.

[0143] By the parallel cooperation processing of the occupant determination unit and the loss determination unit, in this embodiment, regardless of the environment and state of the internal camera 61, when there is a possibility that the occupant is in an inappropriate driving state such as squinting or drowsiness, the occupant can be notified of this situation through an alarm. In this embodiment, it is possible to achieve both accurate judgment that the occupant is not in an inappropriate driving state such as squinting or drowsiness, which are mutually contradictory requests, and reliable output of an alarm about this situation when there is a possibility that the occupant is in an inappropriate driving state such as squinting or drowsiness. The driving assistance system 80 of this embodiment can achieve both accurate recognition judgment about occupant monitoring in the automobile 1 and high safety assurance at a high level, and can therefore be appropriately used in the automobile 1.

[0144] In addition, the occupant alarm device of the automobile 1 composed of the occupant monitoring device 81 and the alarm device 87 can also be expected to have the same effects as the above-mentioned device.

[0145] The above-described embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications and changes can be made without departing from the spirit of the invention.

[0146] For example, in the above-described embodiment, the occupant monitoring device 81 and the alarm device 87 are formed separately and are connected to each other via the vehicle network 25 .

[0147] In addition, for example, the occupant monitoring device 81 and the alarm device 87 may be integrally provided in the automobile 1 as an occupant alarm device of one automobile 1, and the entire device may be connected to the vehicle network 25. In this case, the occupant monitoring device 81 includes the alarm unit 88 and the like.

Claims

1. A driving assistance system for a vehicle, comprising: a photographing device capable of outputting a first photographed image of an occupant of the vehicle; an alarm unit that outputs an alarm to an occupant; as well as a control unit capable of executing control using the imaging device and the alarm unit, The control unit functions as the following components: a face recognition unit that performs recognition processing of at least a part of the face of the occupant on the first captured image acquired from the capturing device; an occupant determination unit that determines a driving state of the occupant based on a second captured image of the first captured image that can be output by the camera, in which at least a portion of the face can be recognized by the face recognition unit, and generates a first alarm request when the occupant is not in a state to drive; and a loss determination unit that generates a second alarm request when a plurality of third captured images in which at least a part of the face cannot be recognized by the face recognition unit are generated among the first captured images that can be output by the photographing device, The warning unit outputs a warning to the occupant based on the first warning request from the occupant determination unit and the second warning request from the damage determination unit.

2. The driving assistance system for a vehicle according to claim 1, wherein: The occupant determination unit determines whether the occupant is looking sideways or dozing off based on the state of eyes of the occupant's face in the second captured image.

3. The driving assistance system for a vehicle according to claim 1, wherein: the face recognition unit regards at least the first captured image in which the occupant determination unit cannot determine the driving state of the occupant as the third captured image in which the face cannot be recognized, The loss determination unit determines whether a plurality of the third captured images in which faces cannot be recognized are generated.

4. The driving assistance system for a vehicle according to claim 2, wherein: the face recognition unit regards at least the first captured image in which the occupant determination unit cannot determine the driving state of the occupant as the third captured image in which the face cannot be recognized, The loss determination unit determines whether a plurality of the third captured images in which faces cannot be recognized are generated.

5. The vehicle driving assistance system according to any one of claims 1 to 4, wherein: When the first captured image cannot be obtained from the shooting device, the loss judgment unit is set to generate the third captured image in which the face recognition unit cannot recognize the face, and judge whether multiple third captured images in which the face cannot be recognized are generated.

6. The driving assistance system for a vehicle according to any one of claims 1 to 4, wherein: The loss determination unit generates the second alarm request when the facial recognition unit continues to be unable to recognize the face in the first captured image that can be output by the capturing device.

7. The driving assistance system for a vehicle according to any one of claims 1 to 4, wherein: The loss determination unit includes: a first loss determination unit that generates a third alarm request as the second alarm request when a state in which the face recognition unit cannot recognize the face continues to occur continuously; and The second loss determination unit generates a fourth alarm request as the second alarm request when the state in which the face recognition unit cannot recognize the face continues to occur in an intermittent and repeated manner, The warning unit outputs a warning to the occupant based on the third warning request of the first damage determination unit and the fourth warning request of the second damage determination unit as the second warning request.

8. The driving assistance system for a vehicle according to claim 7, wherein: The second loss determination unit determines that repetition occurs intermittently over a period longer than a period during which the first loss determination unit determines that continuity occurs.

9. The driving assistance system for a vehicle according to any one of claims 1 to 4, wherein: A detection unit is provided for detecting whether an operating member operated by an occupant when driving the vehicle is in a state in which the occupant can operate the operating member. When generating the first alarm request of the occupant determination unit or the second alarm request of the damage determination unit, the alarm unit determines whether the occupant is in a state where the occupant can operate the operating member based on the detection result of the detection unit. When it cannot be determined that the operating member is in a state where the operating member can be operated, an alarm is output to the occupant to request the occupant to operate the operating member.

10. The driving assistance system for a vehicle according to claim 9, wherein: A travel control device is provided, which controls the travel of the vehicle, The travel control device decelerates or stops the vehicle if the detection result of the detection unit does not change to indicate that the occupant is in a state where the operating member can be operated after the alarm unit outputs the alarm.

Images