Vigilance level assessment device, device to support automated driving and vigilance assessment procedure

Abstract

Vigilance level assessment device, comprehensive a portrait acquisition component for acquiring a portrait, which is an image taken of the face of an occupant of a vehicle, an eye opening degree calculation component that calculates the eye opening degree of the occupant based on the portrait, an open / closed eyes assessment component that evaluates whether the eye opening level is less than a predefined threshold or not, a smile assessment angle calculation component that, based on the portrait, determines the positions of both corners of the mouth and the midpoint of both inner corners of the eyes of the occupant, and calculates a smile assessment angle as an angle from a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes. a smile assessment component that assesses whether the occupant is smiling or not based on the smile assessment angle, and an alertness level assessment component that assesses a decrease in the inmate's alertness level when the eye opening level has become less than a threshold, and it has also been assessed that the inmate is not smiling.

Description

Technical field

[0001] The present invention relates to a technique for assessing the level of alertness of vehicle occupants. Background technology

[0002] A vigilance level assessment device which assesses the vigilance level of the driver (presence / absence of fatigue) based on the eye opening level (openness degree of the eyes, i.e., the state of the eyes) determined from an image of the driver's face taken by means of an onboard camera is known (see, e.g., patent documents 1 to 3).

[0003] It sometimes happens that a driver smiles while talking to other passengers or during a hands-free phone call. Generally, the eyes narrow when a person smiles, so the eye opening level during a smile is close to the eye opening level associated with a decrease in alertness. Therefore, the challenge for the alertness level assessment device is to prevent a misinterpretation of a decrease in alertness when the driver smiles.

[0004] In the vigilance level assessment devices described in patent documents 1 to 3, an assessment of the vigilance level is made based on the driver's eye opening degree, along with an evaluation of whether the driver is smiling or not, whereby a high vigilance level is assessed even if the driver's eye opening degree has decreased, thus preventing this misjudgment when the driver is smiling.

[0005] Patent document 4 further discloses a vehicle driver assistance system comprising one or more processors configured to execute a general driver model learning program configured to create a general driver model applicable to a plurality of drivers based on driving data from the plurality of drivers, and an individual driver model learning program configured to create an individual driver model unique to a specific driver based on driving data from that specific driver. The system also includes an on-board controller provided in a vehicle operated by that specific driver. The individual driver model learning program includes a vehicle control update program configured to cause the on-board controller to update the vehicle control settings.-control processing based on the general driver model and the individual driver model. The vehicle control / regulation update program acquires the general driver model and the individual driver model and, according to a predefined condition, determines a driver model on which the vehicle control / regulation processing is updated, specifically between the general driver model and the individual driver model. State-of-the-art documents, patent documents Patent document 1: JP Patent Publication No. JP 2016 - 115 120 A Patent document 2: JP Patent Publication No. JP 2010 - 9 093 A Patent document 3: JP Patent Publication No. JP 2010 - 142 345 A Patent document 4: DE 10 2018 001 342 A1 Overview of the invention Problem to be solved by the invention

[0006] Patent documents 1 to 3 propose various methods for assessing a driver's smile, but a new assessment method is desirable to further improve the accuracy of the assessment of the level of alertness.

[0007] The present invention aims to solve the aforementioned problems and to provide a vigilance level assessment device in which the vigilance level of the occupant in question can be assessed with high precision by taking into account whether the occupant of a vehicle is smiling or not. Means of solving the task

[0008] The alertness level assessment device according to the present invention comprises a portrait acquisition component for acquiring a portrait, which is an image taken of the face of a vehicle occupant; an eye opening level calculation component that calculates the occupant's eye opening level based on the portrait; an open / closed eye assessment component that assesses whether the eye opening level is less than a predetermined threshold; and a smile assessment angle calculation component that determines the positions of both corners of the mouth and the midpoint of both inner corners of the occupant's eyes based on the portrait and calculates a smile assessment angle as the angle formed by a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes.a smile assessment component that assesses whether the occupant is smiling or not based on the smile assessment angle, and an alertness level assessment component that assesses a decrease in the occupant's alertness level when the eye opening level has become less than a threshold, and it has also been assessed that the occupant is not smiling. Effects of the invention

[0009] According to the present invention, a decrease in an occupant's alertness level is assessed if the occupant's eye opening falls below a threshold and it is also determined that the occupant is not smiling. That is, if it is determined that the occupant is smiling, the alertness level is assessed as high even if the occupant's eye opening is below a threshold. This prevents a false assessment of a decrease in alertness when the occupant is smiling and improves the accuracy of the alertness assessment. Furthermore, the assessment of whether an occupant is smiling is based on a smile assessment angle calculated from the position of the two corners of the mouth and the midpoint of both inner corners of the occupant's eyes.In general, the position of the corners of the mouth changes very strongly when a person smiles or does not smile, so that the assessment of whether an inmate is smiling or not can be carried out easily and with a high degree of precision.

[0010] The purpose, characteristics, aspects and advantages of the present invention will become clearer from the following detailed explanation and the attached figures. Brief explanation of the characters [ Fig. 1] Fig. Figure 1 is a block diagram showing the structure of the vigilance level assessment device according to a first embodiment. [ Fig. 2] Fig. Figure 2 is a flowchart showing the operation of the vigilance level assessment device according to the first embodiment. [ Fig. 3] Fig. Figure 3 is a block diagram showing a modified example of the vigilance level assessment device according to the first embodiment. [ Fig. 4] Fig. Figure 4 is a view showing a hardware structure example of the vigilance level assessment device. [ Fig. 5] Fig. Figure 5 is a view showing a hardware structure example of the vigilance level assessment device. [ Fig. 6] Fig. Figure 2 is a block diagram showing the structure of the vigilance level assessment device according to a second embodiment. [ Fig. 7] Fig. Figure 7 is a flowchart showing the operation of the vigilance level assessment device according to the second embodiment. [ Fig. 8] Fig. Figure 8 is a block diagram showing the construction of a device to support automatic driving according to a third embodiment. Forms for carrying out the invention<Erste Ausführungsform>

[0011] Fig. Figure 1 is a block diagram showing the structure of a vigilance level assessment device 10 according to a first embodiment. In the present embodiment, it is assumed that the vigilance level assessment device 10 is installed in a vehicle. However, there is no restriction to the vigilance level assessment device 10 being permanently installed in a vehicle; it can also be set up in a mobile device that can be taken into a vehicle, such as a mobile phone, smartphone, mobile navigation device, etc. Furthermore, part or all of the vigilance level assessment device 10 can also be set up on a server located outside a vehicle and capable of communicating with a vehicle.

[0012] As in Fig. As shown in Figure 1, the vigilance level assessment device 10 is connected to a camera 21 and a warning device 22 with which the vehicle is equipped.

[0013] Camera 21 is used to record the passenger compartment and is installed in a position that allows for the recording of the faces of the vehicle's occupants. In the present embodiment, camera 21 is a wide-angle camera with a recording area that enables the recording of the faces of the occupants in all seats of the vehicle. However, if the vigilance level assessment device 10 is intended, for example, to assess only the vigilance of the occupant in the driver's seat (driver), a recording area of ​​camera 21 sufficient to capture at least the driver's seat is adequate.

[0014] The alertness level assessment device 10 uses an image of the occupant's face (hereinafter referred to as the "portrait") captured by the camera 21 to assess the occupant's alertness level (presence / absence of fatigue) and outputs this assessment result to the warning device 22. In the present embodiment, the alertness level assessment device 10 is configured to assess the alertness level of all occupants captured by the camera 21. In this case, the alertness level assessment device 10 outputs data on the seat position of each occupant along with the alertness level assessment result. That is, the alertness level assessment device 10 outputs data indicating which occupant was assessed as having a high level of alertness and which occupant was assessed as having a low level of alertness.

[0015] The warning device 22 emits a warning into the passenger compartment according to the occupant's level of alertness, as assessed by the alertness assessment device 10. It includes a loudspeaker for emitting a warning tone or message, a display for showing a warning screen, etc. In the present embodiment, the warning device 22 emits a warning when it is assessed that the level of alertness of the occupant in the driver's seat (driver) has decreased. However, there is no limitation on the operational content of the warning device 22, so that, for example, a warning can also be issued if it is assessed that the level of alertness of another occupant besides the driver has decreased.

[0016] As in Fig. 1, the alertness level assessment device 10 comprises a portrait acquisition component 11, an eye opening level calculation component 12, an open eyes / closed eyes assessment component 13, a smile assessment angle calculation component 14, a smile assessment component 15 and an alertness level assessment component 16.

[0017] The portrait acquisition component 11 acquires the portrait of an occupant captured by the camera 21. Since, in the present embodiment, the camera 21 is a wide-angle camera capable of capturing all seats in the vehicle, the portrait acquisition component 11 extracts one portrait per occupant from the image of the passenger compartment captured by the camera 21 using facial recognition technology.

[0018] The eye opening degree calculation component 12 calculates the eye opening degree (opening state of the eyes) of the occupant in question based on the portrait of the occupant acquired by the portrait acquisition component 11. If portraits of multiple occupants were acquired by the portrait acquisition component 11, the eye opening degree calculation component 12 calculates the respective eye opening degree of each of those multiple occupants. The method for calculating the eye opening degree in the eye opening degree calculation component 12 is not particularly restricted; for example, a method described in patent documents 1 to 3 may also be used.

[0019] In the present embodiment, the eye opening degree calculation component 12 calculates as eye opening degree a value which is obtained by determining, starting from the portrait of an occupant, the position of the inner corners of the eyes, the position of the outer corners of the eyes and the position of the highest point of the occupant's upper eyelids, by dividing the distance between a straight line connecting the inner and outer corners of the eyes and the highest point of the upper eyelid by the distance between the inner and outer corners of the eyes, a degree of flattening of the eye, and dividing the degree of flattening by a predetermined reference value.Generally, determining the contour line of a lower eyelid from a portrait is more difficult than determining the contour line of an upper eyelid. However, this method calculates the degree of eye opening without using data from the lower eyelid contour line, allowing for a highly precise calculation. The highest point of the upper eyelid is the point (tip) of the upper eyelid furthest from a line connecting the outer and inner corners of the eye.

[0020] The reference value for the degree of eye flattening can, for example, be a constant value defined by the manufacturer of the alertness level assessment device 10 as the average degree of eye flattening. If the alertness level assessment device 10 can identify individual occupants using facial recognition, etc., reference values ​​for each occupant can be recorded in the alertness level assessment device 10. Using different reference values ​​for each occupant prevents the calculated eye opening degree from being influenced by individual differences in eye size.

[0021] The Open Eyes / Closed Eyes Assessment Component 13 compares the occupant's eye opening level calculated by the Eye Opening Level Calculation Component 12 with a predefined threshold and, based on whether the occupant's eye opening level is greater than or equal to the threshold, assesses whether the occupant's eyes are open or closed. Hereinafter, this assessment is referred to as the "Open Eyes / Closed Eyes Assessment," where a state in which the eye opening level is greater than or equal to the threshold is referred to as the "Open Eyes State" and a state in which the eye opening level is less than the threshold is referred to as the "Closed Eyes State."As previously described, the precision of the eye opening degree calculated by the eye opening degree calculation component 12 is high, so that a high precision of the open eyes / closed eyes assessment by the open eyes / closed eyes assessment component 13 also results.

[0022] The smile assessment angle calculation component 14, starting from the portrait, determines the positions of both corners of the mouth and the midpoint of both inner corners of the eyes, and calculates the angle formed by a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes. This angle is defined as the "smile assessment angle".

[0023] Smile Assessment Component 15 assesses whether an occupant is smiling or not based on their Smile Assessment Angle, which is calculated by Smile Assessment Angle Calculation Component 14. Specifically, Smile Assessment Component 15 compares the occupant's Smile Assessment Angle to a predefined threshold and assesses that the occupant is smiling if the Smile Assessment Angle is greater than or equal to the threshold. This assessment is referred to hereafter as the "smile assessment."

[0024] In the present embodiment, the occupant's smile is assessed based on a smile assessment angle calculated from the positions of both corners of the mouth and the midpoint of both inner corners of the occupant's eyes. Generally, the position of the mouth corners changes significantly when a person smiles or does not smile, so this smile assessment method allows for a simple and highly precise evaluation of the occupant's smile.

[0025] Vigilance Level Assessment Component 16 assesses whether an occupant's vigilance level has decreased based on the results of the Open Eyes / Closed Eyes Assessment Component 13 and the Smile Assessment Component 15. Specifically, Vigilance Level Assessment Component 16 assesses a decrease in an occupant's vigilance level if the occupant's eyes were assessed as closed and they were not smiling. In other words, a decrease in an occupant's vigilance level is assessed if the occupant's eye opening is less than a threshold and their smile angle is also less than a threshold.Since, as described above, the Open Eyes / Closed Eyes Assessment Component 13 can perform the open eyes / closed eyes assessment with high precision, and the Smile Assessment Component 15 can perform the smile assessment with high precision, the inmate alertness level assessment by the alertness level assessment Component 16 also achieves high precision.

[0026] Therefore, by taking into account whether an occupant is smiling or not, the vigilance level of the occupant can be assessed with high precision by the vigilance level assessment device 10 according to the first embodiment.

[0027] Fig. Figure 2 is a flowchart illustrating the operation of the vigilance level assessment device 10 according to the first embodiment. The following refers to Fig. 2 explains the operation of the vigilance level assessment device 10 of the first embodiment.

[0028] When the vehicle's ignition is switched on and the vigilance level assessment device 10 is started, the portrait acquisition component 11 first acquires portraits of the occupants captured by the camera 21 (step S101). For the sake of simplicity, it is assumed here that the image captured by the camera 21 contains only the portrait of one occupant. However, if, as in the present embodiment, the camera 21 is a wide-angle camera, the portraits of multiple persons can be acquired in step S101. In this case, the processing of the following steps S102 to S108 is carried out for the portraits of the respective occupants.

[0029] Next, using the portrait of the occupant acquired in step S101, the eye opening degree calculation component 12 determines the position of the inner corner of the eye, the position of the outer corner of the eye and the position of the highest point of the upper eyelid of the occupant, and calculates the eye opening degree of the occupant based on their positional relationship (step S102).

[0030] Then, the Open Eyes / Closed Eyes Assessment Component 13 performs an open eyes / closed eyes assessment of the occupant in question by comparing the degree of eye opening calculated in step S102 with a predefined threshold (step S103).

[0031] If, in step S103, it was determined that the occupant in question is in a closed-eye state (eye opening degree less than the threshold) (YES in step S103), the smile assessment angle calculation component 14 calculates the smile assessment angle of the occupant in question based on the portrait of the occupant acquired in step S101 (step S104). Then, the smile assessment component 15 performs a smile assessment of the occupant in question by comparing the smile assessment angle calculated in step S104 with a predefined threshold (step S105).

[0032] If no smile of the occupant in question was assessed in step S105 (smile assessment angle less than the threshold) (NO in step S105), the vigilance level assessment component 16 estimates a decrease in the vigilance level of the occupant in question (step S106).

[0033] On the other hand, if in step S103 the occupant in question was assessed to be in an open eye state (eye opening degree greater than or equal to the threshold) (NO in step S103), or if in step S105 the occupant was assessed to be smiling (smile assessment angle greater than or equal to the threshold) (YES in step S105), the vigilance level assessment component 16 will assess a high vigilance level of the occupant in question (step S107).

[0034] The assessment result of the occupant's alertness level according to step S106 or S107 is output to the warning device 22 (step S108). This causes the warning device 22 to issue a warning corresponding to the occupant's alertness assessment result.

[0035] The vigilance level assessment device 10 repeatedly executes the aforementioned sequence. This sequence can be performed regularly, for example, after the vigilance level assessment device 10 is started (after the vehicle's ignition is switched on), or it can be performed only while the vehicle is in motion (not while the vehicle is stationary). With regular execution of this sequence, the occupant's vigilance level is assessed before the vehicle begins to move, thus contributing to improved vehicle safety.In general, however, the requirement to assess the occupant's level of alertness while the vehicle is stopped is not particularly high, so that by executing the relevant process exclusively while the vehicle is moving, a reduction in the processing load of the alertness level assessment device 10 and a reduction in power consumption can also be sought. [Modified example]

[0036] In the vigilance level assessment device 10 of the first embodiment, the vigilance level assessment component 16 outputs the assessment result of the occupant's vigilance level to the warning device 22, the output address being not limited to the warning device 22, but rather any device. As in Fig. As shown in Figure 3, the assessment result of the occupant's level of alertness can, for example, be entered into a device for controlling automatic driving 23 by the alertness level assessment component 16, which is an ECU (Electronic Control Unit) that performs automatic driving of a vehicle.

[0037] As operation of the device for controlling an automatic driving system 23 in Fig. 3 For example, an operation is conceivable in which, if the vigilance level assessment component 16 has assessed a decrease in the vigilance level of the occupant of the driver's seat (driver), and despite the issuance of a warning by the warning device 22 the vigilance level of the driver remains in the decreased state, the device for controlling an automatic driving system 23 automatically stops the vehicle for parking in a safe place (e.g. a wide shoulder, a rest area, etc.). [Hardware structure examples]

[0038] Fig. 4 and Fig. Figure 5 shows views illustrating hardware structure examples of the vigilance level assessment device 10. The respective functions of the components of the device are described in Figure 5. Fig. The vigilance level assessment device 10 shown in 1 is, for example, provided by a device in Fig. 4. The processing circuit 50 shown is implemented. That is, the alertness level assessment device 10 comprises a processing circuit 50 which acquires a portrait, which is an image taken of the face of a vehicle occupant; uses the portrait to calculate the occupant's eye opening level and assesses whether the eye opening level is less than a predetermined threshold; uses the portrait to determine the positions of both corners of the mouth and the midpoint of both inner corners of the occupant's eyes; calculates a smile assessment angle as the angle formed by a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes; and, based on the smile assessment angle, assesses whether the occupant is smiling in order to evaluate a decrease in the occupant's alertness level.when the eye opening degree has become less than a threshold value, and it has also been determined that the occupant is not smiling. The processing circuit 50 can be dedicated hardware, but it can also be formed by means of a processor (also called central processing unit (CPU), processing device, computing device, microprocessor, microcomputer, DSP (digital signal processor)) that executes programs stored in memory.

[0039] If the processing circuit 50 is a special piece of hardware, the processing circuit 50 corresponds, for example, to a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or a combination thereof, etc. The respective functions of the components of the vigilance level assessment device 10 can be implemented by separate processing circuits, but these functions can also be implemented by a single integrated processing circuit.

[0040] Fig. Figure 5 shows an example of a hardware structure of the vigilance level assessment device 10 in the case where the processing device 50 is formed by means of a processor 51 for executing programs. In this case, the functions of the components of the vigilance level assessment device 10 are implemented by software, etc. (software, firmware, or a combination of software and firmware). The software, etc., is represented as a program and stored in a memory 52. ​​The processor 51 reads the programs stored in the memory 52 and executes them, thereby realizing the various functions. That is, the vigilance level assessment device 10 includes the memory 52 for storing programs, and their execution by the processor 51 ultimately results in processing to acquire a portrait, which is an image taken of the face of a vehicle occupant.a processing step to calculate the occupant's eye opening degree based on the portrait, a processing step to assess whether the eye opening degree is less than a predetermined threshold, a processing step that, based on the portrait, determines the positions of both corners of the mouth and the midpoint of both inner corners of the occupant's eyes, and calculates a smile assessment angle as the angle formed by a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes, a processing step that, based on the smile assessment angle, assesses whether the occupant is smiling, and a processing step that assesses a decrease in the occupant's alertness level if the eye opening degree has become less than a threshold, and also assesses that the occupant is not smiling.are executed. In other words, these are programs that execute the sequence and procedure of the operation of the components of the vigilance level assessment component 10 by a computer.

[0041] The memory 52 can be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), Flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), etc., HDD (Hard Disk Drive), magnetic disk, floppy disk, optical disk, Compact Disc, Minidisc, DVD (Digital Versatile Disc) and their driver device, etc., or various storage media used in the future.

[0042] The structure described above assumes that the functions of the components of the vigilance level assessment component 10 are executed by either hardware or software, etc. However, this is not a limitation; the structure can also be such that some components of the vigilance level assessment component 10 are implemented by dedicated hardware, while others are implemented by software, etc. For example, the processing circuit 50, as dedicated hardware, can execute the functions of some components, while the processing circuit 50, acting as a processor 51, reads and executes programs stored in memory 52.

[0043] This allows the vigilance assessment component 10 to implement the respective functions described above through hardware, software, etc., or a combination thereof. <Zweite Ausführungsform>

[0044] Fig. Figure 6 is a block diagram showing the construction of the vigilance level assessment device 10 according to a second embodiment. The construction of the vigilance level assessment device 10 in Fig. 6 was compared to the setup in Fig. 1. An eye shape detection component 17 was added. The remaining structure is basically the same as in Fig. 1, so that only the differences to the vigilance level assessment device 10 are shown here. Fig. 1 will be explained.

[0045] The Eye Shape Assessment Component 17 uses the portrait of the occupant, acquired by the Portrait Acquisition Component 11, to determine information about the occupant's eye shape. Examples of information acquired by the Eye Shape Assessment Component 17 include the distance between the upper and lower eyelids, the distance between the outer and inner corners of the eye, the angle formed by the outer corner, and so on. The angle formed by the outer corner is defined as the angle formed by a straight line connecting one outer corner to the midpoint of both inner corners and a straight line connecting the other outer corner to the midpoint of both inner corners.

[0046] Furthermore, in the second embodiment, the smile assessment component 15 assesses not only the smile assessment angle (angle formed by a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes) of the occupant, which is calculated by the smile assessment angle calculation component 14, but also the eye shape of the occupant in question, which was determined by the eye shape detection component 17, to determine whether the occupant in question is smiling or not.

[0047] When a person smiles, the shape of the eyes generally changes, either by narrowing or by drooping the outer corner. Specifically, when a person smiles, the eye shape changes in that the distance between the upper and lower eyelids shortens, the distance between the outer and inner corners lengthens, and the angle formed by the outer corner decreases. By using a combined approach, the smile assessment component 15 determines whether an occupant is smiling by considering both the smile assessment angle and information about the occupant's eye shape, thus improving the accuracy of smile assessments.

[0048] Fig. Figure 7 is a flowchart illustrating the operation of the vigilance level assessment device 10 according to the second embodiment. The sequence in Fig. 7 was between S104 and S105 of the process in Fig. 2. One step S110 was added.

[0049] The following refers to Fig. 7 explains the working method of the vigilance level assessment device 10 of the second embodiment.

[0050] When the alertness level assessment device 10 is started, the portrait acquisition component 11 first acquires a portrait of the occupant taken by the camera 21 (step S101). Next, the eye opening level calculation component 12 uses the portrait of the occupant acquired in step S101 to determine the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the highest point of the upper eyelid of the occupant, and calculates the eye opening level of the occupant based on the relationship between these positions (step S102). Then, the open / closed eye assessment component 13 performs an open / closed eye assessment of the occupant by comparing the eye opening level of the occupant calculated in step S102 with a predefined threshold (step S103).

[0051] If the occupant's eye state was assessed as closed in step S103 (YES in step S103), the smile assessment angle calculation component 14 calculates the smile assessment angle of the occupant in question (step S104). Additionally, the eye shape determination component 17 determines the eye shape of the occupant in question (step S110). Then, the smile assessment component 15 performs a smile assessment of the occupant in question by combining the smile assessment angle calculated in step S104 with the eye shape information determined in step S110 (step S105).

[0052] If, in step S105, it was assessed that the occupant was not smiling (NO in step S105), the vigilance level assessment component 16 assesses a decrease in the vigilance level of the occupant in question (step S106).

[0053] On the other hand, if in step S103 the open eye state of the occupant in question was assessed (NO in step S103), or if in step S105 the occupant was assessed as smiling (YES in step S105), the vigilance level assessment component 16 assesses a high vigilance level of the occupant in question (step S107).

[0054] The assessment result of the occupant's alertness level according to step S106 or S107 is output to the warning device 22 (step S108). This causes the warning device 22 to issue a warning corresponding to the occupant's alertness assessment result.

[0055] The vigilance level assessment device 10 repeatedly executes the above sequence. In Fig. In the second embodiment, step S110 was inserted after step S104, although both step S110 and step S104 can be performed first. Furthermore, in the second embodiment, the output address of the assessment result of the occupant's vigilance level according to the vigilance level assessment device 10 is not limited to the warning device 22, but can also be, for example, the device for controlling an automatic driving system 23, as in the modified example in Fig. 3 trade. <Dritte Ausführungsform>

[0056] The third embodiment shows an example in which the vigilance assessment component 10 shown in the first embodiment has been applied to a device for supporting automatic driving, which supports the automatic driving of a vehicle.

[0057] This document explains the definitions of automation levels (levels of automated driving) for the automatic driving of a motor vehicle. According to SAE (Society of Automotive Engineers) International standard J3016 (September 2016) and its Japanese reference translation JASO TP18004 (February 2018), the self-driving levels of automated driving systems are defined as follows: Level 0 (No automated driving): The driver performs some or all of the dynamic driving tasks. Level 1 (Assisted Driving): The system performs subtasks of vehicle operation control in either the longitudinal or lateral direction within a limited area. Level 2 (Partially Automated Driving): The system performs subtasks of vehicle operation control in both the longitudinal and lateral directions within a limited area. Level 3 (Conditional Driving Automation): The system performs all dynamic driving tasks within a limited area; however, if continuing the operation is difficult, the driver reacts appropriately in response to an intervention request from the system. Level 4 (Highly Automated Driving): The system performs all dynamic driving tasks within a limited range and reacts to difficult operational continuations within a limited range. Level 5 (Fully Automated Driving): The system performs all dynamic driving tasks and responds to difficult operational continuations without restrictions. "Driving dynamic tasks" here refers to all operational and strategic functions that must be performed in real time when operating a vehicle in road traffic (apart from strategic functions such as route planning and the selection of intermediate stops). Furthermore, the "restricted area" refers to specific conditions that are set for the operation of the system or its functions (including geographical restrictions, road surface restrictions, environmental restrictions, traffic restrictions, speed limits, time restrictions, etc.).

[0058] Fig. Figure 8 is a block diagram showing the construction of a device for supporting automatic driving 30 according to the third embodiment. Fig. 8 are elements with the same functions as those in Fig. 1 with the same reference symbols as in Fig. 1 provided, the explanation of which is omitted here.

[0059] As in Fig.8 The device for assisting automated driving 30 is connected to the camera 21, the device for controlling automated driving 23, and a communication device 24. The device for controlling automated driving 23 is an ECU that performs automated driving of a vehicle and corresponds to the preceding “system.” In the present embodiment, it is assumed that the device for controlling automated driving 23 performs at least Level 3 automated driving. The communication device 24 communicates messages to the occupants of a vehicle and includes a loudspeaker for emitting a message tone or an audio message, a display for showing a warning screen, etc.

[0060] Furthermore, the device for assisting automated driving 30 comprises the vigilance level assessment device 10 and an occupant selection component 31. The vigilance level assessment device 10, as in the one described in the first embodiment, assesses the vigilance level of the occupants of a vehicle and outputs this assessment result to the occupant selection component 31.

[0061] The occupant selection component 31 selects an occupant (the occupant designated as the driver) to intervene in the vehicle's control system in response to an intervention request from the automated driving assistance device 30. The intervention request is input into the occupant selection component 31 by the automated driving assistance device 30 when the device 30 is performing Level 3 automated driving and continuing the automated driving has become difficult. Upon receiving an intervention request from the automated driving assistance device 30, the occupant selection component 31 checks the occupant alertness level assessment result from the alertness level assessment device 10 and selects an occupant with a high alertness level to intervene in the vehicle's control system.

[0062] The selection result according to the occupant selection component 31 is entered into the communication device 24. The communication device 24 then sends a message to the occupant selected by the occupant selection component 31, instructing them to intervene in the vehicle's control system.

[0063] If the occupant in the driver's seat is selected by the occupant selection component 31, the authorization to drive the vehicle can be transferred to that occupant without stopping the vehicle. However, if an occupant other than the driver's seat is selected by the occupant selection component 31, the authorization to drive the vehicle can be transferred to that occupant after the automated driving control device 23 has stopped the vehicle at a safe location and the selected occupant has moved to the driver's seat. If there are multiple occupants whose alertness level has been rated as high, the occupant selection component 31 can select any one of them, although it can also ensure that the occupant in the driver's seat is given preference.

[0064] Thus, in the device for supporting automatic driving 30 of the third embodiment, by the device for supporting automatic driving 30 transferring the authorization to drive to an occupant and supporting so-called handover processing, it can be prevented that the authorization to drive is transferred to an occupant with a low level of vigilance.

[0065] The third embodiment shows an example in which the vigilance assessment component 10 shown in the first embodiment has been applied to a device for supporting automatic driving, which supports the automatic driving of a vehicle, whereby of course it is also possible to apply the vigilance assessment component 10 shown in the second embodiment to a device for supporting automatic driving.

[0066] In the present invention, any combination of the respective embodiments is possible within the scope of the invention, and the respective embodiments can also be modified or omitted in a suitable manner.

[0067] The present invention has been explained in detail, the foregoing explanation being in all aspects an exemplary representation, and the invention is not limited thereto. Countless modified examples, not presented as examples, are also conceivable without deviation from the scope of this invention. [Explanation of reference symbols] 10 Vigilance Level Assessment Device 11 Portrait acquisition component 12 Eye opening degree calculation component 13 Open Eyes / Closed Eyes Assessment Component 14 Smile Assessment Angle Calculation Component 15 Smile Assessment Component 16 Vigilance Level Assessment Component 17 Eye shape assessment component 21 camera 22 Warning device 23 Device for controlling an automatic driving system 24 Communication device 30 Device for assisting automated driving 31 Inmate Selection Component 50 processing circuits 51 processor 52 memory slots.

Claims

[1] Vigilance level assessment device comprising a portrait acquisition component for acquiring a portrait, which is an image taken of the face of an occupant of a vehicle, an eye opening degree calculation component that calculates the eye opening degree of the occupant based on the portrait, an open / closed eyes assessment component that evaluates whether the eye opening level is less than a predefined threshold or not, a smile assessment angle calculation component that, based on the portrait, determines the positions of both corners of the mouth and the midpoint of both inner corners of the eyes of the occupant, and calculates a smile assessment angle as an angle from a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes. a smile assessment component that assesses whether the occupant is smiling or not based on the smile assessment angle, and an alertness level assessment component that assesses a decrease in the inmate's alertness level when the eye opening level has become less than a threshold, and it has also been assessed that the inmate is not smiling. [2] Vigilance level assessment device according to claim 1, wherein the smile assessment component assesses that the occupant is smiling when the smile assessment angle becomes greater than a predetermined threshold. [3] Vigilance level assessment device according to claim 1, wherein the smile assessment component, taking into account the eye shape of the occupant determined from the portrait, assesses whether the occupant is smiling or not. [4] Vigilance level assessment device according to claim 1, wherein the vigilance level assessment component outputs an assessment result of the vigilance level of the occupant to a warning device, by which a warning is issued to the occupant of the vehicle. [5] Vigilance level assessment device according to claim 1, wherein the vigilance level assessment component outputs the assessment result of the vigilance level of the occupant to a device for controlling automatic driving of the vehicle. [6] Device for assisting automatic driving, which assists the automatic driving of a vehicle, comprising a vigilance level assessment device according to claim 1, and An occupant selection component which, in the event that continuing the vehicle's automatic driving has become difficult, selects an occupant to intervene in the operation of the vehicle based on the assessment result of the occupant's vigilance level by the vigilance level assessment component. [7] Vigilance level assessment procedure, according to the a portrait acquisition component of a vigilance level assessment device acquires a portrait which is an image taken of the face of an occupant of a vehicle, An eye opening level calculation component of the alertness level assessment device calculates the occupant's eye opening level based on the portrait. An open / closed eyes assessment component of the alertness level assessment device evaluates whether the eye opening level is less than a predetermined threshold or not. A smile assessment angle calculation component of the alertness level assessment device uses the portrait to determine the positions of both corners of the mouth and the midpoint of both inner corners of the occupant's eyes, and calculates a smile assessment angle as the angle from a straight line connecting one corner of the mouth and the midpoint of both inner corners of the eyes, and a straight line connecting the other corner of the mouth and the midpoint of both inner corners of the eyes. A smile assessment component of the alertness level assessment device uses the smile assessment angle to judge whether the occupant is smiling or not, and A vigilance level assessment component of the vigilance level assessment device assesses a decrease in the occupant's vigilance level when the eye opening level has become less than a threshold, and it has also been assessed that the occupant is not smiling.

Images