Driver monitoring system

Abstract

To provide a driver monitoring system to improve the accuracy of facial recognition of a driver in a vehicle.SOLUTION: A driver monitoring system that performs facial recognition of a driver from a facial image of a driver of a vehicle includes: an infrared image capturing unit that captures an infrared image of the driver's face using an infrared light emitting unit and an infrared light receiving unit; a visible light image capturing unit that captures a visible light image of the driver's face using a visible light receiving unit; a color recognition unit that recognizes the color of the driver's face based on the visible light image; and a face recognition unit that performs the recognition of the driver's face based on the color of the driver's face and the infrared image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a driver monitoring system. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2006-095008 is known as a technical document related to a driver monitoring system. This publication describes a gaze vector detection system that includes an infrared light source that illuminates the eyes or face, a camera that photographs the eyes or face, and a computer that processes the photographed image data of the camera to calculate the gaze vector, and detects the true bright point and the center of the pupil from the photographed image data to calculate the gaze vector. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-095008 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is known that the reflectance of infrared rays changes depending on the color of the driver's face. Also, the reflectance of infrared rays changes depending on makeup such as eye shadow. As a result, there is a problem that the accuracy of driver's face recognition using infrared images may be reduced. [Means for solving the problem]

[0005] One aspect of the present invention is a driver monitoring system that performs facial recognition of a vehicle driver from a facial image of the driver, and includes an infrared image capture unit that captures an infrared image of the driver's face using an infrared emitter and an infrared receiver, a visible light image capture unit that captures a visible light image of the driver's face using a visible light receiver, a color recognition unit that recognizes the color of the driver's face based on the visible light image, and a facial recognition unit that performs facial recognition of the driver based on the color of the driver's face and the infrared image.

[0006] According to a driver monitoring system of one embodiment of the present invention, the driver's face is recognized from the color of the driver's face recognized based on a visible light image and an infrared image. Therefore, by combining an infrared image, which is less affected by sunlight and interior lighting, with a visible light image, which provides specific color information, the accuracy of driver's face recognition can be improved.

[0007] In a driver monitoring system according to one aspect of the present invention, the color of the driver's face may include the color of the driver's eyes, and the face recognition unit may change the logic for recognizing the driver's face in the infrared image depending on the color of the driver's eyes. According to this driver monitoring system, a visible light image of the driver is captured to recognize the driver's eye color, and the logic for recognizing the driver's face using infrared images is changed based on the driver's eye color, thereby improving the accuracy of face recognition compared to when the driver's face is recognized from infrared images without taking into account the driver's eye color.

[0008] In a driver monitoring system according to one embodiment of the present invention, if the driver's eyes are black, facial recognition logic for black eyes may be used to recognize the driver's face from infrared images, and if the driver's eyes are blue, facial recognition logic for blue eyes may be used to recognize the driver's face from infrared images. According to this driver monitoring system, if the driver's eyes are black, the driver's face is recognized from the infrared image using face recognition logic for black eyes, and if the driver's eyes are blue, the driver's face is recognized from the infrared image using face recognition logic for blue eyes, thereby improving the accuracy of face recognition compared to using the same face recognition logic regardless of eye color.

[0009] In the driver monitoring system according to one aspect of the present invention, the visible light image capturing section may capture a visible light image of the driver when the driver sits in the driver's seat of the vehicle or when the vehicle engine is started. According to this driver monitoring system, the driver's eye color can be appropriately recognized even at night by capturing a visible light image of the driver when he or she is seated or when the engine is started.

[0010] In a driver monitoring system according to one embodiment of the present invention, the face recognition unit may perform face recognition of the driver from a facial image of the driver using a face recognition model that is a machine learning model, select one face recognition model from a plurality of face recognition models that are pre-prepared according to the driver's eye color, and perform face recognition of the driver in an infrared image using the selected face recognition model. According to this driver monitoring system, a visible light image of the driver is captured to recognize the driver's eye color, and a face recognition model of the driver using infrared images is changed based on the driver's eye color, thereby improving the accuracy of face recognition compared to when the driver's face is recognized from infrared images without taking the driver's eye color into consideration.

[0011] In a driver monitoring system according to one aspect of the present invention, the system further includes an opening / closing determination unit that determines whether the driver's eyes are open or closed based on the skin color around the driver's eyes, which is included in the driver's facial color, and the infrared image, and the opening / closing determination unit may change the opening / closing determination logic for the driver's eyes in the infrared image depending on the skin color around the driver's eyes. According to this driver monitoring system, by capturing a visible light image of the driver and recognizing the skin color around the driver's eyes, the logic for determining whether the driver's eyes are open or closed using an infrared image can be changed based on the skin color around the driver's eyes. This improves the accuracy of the open / closed eye determination compared to when the driver's eyes are determined to be open or closed from an infrared image without taking into account the skin color around the driver's eyes, which may be due to makeup that changes the light reflectance.

[0012] In a driver monitoring system according to one aspect of the present invention, the color of the driver's face includes the color of the driver's eyes, and the open / closed determination unit may change the open / closed determination logic for the driver's eyes in the infrared image depending on the color of the driver's eyes and the color of the skin around the driver's eyes. According to this driver monitoring system, by recognizing the driver's eye color in addition to the skin color around the driver's eyes, the opening / closing judgment logic can be changed taking into account the difference between the driver's eye color and the skin color around the eyes, thereby further improving the accuracy of the opening / closing judgment.

[0013] In a driver monitoring system according to one embodiment of the present invention, the open / close judgment unit may use an open / close judgment model, which is a machine learning model, to determine whether the driver's eyes are open or closed from the driver's facial image, select one open / close judgment model from a plurality of open / close judgment models pre-prepared according to the skin color around the driver's eyes, and use the selected open / close judgment model to determine whether the driver's eyes are open or closed in the infrared image. According to this driver monitoring system, by capturing a visible light image of the driver and recognizing the skin color around the driver's eyes, a model for determining whether the driver's eyes are open or closed using an infrared image can be selected taking into account the skin color around the driver's eyes. This improves the accuracy of the open / closed eye determination compared to when the driver's eyes are determined to be open or closed from an infrared image without taking into account the skin color around the driver's eyes, which may be due to makeup that changes the light reflectance.

[0014] In one aspect of the present invention, the driver monitoring system may further include an opening degree calculation unit that calculates the degree of mouth opening of the driver based on the color of the driver's face including the color of the driver's lips, and the opening degree calculation unit may extract feature points of the driver's mouth from the infrared image, correct the feature points of the mouth based on the boundary between the color of the driver's lips and the color of the driver's face's skin from the visible light image, and calculate the degree of mouth opening of the driver based on the corrected feature points of the mouth. This driver monitoring system extracts feature points of the driver's mouth from an infrared image, corrects the feature points of the mouth based on the boundary between the color of the driver's lips and the color of the driver's facial skin in a visible light image, and calculates the degree of mouth opening of the driver based on the corrected feature points of the mouth. This improves the accuracy of calculating the degree of mouth opening compared to calculating the degree of mouth opening of the driver from an infrared image without taking into account the color of the driver's lips.

[0015] In a driver monitoring system according to one aspect of the present invention, the mouth opening calculation unit determines whether the color of the driver's lips is within a non-makeup range estimated from the color of the skin on the face, and calculates the mouth opening degree if the color of the driver's lips is within the non-makeup range, and may not calculate the mouth opening degree if the color of the driver's lips is not within the non-makeup range. According to this driver monitoring system, if the color of the driver's lips is not within the no-makeup range estimated from the color of the skin on their face, it is believed that the accuracy of detecting the driver's mouth has been reduced due to the influence of lipstick, etc., so by not calculating the mouth opening degree, it is possible to avoid erroneous calculation of the mouth opening degree. Effect of the Invention

[0016] According to the driver monitoring system according to each aspect of the present invention, it is possible to improve the accuracy of facial recognition of a vehicle driver. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a driver monitoring system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of an arrangement of a driver camera monitor. [Diagram 3] 11 is a flowchart illustrating an example of a face recognition process. [Figure 4] FIG. 11 is a block diagram showing a driver monitoring system according to a second embodiment. [Diagram 5] 13 is a flowchart illustrating an example of an open / close determination process. [Figure 6] FIG. 11 is a block diagram showing a driver monitoring system according to a third embodiment. [Figure 7] 1A is a diagram showing an example in which erroneous detection of feature points occurs due to uneven lighting on the driver's face, and FIG. 1B is a diagram showing another example in which erroneous detection of feature points occurs due to uneven lighting on the driver's face. [Figure 8] 13 is a flowchart illustrating an example of an opening degree calculation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] [First embodiment] Fig. 1 is a block diagram showing a driver monitoring system 100 according to a first embodiment. The driver monitoring system 100 shown in Fig. 1 is mounted on a vehicle such as a passenger car or a freight vehicle, and monitors the state of the driver of the vehicle. The driver monitoring system 100 performs face recognition of the driver based on a face image of the driver captured by a driver monitor camera 1 provided in the vehicle. The driver monitoring system 100 may constitute a part of an automatic driving system or an advanced driving assistance system.

[0020] The driver monitoring system 100 takes into account the driver's eye color in the face recognition of the driver. It has been found that the light reflectance changes depending on the driver's eye color, which changes the infrared light reception results described below. For this reason, the driver monitoring system 100 improves the recognition accuracy of face recognition such as the driver's line of sight angle by changing the face recognition logic in consideration of the driver's eye color. This makes it possible to improve the accuracy of the driver's inattentiveness judgment.

[0021] The driver monitoring system 100 has a face recognition ECU 10 that manages the system. The face recognition ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit. The storage unit is composed of, for example, a ROM [Read Only Memory], a RAM [Random Access Memory], and an EEPROM [Electrically Erasable Programmable Read-Only Memory]. The face recognition ECU 10 executes various functions by having the CPU execute programs stored in the storage unit. The face recognition ECU 10 may be composed of multiple electronic units.

[0022] The face recognition ECU 10 is connected to a driver monitor camera 1 and an HMI [Human Machine Interface] 2 .

[0023] The driver monitor camera 1 is an imaging device for capturing an image of the face of a driver of a vehicle. The driver monitor camera 1 is provided, for example, in a position in front of the driver inside the vehicle. Here, FIG. 2 is a diagram for explaining an example of the arrangement of the driver monitor camera 1. FIG. 2 shows a vehicle 50, a steering wheel 51, a steering column 52, and the driver monitor camera 1. As shown in FIG. 2, the driver monitor camera 1 is provided, for example, on the steering column 52, and captures an image of the head including the driver's face at a predetermined frame rate.

[0024] 2, the projection range of the driver monitor camera 1 is indicated by a dashed line. The driver monitor camera 1 may be provided on the steering wheel 51 of the vehicle 50, a room mirror, a meter panel, a meter hood, or the like.

[0025] Next, the configuration of the driver monitor camera 1 will be described. As shown in Fig. 1, the driver monitor camera 1 has an infrared light emitting unit 3, an infrared light receiving unit 4, and a visible light receiving unit 5. The infrared light emitting unit 3 is a light emitting element that generates infrared light. For example, an infrared LED [Light Emitting Diode] or an infrared laser diode can be used as the infrared light emitting unit 3. The infrared light is used to illuminate the driver's face.

[0026] The infrared receiving unit 4 is a light receiving element that receives infrared rays emitted from the infrared emitting unit 3 and reflected by the driver. An infrared sensor, an infrared camera, or the like can be used as the infrared receiving unit 4. The infrared emitting unit 3 and the infrared receiving unit 4 enable the driver monitor camera 1 to clearly capture the state of the driver and obtain an infrared image even at night or in a dark environment.

[0027] The visible light receiving unit 5 is a light receiving element that receives visible light. For example, a CCD [Charge Coupled Device] or a CIS [CMOS Image Sensor] can be used as the visible light receiving unit 5. The visible light receiving unit 5 receives sunlight reflected on the driver's face or light from lighting outside or inside the vehicle to obtain a visible light image of the driver's face.

[0028] The HMI 2 is an interface for inputting and outputting information between the face recognition ECU 10 and the driver. The HMI 2 includes, for example, a display and a speaker provided in the vehicle cabin. The HMI 2 outputs images from the display and sounds from the speaker in response to a control signal from the face recognition ECU 10. The display may be a MID [Multi Information Display] or a HUD [Head Up Display]. The HMI 2 may include various indicators.

[0029] Next, a description will be given of the functional configuration of the face recognition ECU 10. As shown in Fig. 1, the face recognition ECU 10 has an infrared image capturing unit 11, a visible light image capturing unit 12, a color recognition unit 13, and a face recognition unit .

[0030] The infrared image capturing unit 11 captures an infrared image of the driver's face using the infrared light emitting unit 3 and the infrared light receiving unit 4 of the driver monitor camera 1. The infrared image is an image captured using infrared rays. The infrared image is not affected by the light conditions (brightness or darkness), and can capture a clear image even at night or in a dark place.

[0031] The visible light image capturing unit 12 captures a visible light image of the driver's face using the visible light receiving unit 5 of the driver monitor camera 1. The visible light image is an image captured using visible light. The visible light image contains color information (such as RGB information). The wavelength range of the visible light image is not particularly limited, but may be an embodiment in which only a part of the wavelength range is used.

[0032] The visible light image capturing unit 12 may capture a visible light image for each frame if the situation allows capturing an image, or may capture a visible light image of the driver when the driver sits in the driver's seat of the vehicle 50 or when the vehicle engine is started. The determination of whether the driver is seated may be made based on the detection result of the seat belt fastening by a seat belt sensor in the driver's seat, or may be made based on the detection result of a seating sensor (such as a pressure sensor) in the driver's seat. Even at night or in a situation where sunlight is not illuminating the driver inside a building, the visible light image capturing unit 12 can capture a visible light image of the driver's face by the visible light receiving unit 5 because it is considered that the interior light of the vehicle 50 is on when the driver sits in the driver's seat or when the vehicle engine is started.

[0033] The color recognition unit 13 recognizes the color of the driver's face based on the visible light image. The color of the driver's face may include the color of each part of the driver's face, such as the driver's eyes, nose, mouth, etc., and may also include the color around each part. The color recognition unit 13 recognizes pixel areas constituting each part of the driver's face, such as the driver's eyes, nose, mouth, etc., from the visible light image, for example. The color recognition unit 13 recognizes the color of each part from color information (such as chromaticity parameters) of the pixel area of ​​each part.

[0034] In addition, the color recognition unit 13 may use the past recognition result of the driver's eye color when the vehicle 50 is equipped with a personal authentication function for the driver. Eye color can be considered to be basically unchanging.

[0035] The face recognition unit 14 recognizes the face of the driver based on the color of the driver's face recognized by the color recognition unit 13 and the infrared image captured by the infrared image capturing unit 11. The face recognition unit 14 recognizes the face of the driver by a face recognition logic provided in advance. The face recognition unit 14 recognizes a face area in an infrared image, for example, from a face pattern (for example, the arrangement of eyes, nose, and mouth). For example, Haar-like features and HOG [Histogram of Oriented Gradients] features can be used to recognize the face area. The face recognition unit 14 extracts features that characterize the face from the face area. The face features include, for example, the shape of the face, the positions of the eyes, nose, and mouth, and the like. For extracting the features, methods such as PCA [Principal Component Analysis] and LBP [Local Binary Pattern] can be used. The face recognition unit 14 recognizes the driver's face direction, face position, gaze angle, and the like from the features. The face recognition logic is not limited to the above-mentioned contents, and any well-known logic can be adopted.

[0036] The face recognition unit 14 may be capable of changing the face recognition logic of the driver according to the eye color of the driver recognized by the color recognition unit 13. That is, the face recognition unit 14 may be capable of using a plurality of types of face recognition logic. Specifically, the face recognition unit 14 may be capable of using at least two of a normal face recognition logic, a black-eye face recognition logic, a blue-eye face recognition logic, a brown-eye face recognition logic, a green-eye face recognition logic, a gray-eye face recognition logic, and an amber-eye face recognition logic.

[0037] The black-eyed face recognition logic is a face recognition logic set to increase the accuracy of face recognition on the assumption that the driver has black eyes. In the black-eyed face recognition logic, the detection parameters (such as the threshold value in edge processing) are adjusted so that the detection accuracy of the gaze of black eyes is higher than in the normal face recognition logic, for example. The same is true for the blue-eyed face recognition logic, the brown-eyed face recognition logic, the green-eyed face recognition logic, the gray-eyed face recognition logic, and the amber-eyed face recognition logic. The normal face recognition logic is a standard face recognition logic that does not perform parameter adjustment according to eye color. The normal face recognition logic can be, for example, a well-known face recognition logic that has been used conventionally.

[0038] When the eye color of the driver recognized by the color recognition unit 13 is black, the face recognition unit 14 performs face recognition of the driver using face recognition logic for black eyes. When the eye color of the driver recognized by the color recognition unit 13 is blue, the face recognition unit 14 performs face recognition of the driver using face recognition logic for blue eyes. The same applies to other eye colors.

[0039] The face recognition unit 14 may use a face recognition model 14a as face recognition logic. The face recognition model 14a is, for example, a neural network such as a convolutional neural network (CNN). The neural network may include multiple layers including multiple convolution layers and a pooling layer. A deep learning network using deep learning is used as the neural network. A recurrent neural network (RNN) may be used as the face recognition model 14a.

[0040] The face recognition model 14a receives, for example, an infrared image captured by the infrared image capturing unit 11, and outputs the result of face recognition of the driver. The face recognition model 14a may further receive a visible light image or a result of recognition of the color of the driver's face by the color recognition unit 13 as an input.

[0041] The face recognition unit 14 may select one face recognition model from a plurality of types of face recognition models 14a provided in advance according to the eye color of the driver recognized by the color recognition unit 13, and may use the selected face recognition model to recognize the face of the driver. That is, the face recognition unit 14 may have a plurality of types of face recognition models 14a. For example, the face recognition unit 14 has at least two of a normal face recognition model, a black eye face recognition model, a blue eye face recognition model, a brown eye face recognition model, a green eye face recognition model, a gray eye face recognition model, and an amber eye face recognition model as the face recognition model 14a provided in advance.

[0042] The black-eyed face recognition model is a machine learning model trained by including facial images of black-eyed drivers in a dataset. The black-eyed face recognition model has improved accuracy in recognizing black-eyed drivers' faces compared to the normal face recognition model. The same is true for the blue-eyed face recognition model, brown-eyed face recognition model, green-eyed face recognition model, gray-eyed face recognition model, and amber-eyed face recognition model.

[0043] The normal face recognition model is a machine learning model that is trained without intentional bias of the data set due to eye color with the intention of versatility. Note that, "without intentional bias of the data set" does not mean that there is no bias of the data set. Various models may be adopted depending on the country or region. Various models may be downloadable from a server that can communicate with the vehicle 50.

[0044] When the eye color of the driver recognized by the color recognition unit 13 is black, the face recognition unit 14 performs face recognition using a face recognition model for black eyes. When the eye color of the driver recognized by the color recognition unit 13 is blue, the face recognition unit 14 performs face recognition using a face recognition model for blue eyes. The same applies to other eye colors.

[0045] The face recognition unit 14 uses a face recognition model 14a corresponding to the eye color of the driver to perform face recognition of the driver's face position, face direction, line of sight, etc. The face recognition unit 14 may determine whether the driver is looking away based on the result of the face recognition of the driver. If the face recognition unit 14 determines that the driver is looking away, it may alert the driver by displaying an image or outputting a sound via the HMI 2.

[0046] Next, the processing of the driver monitoring system 100 will be described with reference to Fig. 3. Fig. 3 is a flowchart for explaining an example of face recognition processing. The face recognition processing is executed, for example, when the driver monitoring function of the vehicle 50 is turned on.

[0047] 3, in S10, the face recognition ECU 10 of the driver monitoring system 100 captures an infrared image of the driver's face using the infrared image capturing unit 11. After that, the face recognition ECU 10 proceeds to S11.

[0048] In S11, the face recognition ECU 10 captures a visible light image of the driver's face using the visible light image capturing unit 12. The face recognition ECU 10 also recognizes the color of the driver's face (including the color of the eyes) using the color recognition unit 13. After that, the face recognition ECU 10 proceeds to S12. Note that the order of S10 and S11 may be reversed, or they may be performed simultaneously.

[0049] In S12, the face recognition ECU 10 judges whether the driver's eye color is black. If the face recognition ECU 10 judges that the driver's eye color is black (S12: YES), the process proceeds to S13. If the face recognition ECU 10 does not judge that the driver's eye color is black (S12: NO), the process proceeds to S14.

[0050] In S13, the face recognition ECU 10 executes face recognition of the driver by the face recognition logic for black eyes using the face recognition unit 14. The face recognition unit 14 may use a face recognition model for black eyes as the face recognition logic for black eyes. The face recognition unit 14 receives the face color of the driver recognized by the color recognition unit 13 and the infrared image captured by the infrared image capturing unit 11 as inputs, and outputs the result of face recognition of the driver from the face recognition model for black eyes. Then, the face recognition process ends.

[0051] In S14, the face recognition ECU 10 judges whether the driver's eye color is blue. If the face recognition ECU 10 judges that the driver's eye color is blue (S14: YES), the process proceeds to S15. If the face recognition ECU 10 does not judge that the driver's eye color is blue (S14: NO), the process proceeds to S16.

[0052] In S15, the face recognition ECU 10 executes face recognition of the driver using face recognition logic for blue eyes by the face recognition unit 14. The face recognition unit 14 may use a face recognition model for blue eyes as the face recognition logic for blue eyes. The face recognition unit 14 receives the face color of the driver recognized by the color recognition unit 13 and the infrared image captured by the infrared image capturing unit 11 as inputs, and outputs the result of face recognition of the driver from the face recognition model for blue eyes. Thereafter, the face recognition process ends.

[0053] In S16, the face recognition ECU 10 judges whether the driver's eye color is brown. If the face recognition ECU 10 judges that the driver's eye color is brown (S16: YES), the process proceeds to S17. If the face recognition ECU 10 does not judge that the driver's eye color is brown (S16: NO), the process proceeds to S18.

[0054] In S17, the face recognition ECU 10 executes face recognition of the driver by the face recognition logic for brown eyes using the face recognition unit 14. The face recognition unit 14 may use a face recognition model for brown eyes as the face recognition logic for brown eyes. The face recognition unit 14 receives the face color of the driver recognized by the color recognition unit 13 and the infrared image captured by the infrared image capturing unit 11 as inputs, and outputs the result of face recognition of the driver from the face recognition model for brown eyes. Then, the face recognition process ends.

[0055] In S18, the face recognition ECU 10 executes face recognition of the driver by the face recognition unit 14 using the normal face recognition logic. The face recognition unit 14 may use a normal face recognition model as the normal face recognition logic. The face recognition unit 14 receives the face color of the driver recognized by the color recognition unit 13 and the infrared image captured by the infrared image capturing unit 11 as inputs, and outputs the result of face recognition of the driver from the normal face recognition model. Thereafter, the face recognition process ends.

[0056] According to the driver monitoring system 100 according to the first embodiment described above, a visible light image of the driver is captured to recognize the color of the driver's eyes, and the logic (face recognition model) for recognizing the driver's face using an infrared image is changed based on the color of the driver's eyes. This improves the accuracy of face recognition compared to when the driver's face is recognized from an infrared image without considering the color of the driver's eyes. In addition, the driver monitoring system 100 recognizes the driver's face using a hybrid type of visible light image and infrared image passively acquired by the visible light receiving unit 5 of the driver monitor camera 1. As a result, the driver monitoring system 100 can prevent the driver from feeling dazzled by light entering his or her eyes, unlike a configuration in which a highly accurate visible light image is acquired by irradiating the driver's face with visible light.

[0057] Furthermore, according to the driver monitoring system 100, if the driver's eyes are black, the driver's face is recognized from the infrared image using face recognition logic for black eyes, and if the driver's eyes are blue, the driver's face is recognized from the infrared image using face recognition logic for blue eyes, thereby improving the accuracy of face recognition compared to using the same face recognition logic regardless of eye color.

[0058] Furthermore, according to the driver monitoring system 100, a visible light image of the driver is captured to recognize the driver's eye color, and the driver's face recognition model using an infrared image is changed based on the driver's eye color, thereby improving the accuracy of face recognition compared to when the driver's face is recognized from an infrared image without taking the driver's eye color into consideration.

[0059] Furthermore, the driver monitoring system 100 can appropriately recognize the eye color of the driver even at night by capturing a visible light image of the driver when he or she sits down or when the engine is started.

[0060] [Second embodiment] Next, a driver monitoring system 101 according to a second embodiment will be described with reference to the drawings. The same or corresponding components as those in the first embodiment are denoted by the same reference numerals, and duplicated descriptions will be omitted. Fig. 4 is a block diagram showing the driver monitoring system 101 according to the second embodiment.

[0061] A driver monitoring system 101 according to the second embodiment shown in Fig. 4 performs face recognition of the driver by determining whether the driver's eyes are open or closed. As shown in Fig. 4, the face recognition ECU 20 of the driver monitoring system 101 is different from that of the first embodiment in that it has an open / closed eye determination unit 21.

[0062] The driver monitoring system 101 considers the skin color around the driver's eyes when determining whether the driver's eyes are open or closed. It has been found that the light reflectance of the skin and eyes changes depending on the skin color and makeup around the driver's eyes, and the reflection luminance of the curved vertices changes, causing a decrease in the accuracy of the open / closed determination. For this reason, the driver monitoring system 101 suppresses the decrease in accuracy of the open / closed determination by determining whether the driver's eyes are open or closed while considering the skin color around the driver's eyes.

[0063] The open / closed determination unit 21 determines whether the driver's eyes are open or closed based on the skin color (including the color of makeup on the skin) around the driver's eyes included in the driver's facial color recognized by the color recognition unit 13 and the infrared image of the infrared image capture unit 11. The skin color around the driver's eyes can be, for example, the skin color of the driver's facial skin within a certain distance from the center point on the image of the driver's eyes on the visible light image. Note that the skin color around the driver's eyes does not necessarily have to be the skin color within a certain distance from the eyes, and may be the skin color within a predetermined shape including the eyes.

[0064] The open / close determination unit 21 performs an open / close determination of the driver's eyes using a previously provided open / close determination logic. The open / close determination unit 21 detects an eye area including the driver's eyes recognized by the face recognition unit 14, and detects the positions of the upper eyelid and the lower eyelid from the eye area. The positions of the upper eyelid and the lower eyelid are determined, for example, by detecting the edges (boundaries) of the eyelids. Thereafter, the open / close determination unit 21 measures the distance between the positions of the upper eyelid and the lower eyelid, and determines that the eyes are closed if the distance between the positions of the upper eyelid and the lower eyelid is less than a certain threshold. The open / close determination unit 21 may warn that the driver may be asleep if the state in which the distance between the positions of the upper eyelid and the lower eyelid is less than a certain threshold continues for a certain period of time or more. The open / close determination logic is not limited to the above-mentioned content, and any well-known logic may be adopted.

[0065] The open / close determination unit 21 may be capable of changing the open / close determination logic of the driver's eyes in the infrared image according to the skin color around the driver's eyes. That is, the open / close determination unit 21 may be capable of using a plurality of types of open / close determination logic. Specifically, the open / close determination unit 21 may be capable of using at least two of a normal open / close determination logic, an eyelid open / close determination logic, and a makeup open / close determination logic.

[0066] The eyelid opening / closing determination logic is set to increase the accuracy of eye opening / closing determination on the assumption that the driver does not wear heavy makeup around the eyes. In the eyelid opening / closing determination logic, parameters (such as a threshold value in edge processing) are adjusted so that the detection accuracy of the position of the skin-colored eyelids is increased compared to the normal opening / closing determination logic, for example.

[0067] The makeup open / close judgment logic is an open / close judgment logic set to suppress a decrease in accuracy of eye open / close judgment on the assumption that the driver is wearing makeup around the eyes. In the makeup open / close judgment logic, parameters are adjusted so that erroneous detection of a line formed by makeup such as eye shadow as the position of the eyelid is less likely to occur compared to the normal open / close judgment logic. The normal open / close judgment logic is a standard open / close judgment logic that does not perform parameter adjustment taking into account the presence or absence of makeup. The normal open / close judgment logic can be, for example, a well-known open / close judgment logic that has been used conventionally.

[0068] When the open / close determination unit 21 determines that the driver is not wearing makeup around the driver's eyes based on, for example, the skin color around the driver's eyes, it performs an open / close determination of the driver's eyes using an open / close determination logic for eyelids. The no-makeup state is a state in which heavy makeup such as eyeshadow is not applied, which may lead to erroneous detection of the eyelid position. When the open / close determination unit 21 determines that the driver is wearing makeup around the driver's eyes based on the skin color around the driver's eyes, it may perform an open / close determination of the driver's eyes using an open / close determination logic for makeup.

[0069] The open / close determination unit 21 may further select the open / close determination logic based on the eye color of the driver. For example, the eyelid open / close determination logic may further include at least two of a black eye logic, a blue eye logic, a brown eye logic, a green eye logic, a gray eye logic, and an amber eye logic.

[0070] The black eye logic is an open / closed judgment logic that is set to increase the accuracy of face recognition on the assumption that the driver has black eyes among the eyelid open / closed judgment logics. In the black eye logic, the detection parameters (such as the threshold value in edge processing) are adjusted so that the detection accuracy of the position of the black eyelid is high. The same is true for the blue eye logic, brown eye logic, green eye logic, gray eye logic, and amber eye logic.

[0071] Similarly, the makeup open / close determination logic may further include at least two of a black eye logic, a blue eye logic, a brown eye logic, a green eye logic, a gray eye logic, and an amber eye logic. By selecting the open / close determination logic in this way while taking into account the driver's eye color, the accuracy of the open / close determination of the driver's eyes can be improved.

[0072] The open / close determination unit 21 may use an open / close determination model 21a as an open / close determination logic. The open / close determination model 21a may be a neural network similar to the face recognition model 14a described above. Note that the neural network configurations of the open / close determination model 21a and the face recognition model 14a may be the same or different.

[0073] The open / closed determination model 21a receives, for example, an infrared image captured by the infrared image capturing unit 11, and outputs a determination result of whether the driver's eyes are open or closed. The open / closed determination model 21a may further receive, as an input, a visible light image captured by the visible light image capturing unit 12 or a recognition result of the color of the driver's face by the color recognition unit 13. The open / closed determination model 21a may be configured to output the distance between the driver's upper eyelid and lower eyelid or the position of the eyelid, instead of the determination result of whether the driver's eyes are open or closed.

[0074] The open / close determination unit 21 may select one open / close determination model from among a plurality of types of open / close determination models 21a provided in advance based on the skin color around the driver's eyes, and may perform open / close determination of the driver's eyes using the selected open / close determination model. That is, the open / close determination unit 21 may have a plurality of types of open / close determination models 21a (open / close determination logic). The open / close determination unit 21 has at least two of a normal open / close determination model, an eyelid open / close determination model, and a makeup open / close determination model provided in advance as the open / close determination model 21a, for example.

[0075] The eyelid open / close judgment model is a machine learning model trained by including in a data set mainly face images of drivers who do not wear heavy makeup around the eyes. The eyelid open / close judgment model has improved accuracy in determining whether the eyes of a driver who is not wearing makeup, compared to the normal open / close judgment model. The makeup open / close judgment model is a machine learning model trained by including in a data set mainly face images of drivers who wear makeup such as eyeshadow around the eyes. The makeup open / close judgment model has improved accuracy in determining whether the eyes of a driver who is wearing makeup, compared to the normal open / close judgment model. The normal open / close judgment model is a machine learning model trained by using a data set prepared regardless of whether makeup is worn for versatility.

[0076] When the open / close determination unit 21 determines that the driver is not wearing makeup around the driver's eyes based on, for example, the skin color around the driver's eyes, it performs an open / close determination of the driver's eyes using an open / close determination model for eyelids. Also, when the open / close determination unit 21 determines that the driver is wearing makeup around the driver's eyes based on the skin color around the driver's eyes, it may perform an open / close determination of the driver's eyes using an open / close determination model for makeup.

[0077] The open / close determination unit 21 may select one of a plurality of types of eyelid open / close determination models based on the eye color of the driver. For example, the eyelid open / close determination models may further include at least two of a black eye model, a blue eye model, a brown eye model, a green eye model, a gray eye model, and an amber eye model.

[0078] The black eye model is an open / closed judgment model that is set to improve the accuracy of face recognition on the assumption that the driver has black eyes among the eyelid open / closed judgment models. The black eye model is trained to improve the accuracy of detecting the position of the black eyelid. The same is true for the blue eye model, brown eye model, green eye model, gray eye model, and amber eye model.

[0079] Similarly, the makeup open / close determination model may further include at least two of a black eye model, a blue eye model, a brown eye model, a green eye model, a gray eye model, and an amber eye model. By selecting an open / close determination model in this manner while taking into account the driver's eye color, the accuracy of determining whether the driver's eyes are open or closed can be improved.

[0080] Based on the result of the eye opening / closing determination, for example, when the driver's eyes are closed for a certain period of time or more while the vehicle 50 is running, the open / closed determination unit 21 executes a sound output to alert the driver via the HMI 2. Note that the alert is not essential.

[0081] Next, the processing of the driver monitoring system 101 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of an open / close determination processing. The open / close determination processing is executed, for example, when the driver monitoring function of the vehicle 50 is turned on.

[0082] 5, in S20, the face recognition ECU 20 of the driver monitoring system 101 captures an infrared image of the driver's face using the infrared image capturing unit 11. After that, the face recognition ECU 20 proceeds to S21.

[0083] In S21, the face recognition ECU 20 captures a visible light image of the driver's face using the visible light image capturing unit 12. After that, the face recognition ECU 20 proceeds to S22. Note that S20 and S21 may be performed in the reverse order, or simultaneously.

[0084] In S22, the face recognition ECU 20 recognizes the eye color of the driver and the color around the driver's eyes by the color recognition unit 13. Note that recognition of the eye color of the driver is not essential. After that, the face recognition ECU 20 proceeds to S23.

[0085] In S23, the face recognition ECU 20 judges whether or not the driver is not wearing makeup around the eyes by the open / close judgment unit 21. If the face recognition ECU 20 judges that the driver is not wearing makeup around the eyes (S23: YES), the process proceeds to S24. If the face recognition ECU 20 does not judge that the driver is not wearing makeup around the eyes (S23: NO), the process proceeds to S25.

[0086] In S24, the face recognition ECU 20 determines whether the driver's eyes are open or closed using the eyelid open / closed judgment logic by the open / closed judgment unit 21. The open / closed judgment unit 21 may use an eyelid open / closed judgment model as the eyelid open / closed judgment logic. The open / closed judgment unit 21 receives, for example, the color of the driver's face recognized by the color recognition unit 13 (including the color of the eyes and the color around the eyes) and the infrared image captured by the infrared image capturing unit 11 as input, and outputs the result of the judgment of whether the driver's eyes are open or closed from the eyelid open / closed judgment model. Thereafter, the open / closed judgment process ends.

[0087] In S25, the face recognition ECU 20 judges whether or not the driver has makeup around the eyes by the open / close judgment unit 21. If the face recognition ECU 20 judges that the driver has makeup around the eyes (S25: YES), the process proceeds to S26. If the face recognition ECU 20 does not judge that the driver has no makeup around the eyes (S25: NO), the process proceeds to S27.

[0088] In S26, the face recognition ECU 20 performs an open / closed determination of the driver's eyes using the open / closed determination logic for makeup by the open / closed determination unit 21. The open / closed determination unit 21 may use an open / closed determination model for makeup as the open / closed determination logic for makeup. The open / closed determination unit 21 receives, for example, the color of the driver's face and an infrared image, and outputs the result of the open / closed determination of the driver's eyes from the open / closed determination model for makeup. Thereafter, the open / closed determination process ends.

[0089] In S27, the face recognition ECU 20 performs an open / closed determination of the driver's eyes by the open / closed determination unit 21 using the normal open / closed determination logic. The open / closed determination unit 21 may use a normal open / closed determination model (open / closed determination model 21a) as the normal open / closed determination logic. The open / closed determination unit 21 receives, for example, the color of the driver's face and an infrared image, and outputs the result of the open / closed determination of the driver's eyes from the normal open / closed determination model. Thereafter, the open / closed determination process ends.

[0090] The face recognition ECU 20 may not perform the determination in S25 and may proceed to S26 if the result in S23 is NO. The face recognition ECU 20 may select the open / close determination logic or the open / close determination model based on the eye color of the driver, as in the first embodiment.

[0091] According to the driver monitoring system 101 of the second embodiment described above, by capturing a visible light image of the driver and recognizing the skin color around the driver's eyes, the logic for determining whether the driver's eyes are open or closed (open or closed judgment model) using an infrared image can be changed based on the skin color around the driver's eyes. This improves the accuracy of the open or closed eye judgment compared to when the open or closed eye judgment is performed from an infrared image without taking into account the skin color around the driver's eyes, which may be due to makeup that changes the light reflectance.

[0092] In addition, according to the driver monitoring system 101, by recognizing the driver's eye color in addition to the skin color around the driver's eyes, the opening / closing judgment logic can be changed taking into account the difference between the driver's eye color and the skin color around the eyes, thereby further improving the accuracy of the opening / closing judgment.

[0093] [Third embodiment] Next, a driver monitoring system 102 according to a third embodiment will be described with reference to the drawings. Fig. 6 is a block diagram showing the driver monitoring system 102 according to the third embodiment.

[0094] A driver monitoring system 102 according to the third embodiment shown in Fig. 6 calculates the mouth opening degree of the driver as a means of facial recognition of the driver. As shown in Fig. 6, the face recognition ECU 30 of the driver monitoring system 102 is different from that of the first embodiment in that it has an opening degree calculation unit 31.

[0095] The driver monitoring system 102 takes into consideration the recognition results of the skin color of the driver's face and the color of the driver's lips when calculating the degree of mouth opening of the driver. When there is an obstruction such as a plastic bottle in front of the driver's mouth as viewed from the driver monitor camera 1, when light is unevenly shining on the driver's mouth, when the driver's beard is thick and covers part of the mouth, etc., an erroneous calculation of the degree of mouth opening is performed. For this reason, the driver monitoring system 102 calculates the degree of mouth opening of the driver taking into consideration the recognition results of the skin color of the driver's face and the color of the driver's lips, thereby suppressing erroneous calculation of the degree of mouth opening.

[0096] In the third embodiment, the visible light image capturing unit 12 does not continue to use a visible light image of the driver that has been captured once, but recognizes the skin color of the driver's face and the color of the driver's lips using the latest visible light image captured for each frame.

[0097] The mouth opening calculation unit 31 calculates the mouth opening degree of the driver. The mouth opening degree is the degree of opening of the driver's mouth used for yawning judgment of the driver. The mouth opening calculation unit 31 calculates the mouth opening degree of the driver based on the color of the driver's face (including the color of the driver's lips) recognized by the color recognition unit 13 and the infrared image of the driver's face captured by the infrared image capturing unit 11. The mouth opening calculation unit 31 calculates the mouth opening degree with higher accuracy than when only the infrared image is used by utilizing the boundary between the part colored as the lips and the skin color part of the face. Note that the mouth opening calculation unit 31 may calculate the mouth opening degree only from the infrared image when a visible light image cannot be captured, such as when no light from outside enters the vehicle at night.

[0098] First, the mouth opening calculation unit 31 extracts the feature points of the driver's mouth from the infrared image. The mouth opening calculation unit 31 recognizes the mouth area by, for example, well-known image processing, and extracts the feature points of the mouth by edge detection or feature amount extraction processing. The mouth opening calculation unit 31 may extract the feature points of the mouth using a machine learning model.

[0099] The mouth opening calculation unit 31 detects the boundary between the color of the driver's lips (lip color recognition area) and the color of the skin of the face based on the recognition result of the color recognition unit 13. The color recognition unit 13 recognizes, for example, a part around the driver's mouth area that has a certain or greater difference in color from the skin of the driver's face area as the lips. If the difference in color from the skin of the face area is less than a certain amount, the color recognition unit 13 may expand the target range including the mouth area to recognize the lips.

[0100] The mouth opening calculation unit 31 judges whether the color of the driver's lips is within a non-makeup range estimated from the color of the driver's face. In other words, the mouth opening calculation unit 31 judges whether it is difficult to calculate the mouth opening degree of the driver. The state in which it is difficult to calculate the mouth opening degree of the driver is, for example, a state in which an obstruction such as a glasses frame or a plastic bottle is present in front of the driver's mouth, a state in which light is unevenly shining on the driver's mouth, a state in which the beard around the driver's mouth is thick, etc. The mouth opening calculation unit 31 makes the above judgment using, for example, correspondence table data that associates the skin color of the face with the range of the lip color estimated from the skin color of the face. The color can be determined as a numerical parameter such as RGB.

[0101] FIG. 7(a) is a diagram showing an example in which erroneous detection of feature points occurs due to uneven light hitting the driver's face. Feature points are shown as black dots. FIG. 7(b) is a diagram showing another example in which erroneous detection of feature points occurs due to uneven light hitting the driver's face. As shown in FIG. 7(a) and FIG. 7(b), when light hits the driver's mouth unevenly, erroneous detection occurs in feature points extracted from the infrared image. Whether or not light is hitting the driver unevenly enough to cause erroneous detection of feature points can be determined from the relationship between the color of the driver's face recognized from the visible light image and the color of the driver's lips.

[0102] When the color of the driver's lips is not within the no-makeup range estimated from the color of the driver's face, the mouth opening calculation unit 31 invalidates the feature points of the mouth and does not calculate the mouth opening degree because there is a high possibility that the situation will cause an erroneous detection of the feature points. When it is determined that the color of the driver's lips is within the no-makeup range estimated from the color of the driver's face, the mouth opening calculation unit 31 determines whether or not a certain number of feature points of the mouth are present within a certain distance from the lip color recognition area. The lip color recognition area is an area in which the color of the lips is recognized in the visible light image. The values ​​of the certain number and the certain distance are not particularly limited, but may be appropriate values ​​to ensure the reliability of the calculation of the mouth opening degree.

[0103] When a certain number or more of the feature points of the mouth are not present within a certain distance from the lip color recognition region, the mouth opening calculation unit 31 does not calculate the mouth opening degree because there is a high possibility that the feature points have been erroneously detected. On the other hand, when a certain number or more of the feature points of the mouth are present within a certain distance from the lip color recognition region, the mouth opening calculation unit 31 corrects the positions of the feature points so that they are included within the lip color recognition region based on the boundary between the driver's lip color recognition region and the skin color part of the face.

[0104] The mouth opening calculation unit 31 may correct the feature points using a mouth opening calculation model 31a, which is a machine learning model. The mouth opening calculation model 31a may be a neural network similar to the face recognition model 14a described above.

[0105] The neural network configurations of the mouth opening calculation model 31a and the face recognition model 14a may be the same or different. The mouth opening calculation model 31a receives, for example, the face and lip color recognition result by the color recognition unit 13 and the feature points of the mouth extracted from the infrared image, and outputs the corrected feature points of the mouth. The mouth opening calculation model 31a may receive, instead of the feature points of the mouth extracted from the infrared image, the infrared image itself and the color recognition result of the face and lips. The calculation result of the mouth opening is used, for example, to determine whether the driver is yawning.

[0106] Next, the processing of the driver monitoring system 102 will be described with reference to Fig. 5. Fig. 8 is a flowchart showing an example of the opening degree calculation processing. The opening degree calculation processing is executed, for example, when the driver monitoring function of the vehicle 50 is turned on.

[0107] 8, in S30, the face recognition ECU 30 of the driver monitoring system 102 captures an infrared image of the driver's face using the infrared image capturing unit 11. After that, the face recognition ECU 30 proceeds to S31.

[0108] In S31, the face recognition ECU 30 extracts feature points of the driver's face and mouth by the mouth opening calculation unit 31. The facial feature points may be extracted by the face recognition unit 14. After that, the face recognition ECU 30 proceeds to S32.

[0109] In S32, the face recognition ECU 30 captures a visible light image of the driver's face using the visible light image capturing unit 12. After that, the face recognition ECU 30 proceeds to S33.

[0110] In S33, the face recognition ECU 30 recognizes the face color (including the skin color) and the lip color of the driver by the color recognition unit 13. After that, the face recognition ECU 30 proceeds to S34.

[0111] In S34, the face recognition ECU 30 determines whether or not the color of the driver's lips is within the non-makeup range estimated from the color of the driver's face by the mouth opening calculation unit 31. If the face recognition ECU 30 determines that the color of the driver's lips is not within the non-makeup range estimated from the color of the driver's face (S34: NO), the process proceeds to S35. If the face recognition ECU 30 determines that the color of the driver's lips is within the non-makeup range estimated from the color of the driver's face (S34: YES), the process proceeds to S36.

[0112] In S35, the face recognition ECU 30 invalidates the feature points of the driver's mouth by the mouth opening calculation unit 31. After that, the face recognition ECU 30 ends the mouth opening calculation process without calculating the mouth opening degree.

[0113] In S36, the face recognition ECU 30 determines whether or not a certain number of mouth feature points exist within a certain distance from the lip color recognition region by the mouth opening calculation unit 31. If the face recognition ECU 30 does not determine that a certain number of mouth feature points exist within a certain distance from the lip color recognition region (S36: NO), the face recognition ECU 30 ends the mouth opening calculation process without calculating the mouth opening degree. If the face recognition ECU 30 determines that a certain number of mouth feature points exist within a certain distance from the lip color recognition region (S36: YES), the face recognition ECU 30 proceeds to S37.

[0114] In S37, the face recognition ECU 30 corrects the feature points of the mouth so that they are within the color recognition region of the lips, and calculates the mouth opening degree by the mouth opening degree calculation unit 31. The mouth opening degree calculation unit 31 may correct the feature points or calculate the mouth opening degree by using the mouth opening degree calculation model 31a.

[0115] According to the driver monitoring system 102 of the third embodiment described above, feature points of the driver's mouth are extracted from an infrared image, the feature points of the mouth are corrected based on the boundary between the color of the driver's lips and the color of the driver's facial skin in a visible light image, and the degree of mouth opening of the driver can be calculated based on the corrected feature points of the mouth. Therefore, it is possible to improve the calculation accuracy of the degree of mouth opening of the driver compared to the case where the degree of mouth opening of the driver is calculated from an infrared image without taking into account the color of the driver's lips.

[0116] Furthermore, according to the driver monitoring system 102, when the color of the driver's lips is not within the no-makeup range estimated from the color of the skin on the face, it is considered that the accuracy of detecting the driver's mouth has decreased due to the influence of lipstick or the like, so by not calculating the mouth opening degree, it is possible to avoid erroneous calculation of the mouth opening degree.

[0117] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

[0118] The face recognition unit 14 does not necessarily have to have the face recognition model 14a. The face recognition unit 14 may perform face recognition of the driver without using a machine learning model. The same applies to the opening / closing determination unit 21 and the opening degree calculation unit 31. [Explanation of symbols]

[0119] 1...driver monitor camera, 2...HMI, 3...infrared emitter, 4...infrared receiver, 5...visible light receiver, 10, 20, 30...face recognition ECU, 11...infrared image capture unit, 12...visible light image capture unit, 13...color recognition unit, 14...face recognition unit, 14a...face recognition model, 21...opening / closing judgment unit, 21a...opening / closing judgment model (opening / closing judgment logic), 31...opening degree calculation unit, 31a...opening degree calculation model, 50...vehicle, 100, 101, 102...driver monitoring system.

Claims

1. A driver monitoring system that performs face recognition of a vehicle driver from a facial image of the driver, an infrared image capturing unit that captures an infrared image of the driver's face using an infrared light emitting unit and an infrared light receiving unit; a visible light image capturing unit that captures a visible light image of the driver's face using a visible light receiving unit; a color recognition unit that recognizes a color of the driver's face based on the visible light image; a face recognition unit that performs face recognition of the driver based on the color of the driver's face and the infrared image; A driver monitoring system comprising:

2. the facial color of the driver includes the eye color of the driver; The driver monitoring system of claim 1 , wherein the facial recognition unit changes facial recognition logic of the driver in the infrared image depending on the driver's eye color.

3. 3. The driver monitoring system of claim 2, wherein if the driver's eye color is black, facial recognition of the driver is performed from the infrared image using a black-eye facial recognition logic, and if the driver's eye color is blue, facial recognition of the driver is performed from the infrared image using a blue-eye facial recognition logic.

4. The driver monitoring system according to claim 2 , wherein the visible light image capturing unit captures a visible light image of the driver when the driver sits in a driver's seat of the vehicle or when an engine of the vehicle is started.

5. The face recognition unit performs face recognition of the driver from a face image of the driver using a face recognition model that is a machine learning model, 3. The driver monitoring system of claim 2, further comprising: selecting one of a plurality of face recognition models preliminarily prepared according to the eye color of the driver; and performing face recognition of the driver in the infrared image using the selected face recognition model.

6. An eye opening / closing determination unit that determines whether the driver's eyes are open or closed based on a skin color around the driver's eyes, which is included in a facial color of the driver, and the infrared image, The driver monitoring system according to claim 1 , wherein the open / closed determination unit changes a logic for determining whether the driver's eyes in the infrared image are open or closed depending on a skin color around the driver's eyes.

7. the facial color of the driver includes the eye color of the driver; The driver monitoring system according to claim 6 , wherein the open / closed determination unit changes a logic for determining whether the driver's eyes in the infrared image are open or closed depending on the color of the driver's eyes and the color of the skin around the driver's eyes.

8. The opening / closing determination unit determines whether the driver's eyes are open or closed from a face image of the driver by using an opening / closing determination model that is a machine learning model, 7. The driver monitoring system according to claim 6, further comprising: a first open / close determination model selected from a plurality of open / close determination models that are preliminarily prepared according to a skin color around the driver's eyes; and a determination as to whether the driver's eyes are open or closed in the infrared image is made using the selected open / close determination model.

9. a mouth opening degree calculation unit that calculates a mouth opening degree of the driver based on a facial color of the driver including a lip color of the driver and the infrared image, 2. The driver monitoring system according to claim 1, wherein the mouth opening calculation unit extracts feature points of the driver's mouth from the infrared image, corrects the feature points of the mouth based on a boundary between a portion of the driver's lip color and a portion of the driver's face of a skin color from the visible light image, and calculates a mouth opening degree of the driver based on the corrected feature points of the mouth.

10. 10. The driver monitoring system according to claim 9, wherein the mouth opening calculation unit determines whether or not a color of the driver's lips is within a non-makeup range estimated from a skin color of the face, calculates the mouth opening degree when the color of the driver's lips is within the non-makeup range, and does not calculate the mouth opening degree when the color of the driver's lips is not within the non-makeup range.

Images