Drowsiness level prediction model generation device, drowsiness model generation system, and drowsiness determination device

Abstract

To provide a drowsiness level prediction model generation device that can reduce the amount of data and processing time and can generate a prediction model configured to highly accurately determine drowsiness.SOLUTION: A drowsiness level prediction model generation device of the present invention includes: a drowsiness level model construction unit configured to use face feature data that is time-series data of a plurality of face features extracted from an image of a face and a drowsiness level determination value for the face feature data as training data and construct, by machine learning, an initial drowsiness level prediction model using a neural network on which the drowsiness level determination value for the face feature data is an output; and a drowsiness level model optimization unit configured to generate the drowsiness level prediction model with improved determination accuracy for the drowsiness level determination value output by the initial drowsiness level prediction model in accordance with sparsification of the neural network of the initial drowsiness level prediction model or changing of a bias value.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a drowsiness level prediction model generation device, a drowsiness model generation system, and a drowsiness determination device.

Background Art

[0002] A driver driving a vehicle may sometimes feel drowsy. A drowsy driver tends to have reduced alertness and slow movements, which may lead to a serious accident.

[0003] To prevent such accidents, a drowsiness prevention system for drivers may be installed in vehicles in some cases. Such a drowsiness prevention system uses, for example, a drowsiness level estimation device using a trained model that estimates the driver's drowsiness level from a face image showing facial expressions, a so-called AI.

[0004] When attempting to construct a trained model that directly estimates a driver's drowsiness level from a face image, the amount of data to be processed is extremely large, and since it is necessary to store the process of expression changes as a transition map, the data volume of the transition map itself also becomes enormous, making the construction of the trained model difficult in terms of data volume and processing time. For this reason, a method of constructing a trained model using time-series data of facial feature points obtained by extracting characteristic parts of the face from a face image has been proposed (see Japanese Unexamined Patent Application Publication No. 2007-257043).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0006] However, in conventional sleepiness level estimation devices that use a predictive model built using the aforementioned time-series data of facial feature points, while data size and processing time can be reduced by compressing the data into time-series data of facial feature points, the accuracy of the prediction using this predictive model tends to decrease compared to when facial images are used directly.

[0007] The present invention has been made based on the circumstances described above, and aims to provide a sleepiness level prediction model generation device and sleepiness model generation system that can reduce the amount of data and shorten processing time, and generate a prediction model with high accuracy in determining sleepiness, as well as a sleepiness determination device that uses the prediction model generated by the sleepiness level prediction model generation device or the sleepiness model generation system. [Means for solving the problem]

[0008] A sleepiness level prediction model generation device according to one embodiment of the present invention is a sleepiness level prediction model generation device for determining a person's sleepiness level from a face image, comprising: a sleepiness level model construction unit that constructs an initial sleepiness level prediction model using a neural network that outputs sleepiness level determination values ​​for face feature data, using face feature data which is time-series data of multiple face feature points extracted from the face image and sleepiness level determination values ​​for the face feature data as training data; and a sleepiness level model optimization unit that generates a sleepiness level prediction model in which the determination accuracy of the sleepiness level determination values ​​output by the initial sleepiness level prediction model is improved by sparsifying the neural network of the initial sleepiness level prediction model or changing the bias value.

[0009] A sleepiness model generation system according to another embodiment of the present invention is a sleepiness model generation system for determining a person's sleepiness from a facial image, comprising: an image acquisition unit for acquiring a person's facial image; a facial feature data extraction unit for extracting facial feature data, which is time-series data of multiple facial feature points, from the facial image acquired by the image acquisition unit; a classification model generation unit for generating a sleepiness classification prediction model for classifying the person's sleepiness stage from the facial feature data; and a sleepiness model generation unit for generating a sleepiness level prediction model for determining the person's sleepiness level from the facial feature data, wherein the sleepiness stage includes a level determination required stage where a sleepiness level determination is necessary, and the classification model generation unit constructs an initial sleepiness stage prediction model using machine learning with the facial feature data and the sleepiness stage determination value for the facial feature data as training data, outputting a sleepiness stage determination value for the facial feature data. The system includes a sleepiness stage model construction unit and a sleepiness stage model optimization unit that generates a sleepiness stage prediction model in which the accuracy of the sleepiness stage judgment value output by the initial sleepiness stage prediction model is improved by sparsifying the neural network of the initial sleepiness stage prediction model or changing the bias value. The sleepiness model generation unit includes a sleepiness level model construction unit that uses facial feature data for which the sleepiness stage is a stage requiring level determination and the sleepiness level judgment value for that facial feature data as training data, and constructs an initial sleepiness level prediction model using machine learning with a neural network that outputs a sleepiness level judgment value for facial feature data. The system also includes a sleepiness level model optimization unit that generates a sleepiness level prediction model in which the accuracy of the sleepiness level judgment value output by the initial sleepiness level prediction model is improved by sparsifying the neural network of the initial sleepiness level prediction model or changing the bias value.

[0010] A sleepiness determination device according to yet another embodiment of the present invention comprises: an image acquisition unit that acquires a person's face image; a face feature data extraction unit that extracts face feature data, which is time-series data of multiple face feature points, from the face image acquired by the image acquisition unit; a classification determination unit that classifies the person's sleepiness stage from the face feature data using a sleepiness stage prediction model that has been machine-trained using known face feature data and sleepiness stage determination values ​​for that face feature data as training data; and a sleepiness determination unit that determines the person's sleepiness level from the face feature data using a sleepiness level prediction model that has been machine-trained using known face feature data and sleepiness level determination values ​​for that face feature data as training data. The sleepiness stage includes a level determination required stage where a sleepiness level determination is necessary, and the sleepiness level is determined by the sleepiness determination unit for face feature data where the classification determination unit has determined that the sleepiness stage is a level determination required stage. The sleepiness level prediction model used is a sleepiness level prediction model generated by the sleepiness level prediction model generation device or sleepiness model generation system of the present invention. [Effects of the Invention]

[0011] The sleepiness level prediction model generation device and sleepiness model generation system of the present invention can reduce the amount of data and shorten processing time, and can generate prediction models with high accuracy in determining sleepiness. Therefore, the sleepiness determination device of the present invention that uses the prediction model generated by the sleepiness level prediction model generation device or sleepiness model generation system has high accuracy in determining sleepiness. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a schematic diagram showing the configuration of a drowsiness detection device according to one embodiment of the present invention. [Figure 2] Figure 2 is an explanatory diagram showing the method for extracting facial feature data. [Figure 3] Figure 3 is a schematic diagram showing a sleepiness model generation system according to one embodiment of the present invention. [Figure 4] Figure 4 is a schematic diagram showing a sleepiness model generation system different from that shown in Figure 3. [Modes for carrying out the invention]

[0013] [Description of Embodiments of the Invention] First, embodiments of the present invention will be listed and described.

[0014] A sleepiness level prediction model generation device according to one embodiment of the present invention is a sleepiness level prediction model generation device for determining a person's sleepiness level from a face image, comprising: a sleepiness level model construction unit that constructs an initial sleepiness level prediction model using a neural network that outputs sleepiness level determination values ​​for face feature data, using face feature data which is time-series data of multiple face feature points extracted from the face image and sleepiness level determination values ​​for the face feature data as training data; and a sleepiness level model optimization unit that generates a sleepiness level prediction model in which the determination accuracy of the sleepiness level determination values ​​output by the initial sleepiness level prediction model is improved by sparsifying the neural network of the initial sleepiness level prediction model or changing the bias value.

[0015] This sleepiness level prediction model generator uses facial feature data, which is time-series data of multiple facial feature points extracted from facial images. Compared to using facial images directly, the amount of data is reduced and processing time is shortened. Furthermore, this sleepiness level prediction model generator can improve the accuracy of sleepiness level determination by sparsifying the neural network or changing the bias value.

[0016] It is preferable that the facial feature points described above are represented with reference points located in the center of the facial image as the starting point. By representing the facial feature points with reference points located in the center of the facial image as the starting point in this way, it becomes easier to capture the relative movement of the entire face from the facial feature points, thereby further improving the accuracy of the sleepiness level determination value.

[0017] A sleepiness model generation system according to another embodiment of the present invention is a sleepiness model generation system for determining a person's sleepiness from a facial image, comprising: an image acquisition unit for acquiring a person's facial image; a facial feature data extraction unit for extracting facial feature data, which is time-series data of multiple facial feature points, from the facial image acquired by the image acquisition unit; a classification model generation unit for generating a sleepiness classification prediction model for classifying the person's sleepiness stage from the facial feature data; and a sleepiness model generation unit for generating a sleepiness level prediction model for determining the person's sleepiness level from the facial feature data, wherein the sleepiness stage includes a level determination required stage where a sleepiness level determination is necessary, and the classification model generation unit constructs an initial sleepiness stage prediction model using machine learning with the facial feature data and the sleepiness stage determination value for the facial feature data as training data, outputting a sleepiness stage determination value for the facial feature data. The system includes a sleepiness stage model construction unit and a sleepiness stage model optimization unit that generates a sleepiness stage prediction model in which the accuracy of the sleepiness stage judgment value output by the initial sleepiness stage prediction model is improved by sparsifying the neural network of the initial sleepiness stage prediction model or changing the bias value. The sleepiness model generation unit includes a sleepiness level model construction unit that uses facial feature data for which the sleepiness stage is a stage requiring level determination and the sleepiness level judgment value for that facial feature data as training data, and constructs an initial sleepiness level prediction model using machine learning with a neural network that outputs a sleepiness level judgment value for facial feature data. The system also includes a sleepiness level model optimization unit that generates a sleepiness level prediction model in which the accuracy of the sleepiness level judgment value output by the initial sleepiness level prediction model is improved by sparsifying the neural network of the initial sleepiness level prediction model or changing the bias value.

[0018] This sleepiness model generation system uses facial feature data, which is time-series data of multiple facial feature points extracted from facial images. Compared to using facial images directly, the amount of data is reduced and processing time is shortened. Furthermore, this sleepiness model generation system can improve the accuracy of both sleepiness stage determination values ​​and sleepiness level determination values ​​by sparsifying the neural network or changing the bias value.

[0019] It is preferable to further include a vital sensor for estimating the drowsiness level of a person, and in the drowsiness level model optimization unit, the drowsiness level determination value estimated by the vital sensor and the facial feature data thereof are added as new training data. For example, by adding data of a specific person as new training data to optimize the initial drowsiness level prediction model, it is possible to improve the determination accuracy for that specific person, that is, to obtain a drowsiness level prediction model specialized for that person.

[0020] A drowsiness determination apparatus according to still another embodiment of the present invention includes an image acquisition unit that acquires a facial image of a person, a facial feature data extraction unit that extracts facial feature data which is time-series data of a plurality of facial feature points from the facial image acquired by the image acquisition unit, a classification determination unit that classifies the drowsiness stage of the person from the facial feature data using a drowsiness stage prediction model that has been machine-learned using known facial feature data and drowsiness stage determination values for the facial feature data as training data, and a drowsiness determination unit that determines the drowsiness level of the person from the facial feature data using a drowsiness level prediction model that has been machine-learned using known facial feature data and drowsiness level determination values for the facial feature data as training data, the drowsiness stages include a level determination required stage that requires determination of a drowsiness level, with respect to facial feature data for which the drowsiness stage has been determined to be the level determination required stage by the classification determination unit, the drowsiness level is determined by the drowsiness determination unit, and as the drowsiness level prediction model, the drowsiness level prediction model generated by the drowsiness level prediction model generation apparatus of the present invention or the drowsiness model generation system of the present invention is used.

[0021] Since the drowsiness determination apparatus uses the drowsiness level prediction model generated by the drowsiness level prediction model generation apparatus of the present invention or the drowsiness model generation system of the present invention, the drowsiness determination accuracy is high.

[0022] The image acquisition unit described above preferably has a function of aligning the face orientation of the face image to a predetermined angle. Since the image acquisition unit has the function of aligning the face orientation of the face image to the predetermined angle in this manner, the association with the drowsiness stage prediction model and the drowsiness level prediction model is facilitated, so that the drowsiness determination accuracy can be improved.

[0023] [Details of Embodiment of the Present Invention] Hereinafter, a drowsiness level prediction model generation device, a drowsiness model generation system, and a drowsiness determination device according to an embodiment of the present invention will be described with appropriate reference to the drawings.

[0024] [Drowsiness Determination Device] The drowsiness determination device 1 shown in Figure 1 is a drowsiness determination device that determines the drowsiness of a driver driving a vehicle X. When the determination indicates that the driver is drowsy, the drowsiness determination device 1 can be incorporated into a vehicle safety support device that applies stimuli to the sense of touch, smell, sight, hearing, etc., in order to arouse the driver.

[0025] The drowsiness determination device 1 includes an image acquisition unit 10, a facial feature data extraction unit 20, a classification determination unit 30, and a drowsiness determination unit 40.

[0026] <Image Acquisition Unit> The image acquisition unit 10 acquires a facial image F of a person. In the drowsiness determination device 1, the image acquisition unit 10 captures the facial expression of the driver described above.

[0027] The image acquisition unit 10 is preferably a camera capable of capturing visible light or near-infrared light. The image acquisition unit 10 may observe the entire living space of the vehicle X, but as shown in Figure 1, it is preferably provided for the driver's seat. By providing the image acquisition unit for the driver's seat in this manner, the accuracy of the facial image F can be improved.

[0028] It is preferable that the image acquisition unit 10 has a function to align the orientation of the face in the face image F to a predetermined angle. Since the face can face in various directions while driving, the face acquired by the image acquisition unit 10 may change over time. If this change happens to exhibit characteristics of a high-drowsiness state in the drowsiness level prediction model 60, the driver may be associated with a high-drowsiness state even though they are not actually drowsy. If the image acquisition unit 10 aligns the orientation of the face in the face image F to a predetermined angle, such erroneous associations can be avoided, making it easier to associate with the drowsiness stage prediction model 50 and the drowsiness level prediction model 60, thereby improving the accuracy of drowsiness detection.

[0029] <Facial Feature Data Extraction Unit> As shown in Figure 2, the facial feature data extraction unit 20 extracts facial feature data G, which is time-series data of multiple facial feature points S, from the facial image F acquired by the image acquisition unit 10.

[0030] Facial feature points S refer to representative points of major facial features, such as eyebrows, eyes, and mouth, that are decisive in estimating drowsiness based on the overall impression of the face. These representative points can be represented as 4 channels using Cartesian coordinates (x, y) with the origin of the reference point M, along with the width (w) and height (h) of the part, or as 2 channels using polar coordinates (r, θ).

[0031] Media Pipe can be used to extract facial features. Media Pipe detects a human face from an image, estimates 468 landmarks for each facial feature, and outputs them. The output is in 3D coordinates (x, y, z).

[0032] The reference point M is set to match the reference point M of the facial feature data G used as training data for the drowsiness stage prediction model 50 and the drowsiness level prediction model 60. In the example in Figure 2, the reference point M is located in the center of the facial image F.

[0033] The facial feature data G represents the time evolution of the data for each channel. Figure 2 shows, as an example, the time-series data (G1-G5) of the y-coordinates of the right eyebrow (S1), left eyebrow (S2), right eye (S3), left eye (S4), and mouth (S5), but it may also include other channels for each part and facial feature data G for other parts.

[0034] While the facial image F is essentially three-dimensional data consisting of two-dimensional x and y image data that are continuous along the time axis, the facial feature data G is two-dimensional data represented by multiple events that are continuous along the time axis of the facial feature data G. This reduction in dimension reduces the amount of data and shortens processing time.

[0035] <Classification judgment section> The classification and determination unit 30 uses a sleepiness stage prediction model 50, which has been trained using known face feature data G and sleepiness stage determination values ​​for that face feature data G as training data, to classify the sleepiness stage of a person from the face feature data G.

[0036] The classification of drowsiness stages is best done using a series of consecutive images (e.g., 16 frames) taken at regular time intervals (e.g., every 5 seconds).

[0037] Furthermore, the above drowsiness stages include a stage where a drowsiness level needs to be determined (Stage 1, hereafter also referred to as the "level determination required stage"), a stage where the person is clearly drowsy, such as by yawning (Stage 2), a state where the person is already asleep (Stage 3), and a stage where the image is not clear and judgment is difficult (Stage 4). The above drowsiness stages are discrete values.

[0038] If the classification determination unit 30 determines that the drowsiness level is a level determination required stage (stage 1), the facial feature data G is sent to the drowsiness determination unit 40. On the other hand, if the classification determination unit 30 determines that the drowsiness level is stage 2 to stage 4, the drowsiness determination device 1 outputs the classification result 31.

[0039] <Sleepiness detection unit> The drowsiness determination unit 40 uses a drowsiness level prediction model 60, which has been trained using known facial feature data G and drowsiness level determination values ​​for that facial feature data G as training data, to determine the drowsiness level 41 of a person from the facial feature data G. For facial feature data G that the classification determination unit 30 has determined to be at the level determination required stage (stage 1), the drowsiness determination unit 40 determines the drowsiness level 41 and outputs it.

[0040] The determination of drowsiness level 41 is preferably made using images of multiple consecutive frames (e.g., 16 frames) at regular time intervals (e.g., every 5 seconds), and it is preferable that these are the same images used in the classification determination unit 30.

[0041] The above drowsiness level is a numerical representation of drowsiness, based on the criteria shown in Table 1 below. The above drowsiness level is a continuous value; for example, if it is exactly halfway between 1 and 2, it will be 1.5.

[0042] [Table 1]

[0043] <Sleepiness stage prediction model and sleepiness level prediction model> The sleepiness stage prediction model 50 and sleepiness level prediction model 60 used are those generated by the sleepiness model generation system 2 of the present invention. Details will be described later.

[0044] <Advantages> The sleepiness detection device 1 uses a sleepiness stage prediction model 50 and a sleepiness level prediction model 60 generated by the sleepiness model generation system of the present invention, and therefore has high accuracy in detecting sleepiness.

[0045] [Sleepiness Model Generation System] [First Embodiment] The drowsiness model generation system 2 shown in Figure 3 is a drowsiness model generation system for determining a person's drowsiness from a facial image F. The drowsiness model generation system 2 comprises an image acquisition unit 11, a facial feature data extraction unit 20, a classification model generation unit 70, and a drowsiness model generation unit 80. The drowsiness stage prediction model 50 and the drowsiness level prediction model 60 generated by the drowsiness model generation system 2 are used in the drowsiness determination device 1 shown in Figure 1.

[0046] <Image acquisition section> The image acquisition unit 11 acquires a person's face image. The image acquisition unit 11 can be configured in the same way as the image acquisition unit 10 in Figure 1.

[0047] The image acquisition unit 11 may be a camera mounted on a different vehicle, or a camera located elsewhere. In other words, as long as the image acquisition unit 11 can acquire face images F, it does not have to be the same device, nor does it have to be located in the same position.

[0048] <Facial Feature Data Extraction Unit> The facial feature data extraction unit 20 extracts facial feature data G, which is time-series data of multiple facial feature points S, from the facial image F acquired by the image acquisition unit 11.

[0049] The facial feature data extraction unit 20 can be configured in the same way as the facial feature data extraction unit 20 of the drowsiness detection device 1 shown in Figure 1.

[0050] The facial feature points S should ideally be represented with reference point M located in the center of the facial image F as the starting point. By representing the facial feature points S with reference point M in the center of the facial image F as the starting point, it becomes easier to capture the relative movement of the entire face from the facial feature points S, thereby further improving the accuracy of the sleepiness stage determination value and sleepiness level determination value. Specifically, the reference point M should ideally be, for example, the nose, which is a part of the face located in the center.

[0051] <Classification Model Generation Unit> The classification model generation unit 70 generates a sleepiness classification prediction model 50 for classifying a person's sleepiness level from facial feature data G.

[0052] The above-mentioned drowsiness stages include a level determination stage that requires the determination of the drowsiness level, and are configured in the same way as the drowsiness stages of the drowsiness determination device 1 shown in Figure 1.

[0053] The classification model generation unit 70 includes a sleepiness stage model construction unit 71 and a sleepiness stage model optimization unit 72.

[0054] (Sleepiness Stage Model Construction Team) The sleepiness stage model construction unit 71 uses facial feature data G and sleepiness stage judgment values ​​for the facial feature data G as training data, and constructs an initial sleepiness stage prediction model 51 using machine learning with a neural network that outputs sleepiness stage judgment values ​​for the facial feature data G.

[0055] The determination of the drowsiness level judgment value corresponding to the facial feature data G can be performed, for example, by a person (expert) who can determine the drowsiness level judgment value from the facial feature data G.

[0056] A known method can be used to construct an early sleepiness stage prediction model 51 from the above training data using machine learning.

[0057] (Sleepiness Stage Model Optimization Unit) The sleepiness stage model optimization unit 72 generates a sleepiness stage prediction model 50 in which the accuracy of the sleepiness stage judgment values ​​output by the initial sleepiness stage prediction model 51 is improved by sparsifying the neural network of the initial sleepiness stage prediction model 51 or changing the bias value.

[0058] Sparsification refers to setting the parameters of some nodes in a neural network to 0. Changing the bias value refers to adjusting the parameter values ​​of some nodes by changing the bias value, which can be set externally.

[0059] As for optimization methods, for example, one can use a method in which the judgment accuracy is defined as the correct answer rate, and repeatedly modify the above parameters and thresholds through heuristic search to improve the judgment accuracy. Alternatively, characteristic nodes can be extracted from the data set of incorrect and correct answers and modified.

[0060] The facial feature data G used in the sleepiness stage model optimization unit 72 may be the same dataset as the facial feature data G used in the sleepiness stage model construction unit 71, or it may be different. For example, the facial feature data G used in the sleepiness stage model optimization unit 72 may be a dataset that adds new facial feature data G to the facial feature data G used in the sleepiness stage model construction unit 71.

[0061] <Sleepiness Model Generation Unit> The sleepiness model generation unit 80 generates a sleepiness level prediction model 60 for determining a person's sleepiness level from facial feature data G.

[0062] The sleepiness model generation unit 80 includes a sleepiness level model construction unit 81 and a sleepiness level model optimization unit 82.

[0063] (Sleepiness Level Model Construction Team) The sleepiness level model construction unit 81 uses machine learning to construct an initial sleepiness level prediction model 61 using a neural network that outputs sleepiness level judgment values ​​for face feature data G, with the above sleepiness stage being a stage requiring level determination and sleepiness level determination values ​​for that face feature data G as training data.

[0064] The determination of the drowsiness level judgment value corresponding to the facial feature data G can be performed, for example, by someone (an expert) who can determine the drowsiness level judgment value from the facial feature data G. Since the drowsiness level can be a decimal value, it is advisable for multiple experts, for example, two experts, to perform the judgment independently and adopt the average value. Performing the judgment by multiple experts can improve the validity of the training data.

[0065] A known method can be used to construct an initial sleepiness level prediction model 61 from the above training data using machine learning.

[0066] The facial feature data G used in the sleepiness level model construction unit 81 can, for example, be the facial feature data G used in the sleepiness stage model construction unit 71 where the sleepiness stage is a stage requiring level determination, but this does not prevent the use of other datasets.

[0067] (Sleep Level Model Optimization Department) The sleepiness level model optimization unit 82 generates a sleepiness level prediction model 60 that improves the accuracy of the sleepiness level judgment value output by the initial sleepiness level prediction model 61 by sparsifying the neural network of the initial sleepiness level prediction model 61 or changing the bias value.

[0068] As for the optimization method, the same optimization method as that used in the sleepiness stage model optimization unit 72 can be used.

[0069] Furthermore, when the output value of the sleepiness level of the initial sleepiness level prediction model 61 changes particularly significantly in response to various input data that are input consecutively, and there are input data that change significantly at that time (for example, the degree of eye opening, the distance between the nose and mouth, etc.), it is also effective to optimize each node of the neural network by increasing the bias value for that input.

[0070] In addition, similar to the classification model generation unit 70, the facial feature data G used in the sleepiness level model optimization unit 82 may be the same dataset as the facial feature data G used in the sleepiness level model construction unit 81, but it may also be a different dataset.

[0071] <Advantages> The sleepiness model generation system 2 uses face feature data G, which is time-series data of multiple face feature points S extracted from a face image F. Therefore, compared to directly using the face image F, the amount of data is reduced and processing time is shortened. Furthermore, the sleepiness model generation system 2 can improve the accuracy of both the sleepiness stage determination value and the sleepiness level determination value by sparsifying the neural network or changing the bias value.

[0072] [Second Embodiment] The drowsiness model generation system 3 shown in Figure 4 is a drowsiness model generation system for determining a person's drowsiness from a facial image F. The drowsiness stage prediction model 50 and drowsiness level prediction model 60 generated by the drowsiness model generation system 2 are used in the drowsiness determination device 1 shown in Figure 1.

[0073] The sleepiness model generation system 3 comprises an image acquisition unit 11 that acquires a person's face image F, a face feature data extraction unit 20 that extracts face feature data G, which is time-series data of multiple face feature points S, from the face image F acquired by the image acquisition unit 11, a classification model generation unit 70 that generates a sleepiness stage prediction model 50 for classifying the sleepiness stage of a person from the face feature data G, and a sleepiness model generation unit 80 that generates a sleepiness level prediction model 60 for determining the sleepiness level of a person from the face feature data G, wherein the sleepiness stage includes a level determination required stage that requires determination of the sleepiness level, and the classification model generation unit 70 constructs an initial sleepiness stage prediction model 51 using machine learning with a neural network that outputs sleepiness stage determination values ​​for face feature data G, using face feature data G and sleepiness stage determination values ​​for the face feature data G as training data, and the initial sleepiness stage prediction model The system includes a sleepiness stage model optimization unit 72 that generates a sleepiness stage prediction model 50 in which the accuracy of the sleepiness stage judgment value output by the initial sleepiness stage prediction model 51 is improved by sparsifying the neural network of Dell 51 or changing the bias value. The sleepiness model generation unit 80 includes a sleepiness level model construction unit 81 that constructs an initial sleepiness level prediction model 61 using machine learning with a neural network that outputs a sleepiness level judgment value for face feature data G, using face feature data G, which is a stage where the sleepiness stage is a stage where level judgment is required, and the sleepiness level judgment value for the face feature data G as training data. The sleepiness level model optimization unit 82 generates a sleepiness level prediction model 60 in which the accuracy of the sleepiness level judgment value output by the initial sleepiness level prediction model 61 is improved by sparsifying the neural network of the initial sleepiness level prediction model 61 or changing the bias value. Each of these configurations is the same as the sleepiness model generation system 2 of the first embodiment, except for the training data input to the sleepiness level model construction unit 81, so the same reference numerals are used and detailed explanations are omitted.

[0074] The sleepiness model generation system 3 further includes a vital sensor 12 for estimating a person's sleepiness level.

[0075] Examples of vital sensors 12 include well-known vital sensors capable of capturing pulse rate, heart rate, heart rate interval, blood pressure, blood glucose level, respiratory rate, etc. The type of vital sensor 12 is not particularly limited, but it is best selected from the perspective of accuracy; for example, a Doppler sensor can be used.

[0076] The vital sensor 12 is mounted on vehicle X in Figure 1. In this case, the vital sensor 12 is placed in the same position as or near the image acquisition unit 10 and acquires the driver's vital information.

[0077] The level of drowsiness can be estimated from the obtained vital information. In the drowsiness model generation system 3, the drowsiness level model optimization unit 82 adds the drowsiness level determination value estimated by the vital sensor 12 and its corresponding facial feature data G as new training data. The facial feature data G can be obtained from the image acquisition unit 10 of the vehicle X and processed by the facial feature data extraction unit 20.

[0078] The sleepiness level model optimization unit 82 uses facial feature data G where the sleepiness stage is a stage requiring level determination, and sleepiness level determination values ​​for that facial feature data G, as training data. However, all of the new training data may be used, assuming that the sleepiness stage is a stage requiring level determination, or the determination may be made based on predicted values ​​using the sleepiness stage prediction model 50 or the initial sleepiness stage prediction model 51. Alternatively, sleepiness stage determination values ​​may be estimated and used using the vital sensor 12.

[0079] By adding the new training data described above, the drowsiness state specific to vehicle X can be reflected in the drowsiness level prediction model 60. In other words, the drowsiness level prediction model 60 can be optimized to have high accuracy for vehicle X.

[0080] Furthermore, the initial drowsiness level prediction model 61 may be optimized for each driver by adding the new training data described above for each individual driver. In this case, a drowsiness level prediction model 60 is generated for each driver, and the drowsiness level prediction model 60 of the drowsiness determination device 1 will select a model corresponding to the driver. In addition to a method of selecting the model by external input, the drowsiness determination unit 40 may also have a function to identify the driver from the facial image F and select the drowsiness level prediction model 60 corresponding to the driver.

[0081] When the drowsiness detection device 1 is configured to select a model corresponding to the driver, it is preferable to generate a drowsiness level prediction model 60 in the drowsiness level model optimization unit 82 without adding the new training data. If there is no model corresponding to an individual driver, selecting this drowsiness level prediction model 60 can suppress a decrease in the detection accuracy.

[0082] The drowsiness model generation system 3 can be installed in vehicle X, but it may also be installed on an external server, and the new training data may be transferred to the external server. In this case, the generated drowsiness level prediction model 60 is transferred from the external server to vehicle X and used there.

[0083] <Advantages> As described above, the sleepiness model generation system 3 can improve the accuracy of its judgment for a specific person, for example, by adding data from that person as new training data to optimize the initial sleepiness level prediction model 61, thereby obtaining a sleepiness level prediction model 60 that is specific to that person.

[0084] [Other embodiments] The above embodiments do not limit the configuration of the present invention. Accordingly, the above embodiments allow for the omission, substitution, or addition of components of each part of the above embodiments based on the description herein and common technical knowledge, and all such omissions, substitutions, or additions should be interpreted as falling within the scope of the present invention.

[0085] In the above embodiment, the drowsiness detection device was described in the case where it detects drowsiness in a driver operating a vehicle. However, the use of the drowsiness detection device is not limited to detecting drowsiness in vehicle drivers. For example, it can be used in situations where it is necessary to prevent drowsiness, such as detecting drowsiness in pilots of aircraft or ships, or in detecting drowsiness in workers handling hazardous materials.

[0086] In the above embodiment, a case where there are four stages of drowsiness was described, but the classification criteria and the number of stages may differ, as long as there is a stage that requires the determination of the drowsiness level.

[0087] In the above embodiment, the case where the drowsiness level is a numerical value based on the judgment criteria in Table 1 was described, but the judgment criteria and numerical values ​​may be different.

[0088] In the above embodiment, a case was described in which the drowsiness detection device comprises a classification unit and a drowsiness detection unit. However, it is also possible to have a configuration that includes only a drowsiness detection unit, i.e., a configuration that does not include a classification unit. In this case, all facial feature data will be input to the drowsiness detection unit. Furthermore, a drowsiness stage prediction model may be unnecessary.

[0089] In this case, the sleepiness level prediction model used is a sleepiness level prediction model generated by the sleepiness level prediction model generation device of the present invention, that is, a sleepiness level prediction model generation device that generates a sleepiness level prediction model for determining a person's sleepiness level from a face image, and which has a sleepiness level model construction unit that constructs an initial sleepiness level prediction model using machine learning with face feature data which is time-series data of multiple face feature points extracted from the face image and a sleepiness level determination value for the face feature data as training data, and a sleepiness level model optimization unit that generates a sleepiness level prediction model in which the determination accuracy of the sleepiness level determination value output by the initial sleepiness level prediction model is improved by sparsifying the neural network of the initial sleepiness level prediction model or changing the bias value.

[0090] The sleepiness level prediction model generation device can be configured similarly to the sleepiness model generation unit shown in Figure 3, except that the training data is not limited to facial feature data where the sleepiness stage is a stage requiring level determination; therefore, a detailed explanation is omitted.

[0091] This sleepiness level prediction model generator uses facial feature data, which is time-series data of multiple facial feature points extracted from facial images. Compared to using facial images directly, the amount of data is reduced and processing time is shortened. Furthermore, this sleepiness level prediction model generator can improve the accuracy of sleepiness level determination by sparsifying the neural network or changing the bias value.

[0092] In the above embodiment, the case in which the sleepiness stage prediction model and sleepiness level prediction model generated by the sleepiness model generation system of the present invention are used as the sleepiness stage prediction model and sleepiness level prediction model was described. However, it is also possible that only the sleepiness level prediction model is a sleepiness level prediction model generated by the sleepiness model generation system of the present invention. In this case, a sleepiness stage prediction model generated by a conventional method is used as the sleepiness stage prediction model. Even with such a configuration, the sleepiness determination device will have the same effect. In this case, it is also possible to use a sleepiness level prediction model generated by the sleepiness level prediction model generation device of the present invention as the sleepiness level prediction model.

[0093] In the sleepiness model generation system of the second embodiment described above, the case in which new training data is added to the sleepiness level model optimization unit was explained. However, the sleepiness stage model optimization unit may also be configured to add the sleepiness stage determination value estimated by the vital sensor and its facial feature data as new training data. By adding new training data to both the sleepiness level model optimization unit and the sleepiness stage model optimization unit, for example, a sleepiness level prediction model and a sleepiness stage prediction model specialized for a specific person can be obtained. [Examples]

[0094] The present invention will be described in detail below based on examples, but the present invention should not be interpreted as being limited based on the description of these examples.

[0095] [Data preparation] Six videos were recorded using the same driving simulation. The subjects operated the steering wheel while watching the video, and because the simulation included vibrations similar to those of a real car, the recordings closely resembled actual driving. All subjects were male.

[0096] From these six videos, facial images consisting of 16 consecutive frames extracted approximately every 5 seconds were extracted to serve as training and evaluation data. Two experts judged the drowsiness stage and drowsiness level of the extracted facial images. If the two experts' judgments on the drowsiness level differed, the average value was adopted.

[0097] (Training data) As shown in Table 2, 1800 facial images each representing drowsiness stages 1 through 4 were selected from the aforementioned facial images to form a dataset for generating a drowsiness stage prediction model. Additionally, 1000 facial images each representing drowsiness levels 1, 1.5, 2, 2.5...5 were selected from the aforementioned facial images to form a dataset for generating a drowsiness level prediction model.

[0098] (Evaluation data) Similar to the training data, as shown in Table 2, we prepared datasets for generating a sleepiness stage prediction model and a sleepiness level prediction model from the aforementioned facial images.

[0099] [Table 2]

[0100] [Generating Predictive Models] When generating the predictive model, the batch size of the learning environment was set to 64 for the separation model generation unit and 31 for the drowsiness model generation unit.

[0101] In the separation model generation unit, the number of neurons in the output layer was set to 4, and softmax was used as the activation function. On the other hand, in the separation model generation unit, the number of neurons in the output layer was set to 1, and the linear identity function was used as the activation function.

[0102] The number of convolutional layers in the separation model generation unit and the sleepiness model generation unit was set to 4 or 3. Note that the unit with 3 convolutional layers was created by removing layers 7 and 8 from the unit with 4 convolutional layers. The number of neurons was set to 512 or 1024. Table 3 shows the number of convolutional layers and neurons in each trial from No. 1 to No. 5.

[0103] In the generation of models in the separation model generation unit and the drowsiness model generation unit, A: A sleepiness stage prediction model and a sleepiness level prediction model generated using the sleepiness stage model optimization unit and sleepiness level model optimization unit of the present invention. B: The initial sleepiness stage prediction model and initial sleepiness level prediction model generated by the sleepiness stage model construction unit and sleepiness level model construction unit are used to create the sleepiness stage prediction model and sleepiness level prediction model. C: A model that generates sleepiness stage prediction models and sleepiness level prediction models using the facial image as is. We obtained sleepiness stage prediction models and sleepiness level prediction models using the following three methods. The methods used in each trial from No. 1 to No. 5 are shown in Table 3.

[0104] [evaluation] Table 3 shows the time required to form the prediction model and the accuracy rate using the evaluation data in Table 2 for the sleepiness stage prediction model and sleepiness level prediction model obtained in No. 1 to No. 5 as described above.

[0105] [Table 3]

[0106] In Table 3, "h5 file" refers to the generated prediction model file, which is a Hierarchical Data Format (HDF) file.

[0107] The results in Table 3 show that the prediction models No. 1, No. 3, and No. 4 generated using the sleepiness model generation system of the present invention have higher accuracy rates than the unoptimized prediction model No. 2, and are superior in accuracy rates to the prediction model No. 5 which uses the face image as is, and require less data and shorter processing time. [Industrial applicability]

[0108] As described above, the sleepiness level prediction model generation device and sleepiness model generation system of the present invention can reduce the amount of data and shorten processing time, and can generate prediction models with high accuracy in determining sleepiness. Therefore, the sleepiness determination device of the present invention that uses the prediction model generated by the sleepiness level prediction model generation device or sleepiness model generation system has high accuracy in determining sleepiness. [Explanation of symbols]

[0109] 1. Drowsiness detection device 2.3 Sleepiness Model Generation System 10, 11 Image acquisition unit 12 Vital Sensors 20. Facial Feature Data Extraction Unit 30 Classification judgment section 31 Classification results 40 Sleepiness detection unit 41 Sleepiness level 50 Sleepiness Stage Prediction Model 51. Early Sleepiness Stage Prediction Model 60 Sleepiness Level Prediction Model 61. Initial Sleepiness Level Prediction Model 70 Classification Model Generation Unit 71 Sleepiness Stage Model Construction Department 72 Sleepiness Stage Model Optimization Unit 80 Sleepiness Model Generation Unit 81 Sleepiness Level Model Construction Department 82 Sleepiness Level Model Optimization Section X Vehicle F Face image S Facial feature points G Facial feature data M reference point

Claims

1. A sleep level prediction model generation device that generates a sleep level prediction model for determining a person's sleep level from a facial image, A sleepiness level model construction unit constructs an initial sleepiness level prediction model using machine learning with a neural network that outputs sleepiness level judgment values ​​for face feature data, using face feature data which is time-series data of multiple face feature points extracted from the above face image and sleepiness level judgment values ​​for that face feature data as training data. A sleep level model optimization unit generates a sleep level prediction model that improves the accuracy of the sleep level judgment value output by the initial sleep level prediction model by sparsifying the neural network of the initial sleep level prediction model or changing the bias value. It has, The sleepiness level prediction model generation device optimizes the neural network by repeatedly modifying the parameters and thresholds of the nodes in the neural network so that the above-mentioned judgment accuracy is improved through heuristic search, using the above-mentioned judgment accuracy as the accuracy rate.

2. The sleepiness level prediction model generation device according to claim 1, wherein the above facial feature points are represented starting from a reference point located in the center of the above facial image.

3. A sleepiness model generation system for determining a person's sleepiness from a facial image, An image acquisition unit that acquires facial images of people, A face feature data extraction unit extracts face feature data, which is time-series data of multiple face feature points, from the face image acquired by the image acquisition unit. A classification model generation unit generates a sleepiness classification prediction model for classifying a person's sleepiness level as a discrete value from the above facial feature data, A sleepiness model generation unit generates a sleepiness level prediction model for determining a person's sleepiness level as a continuous value from the above facial feature data. Equipped with, The above drowsiness stages include a stage requiring a level determination that requires a determination of the drowsiness level. The above classification model generation unit, A sleepiness stage model construction unit constructs an initial sleepiness stage prediction model using machine learning with a neural network that outputs sleepiness stage judgment values ​​for the face feature data, using the above-mentioned face feature data and sleepiness stage judgment values ​​for that face feature data as training data. A sleepiness stage model optimization unit generates a sleepiness stage prediction model that improves the accuracy of sleepiness stage judgment values ​​output by the above-mentioned initial sleepiness stage prediction model by sparsifying the neural network of the above-mentioned initial sleepiness stage prediction model or changing the bias value. It has, In the optimization unit described above, the above-mentioned judgment accuracy is used as the accuracy rate, and the neural network's node parameters and thresholds are repeatedly modified through heuristic search to improve the above-mentioned judgment accuracy. The above sleepiness model generation unit, A sleepiness level model construction unit constructs an initial sleepiness level prediction model using machine learning with a neural network that outputs sleepiness level judgment values ​​for face feature data, using face feature data where the above sleepiness stage is a stage requiring level determination and sleepiness level judgment values ​​for that face feature data as training data. A sleep level model optimization unit generates a sleep level prediction model that improves the accuracy of the sleep level judgment value output by the initial sleep level prediction model by sparsifying the neural network of the initial sleep level prediction model or changing the bias value. A sleepiness model generation system having the following features.

4. To improve the accuracy of judgment for a specific person, the system further comprises a vital sensor for estimating the sleepiness level of a person, The sleepiness model generation system according to claim 3, wherein the sleepiness level determination value estimated by the vital sensor and its facial feature data are added as new training data in the sleepiness level model optimization unit described above.

5. An image acquisition unit that acquires facial images of people, A face feature data extraction unit extracts face feature data, which is time-series data of multiple face feature points, from the face image acquired by the image acquisition unit. A classification and determination unit that classifies a person's sleepiness level from the facial feature data using a sleepiness level prediction model that has been machine-learned using known facial feature data and sleepiness level determination values ​​for that facial feature data as training data, A sleepiness determination unit determines a person's sleepiness level from the facial feature data using a sleepiness level prediction model that has been machine-learned using known facial feature data and sleepiness level determination values ​​for that facial feature data as training data. Equipped with, The above drowsiness stages include a stage requiring a level determination that requires a determination of the drowsiness level. For facial feature data in which the above classification and determination unit has determined that the above drowsiness stage is a stage requiring level determination, the above drowsiness determination unit determines the drowsiness level. A sleepiness determination device that uses a sleepiness level prediction model generated by the sleepiness level prediction model generation device described in claim 1 or the sleepiness model generation system described in claim 3 as the sleepiness level prediction model.

6. The sleepiness detection device according to claim 5, wherein the image acquisition unit has a function to align the orientation of the face in the face image to a predetermined angle.

Images