Prediction device, prediction method, teacher data generation device, teacher data generation method, model training device, model training method, and program

Abstract

To directly predict eye-openness from an image without the need for detecting feature points of an eye.SOLUTION: A prediction device (30) includes: an acquisition unit (311) which acquires an examination image including an eye as a subject; and a prediction unit (312) which predicts eye-openness from the examination image by using a deep learning model (ML2) trained so as to output the eye-openness in response to input of the image including the eye as the subject.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for obtaining facial features from an image. [Background technology]

[0002] There is known a technique for obtaining facial features from an image. An example of a facial feature is the degree of eye opening. The degree of eye opening is information required for evaluating, for example, the degree of alertness. An example of an index for evaluating the level of alertness is PERCLOS (PERcent time of slow eyelid CLOSures), which indicates the percentage of time the eyes are open. PERCLOS is calculated from the degree of eye opening of each image in a time series of images that include eyes as subjects. For this reason, a technique for obtaining the degree of eye opening from an image is required. Non-Patent Document 1 describes a technique for detecting eye feature points by performing segmentation on an image, and calculating the degree of eye opening based on the detected feature points. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Qianyang Zhuang et. al., "Driver fatigue detection method based on eye states with pupil and iris segmentation", Ieee Access, 8:173440-173449, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technique described in Non-Patent Document 1 has a problem in that if eye feature points cannot be detected accurately from an image, the degree of eye opening cannot be obtained accurately. One aspect of the present invention aims to realize a technique for directly predicting the degree of eye opening from an image without the need to detect eye feature points. [Means for solving the problem]

[0005] In order to solve the above problems, a prediction device according to a first aspect of the present invention includes an acquisition unit that acquires a test image including an eye as a subject, and a prediction unit that predicts the eye opening degree from the test image using a deep learning model that has been trained to output the eye opening degree when an image including an eye as a subject is input. With the above configuration, it is possible to directly predict the eye opening degree from the image without the need to detect eye feature points.

[0006] In order to solve the above problem, the prediction device according to aspect 2 of the present invention is, in aspect 1, the deep learning model is trained using teacher data in which an image including an eye as a subject is associated with an eye opening degree, and the teacher data is generated based on two images of the same eye taken by two different image taking devices. In the above configuration, the two images of the same eye taken by two different image taking devices contain more diverse information than one image taken of an eye as a subject, or than two images taken by the same image taking device of the same eye as a subject. Therefore, the teacher data generated based on two images containing such more diverse information is expected to be of higher quality. As a result, a deep learning model trained with high accuracy by referring to such high-quality teacher data can be used, and the eye opening degree can be predicted directly from the test image with high accuracy.

[0007] In order to solve the above problem, the prediction device according to aspect 3 of the present invention is, in aspect 2, the teacher data is generated by associating the degree of eye opening obtained from one of the two images, which has a higher accuracy of the degree of eye opening obtainable based on each image, with the other image. With the above configuration, the teacher data associates an image including an eye as a subject with a degree of eye opening that is more accurate than the degree of eye opening obtainable from the image. As a result, a deep learning model trained using such teacher data can be used, and the degree of eye opening can be directly predicted from the test image with higher accuracy.

[0008] In order to solve the above problem, the prediction device according to aspect 4 of the present invention is the same as that in aspect 3, in which the two photographing devices have different performances from each other, the one image is a first image photographed by a first photographing device having a higher performance among the two photographing devices, and the other image is a second image photographed by a second photographing device having a lower performance among the two photographing devices. According to the above configuration, the degree of eye opening can be predicted directly with high accuracy from a test image photographed by a photographing device having a performance similar to that of the second photographing device. In addition, in order to obtain a test image, it is not necessary to use a photographing device having a performance similar to that of the first photographing device having a higher performance, but it is sufficient to use a photographing device having a performance similar to that of the second photographing device having a lower performance. Since such photographing devices are often inexpensive, a prediction device that accurately predicts the degree of eye opening directly from a test image can be realized at low cost.

[0009] In order to solve the above problem, the teacher data generating device according to the fifth aspect of the present invention includes an acquisition unit that acquires two images captured by two different image capture devices of the same eye as a subject, and a generation unit that generates teacher data in which the image including the eye as a subject is associated with an eye opening degree based on the two images. In the above configuration, the two images captured by the two different image capture devices of the same eye as a subject contain more diverse information than one image captured of the eye as a subject, or than two images captured by the same image capture device of the same eye as a subject. Therefore, better quality teacher data can be generated based on two images containing such more diverse information.

[0010] In order to solve the above problem, the model learning device according to the sixth aspect of the present invention includes a teacher data acquisition unit that acquires teacher data in which an eye opening degree is associated with an image including an eye as a subject, and a learning unit that uses the teacher data to learn a deep learning model that inputs an image including an eye as a subject and outputs an eye opening degree, and the teacher data is generated by referring to two images taken by two different camera devices with the same eye as a subject. In the above configuration, the two images taken by the two different camera devices with the same eye as a subject contain more diverse information than one image taken with the eye as a subject, or than two images taken by the same camera device with the same eye as a subject. Therefore, the teacher data generated based on the two images containing such more diverse information is expected to be of good quality. As a result, by using such teacher data, a deep learning model that directly predicts the eye opening degree from an image can be accurately learned.

[0011] In order to solve the above problem, a prediction method according to aspect 7 of the present invention is a prediction method executed by a computer, and includes an acquisition step in which the computer acquires an examination image including an eye as a subject, and a prediction step in which the computer predicts an eye opening degree from the examination image using a deep learning model trained to input an image including an eye as a subject and output an eye opening degree. According to the above configuration, the same effect as aspect 1 is achieved.

[0012] In order to solve the above problem, a teacher data generating method according to an eighth aspect of the present invention is a teacher data generating method executed by a computer, and includes an acquisition step in which the computer acquires two images captured by two different image capturing devices of the same eye as a subject, and a generation step in which the computer generates teacher data in which the image including the eye as a subject is associated with an eye opening degree by referring to the two images. According to the above configuration, the same effect as that of the fifth aspect can be achieved.

[0013] In order to solve the above problem, a model learning method according to a ninth aspect of the present invention is a model learning method executed by a computer, and includes a teacher data acquisition step in which the computer acquires teacher data in which an image including an eye as a subject is associated with an eye opening degree, and a learning step in which the computer uses the teacher data to learn a deep learning model that takes the image as an input and outputs an eye opening degree, and the teacher data is generated by referring to two images of the same eye as a subject taken by two different photographing devices. According to the above configuration, the same effect as that of the sixth aspect can be achieved.

[0014] In order to solve the above problems, a program according to a tenth aspect of the present invention is a program that causes a computer to function as the prediction device according to any one of aspects 1 to 4, and causes the computer to function as the acquisition unit and the prediction unit. According to the above configuration, the same effect as any one of aspects 1 to 4 is achieved.

[0015] In addition, in order to solve the above problem, a program related to aspect 11 of the present invention is a program that causes a computer to function as a teacher data generation device related to aspect 5, and causes the computer to function as the acquisition unit and the generation unit.

[0016] In order to solve the above problems, a program according to a twelfth aspect of the present invention is a program for making a computer function as a model learning device, and makes the computer function as the acquisition unit and the learning unit. According to the above configuration, the same effects as those of the sixth aspect can be achieved.

[0017] It should be noted that a computer-readable recording medium on which each of the above programs is recorded also falls within the scope of the present invention. Effect of the Invention

[0018] According to one aspect of the present invention, a deep learning model can be used to directly predict eye opening from an image, without the need to detect eye feature points. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a functional configuration of a teacher data generation device according to a first embodiment of the present invention. FIG. [Diagram 2] 1 is a flow diagram showing the flow of a teacher data generating method according to a first embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing the functional configuration of a model learning device according to a second embodiment of the present invention. [Figure 4] FIG. 11 is a flowchart showing the flow of a model learning method according to a second embodiment of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a functional configuration of a prediction device according to a third embodiment of the present invention. [Figure 6] FIG. 11 is a flowchart showing the flow of a prediction method according to a third embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing quantitative evaluation results of a prediction device according to embodiment 3 of the present invention and a comparative system. [Figure 8] FIG. 11 is a diagram comparing the EOD predicted by the prediction device according to the third embodiment of the present invention and a comparison system. [Figure 9] This is a graph plotting the time series distribution of PERCLOS' correct answers and predicted values. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The inventors have invented a method for directly predicting the degree of eye opening from an image including an eye as a subject by using a deep learning model in a regressive manner. They have also invented a method for constructing a set of teacher data for training such a deep learning model. They have also invented a method for training a deep learning model using such a teacher data set. Note that the degree of eye opening in each embodiment described below may be the degree of eye opening of a human or may be the degree of eye opening of a creature other than a human that opens and closes its eyes.

[0021] Here, the conventionally known technology described in Non-Patent Document 1 and the like presupposes a high-quality image capable of detecting eye features in order to calculate the degree of eye opening, and therefore has a problem that the photographing device is expensive and difficult to spread. In addition to the calculation cost of detecting eye features, the calculation cost of calculating the degree of eye opening from the eye features is also high, which is a problem. Each embodiment described below has the advantage that the degree of eye opening can be predicted directly with high accuracy even when the image quality of the test image of the eye as a subject is low, and not only can the calculation cost be reduced, but also an inexpensive photographing device can be used, making it easy to spread.

[0022] [Embodiment 1] The teacher data generating device 10 according to the first embodiment of the present invention will be described in detail with reference to the drawings. The teacher data generating device 10 is a device that generates teacher data in which an eye opening degree is associated with an image including an eye as a subject. The teacher data generated by the teacher data generating device 10 can be used in a model learning device 20 according to the second embodiment described later.

[0023] (Configuration of the teacher data generation device) Fig. 1 is a block diagram showing the functional configuration of a teacher data generation device 10. As shown in Fig. 1, the teacher data generation device 10 includes a control unit 11 and a storage unit 12. The control unit 11 controls each unit of the teacher data generation device 10. The storage unit 12 stores various data used by the control unit 11. For example, when the teacher data generation device 10 is configured by a computer including a processor and a memory, the control unit 11 is configured by the processor, and the storage unit 12 is configured by the memory.

[0024] The control unit 11 includes an acquisition unit 111 and a generation unit 112. The acquisition unit 111 acquires two images captured by two different imaging devices of the same eye as a subject. The generation unit 112 generates teacher data in which an image including an eye as a subject is associated with an eye opening degree based on the two images. The teacher data may be generated by associating the eye opening degree acquired from one of the two images, which has a higher accuracy of the eye opening degree that can be acquired based on each image, with the other image. The generated teacher data is stored in the storage unit 12, for example.

[0025] Here, the two different imaging devices are a first imaging device 91 and a second imaging device 92. As shown in FIG. 1, the teacher data generation device 10 is connected to the first imaging device 91 and the second imaging device 92. The first imaging device 91 and the second imaging device 92 have different performances. Here, the first imaging device 91 has higher performance than the second imaging device 92. The difference in performance between the first imaging device 91 and the second imaging device 92 may be, for example, a difference in resolution, but is not limited to this. For example, the difference in performance may be a difference in performance other than resolution, such as a difference in aperture value, a difference in angle of view, etc. Also, when performance other than resolution is different, the resolution may be the same. In the following, the difference in performance is assumed to be a difference in resolution, and an example of a first image captured by the first imaging device 91, which has higher performance among the two imaging devices, is described as a high-quality image. Also, an example of a second image captured by the second imaging device 92, which has lower performance among the two imaging devices, is described as a low-quality image.

[0026] The performance of the first photographing device 91 is required to be such that it can photograph an image capable of acquiring the degree of eye opening by a known technique to be used as the correct data. For example, the first photographing device 91 may be a photographing device capable of photographing a high-quality image capable of detecting eye feature points for calculating the degree of eye opening by the technique described in Non-Patent Document 1. Also, for example, the first photographing device 91 may be a non-contact eye tracker that outputs the degree of eye opening while photographing a high-quality image. However, specific examples of the first photographing device 91 are not limited to this.

[0027] The performance of the second photographing device 92 does not need to be such that it can photograph an image capable of acquiring the degree of eye opening using a publicly known technique. It is desirable that the second photographing device 92 be an inexpensive device. For example, the second photographing device 92 may be a webcam or a camera built into a smartphone, but is not limited thereto.

[0028] (Flow of teacher data generation method) The teacher data generating device 10 configured as above executes the teacher data generating method S10. For example, the teacher data generating method S10 is a method of generating a teacher data set including a plurality of teacher data while continuously photographing the eye of the same subject using the first photographing device 91 and the second photographing device 92. At the start of the teacher data generating method S10, the first photographing device 91 and the second photographing device 92 are arranged so as to include the eye of the subject in the angle of view, and start continuous photographing. In other words, the first photographing device 91 photographs high-quality images including the subject's eye as a subject at a predetermined interval, thereby outputting a time series of high-quality images. In addition, the second photographing device 92 photographs low-quality images including the subject's eye as a subject at a predetermined interval, thereby outputting a time series of low-quality images.

[0029] Fig. 2 is a flow diagram showing the flow of the teacher data generating method S10. As shown in Fig. 2, the teacher data generating method S10 includes steps S11 to S14.

[0030] In step S11, the acquisition unit 111 acquires, from the first image capturing device 91, a high-quality image including the subject's eye as a subject.

[0031] In step S12, the acquisition unit 111 acquires a low-quality image including the subject's eye as a subject from the second image capture device 92. The high-quality image and the low-quality image acquired in steps S11 and S12 are captured at corresponding times. The corresponding times are preferably simultaneous. However, if the capture timings of the first image capture device 91 and the second image capture device 92 are not completely synchronized, the high-quality image and the low-quality image may be captured at the closest times to each other.

[0032] In step S13, the generating unit 112 obtains the degree of eye opening from the high-quality image. For example, the generating unit 112 may detect eye feature points from the high-quality image using the technique described in Non-Patent Document 1, and calculate the degree of eye opening based on the detected feature points.

[0033] If the first image capturing device 91 has a function of acquiring and outputting the degree of eye opening from a captured high-quality image, the execution of step S12 may be omitted. In this case, the generation unit 112 acquires the degree of eye opening based on the high-quality image from the first image capturing device 91 in step S13.

[0034] In step S14, the generating unit 112 generates teacher data in which the low-quality image acquired in step S11 is associated with the eye opening degree acquired in step S13. The generating unit 112 stores the generated teacher data in the storage unit 12.

[0035] In step S15, the control unit 11 determines whether or not to end the generation of the teacher data. For example, the end condition may be, but is not limited to, that a predetermined time has elapsed since the start of the teacher data generation method S10, that a predetermined number of teacher data have been generated, that at least one of the first image capturing device 91 and the second image capturing device 92 has finished capturing images, or a combination of these.

[0036] If the control unit 11 determines No (not to end) in step S15, it repeats the process from step S11. If the control unit 11 determines Yes (to end) in step S15, it ends the teacher data generating method S10.

[0037] The teacher data generating method S10 is not limited to generating a set of teacher data while continuously photographing the eyes of the same subject. For example, the teacher data generating method S10 may be executed by referring to a time series of low-quality images and a time series of high-quality images stored in the storage unit 12 by continuously photographing the eyes of the same subject in advance. The teacher data generating method S10 is not limited to a time series of low-quality images and a time series of high-quality images, and may be executed for a set of low-quality images and high-quality images including the eyes of the same subject as objects. The teacher data generating method S10 may be executed for each of a plurality of subjects.

[0038] (Effects of this embodiment) The teacher data generating device 10 and the teacher data generating method S10 according to the present embodiment generate teacher data in which the degree of eye opening obtained based on the high-quality image captured by the first image capturing device 91, which has higher performance, of two images captured by the first image capturing device 91 and the second image capturing device 92 with the same eye as a subject is associated with the low-quality image captured by the second image capturing device 92, which has lower performance. The teacher data associates the high-precision degree of eye opening obtained from the high-quality image with the low-quality image. Therefore, the teacher data can be used to train a deep learning model that outputs the degree of eye opening when an image including an eye as a subject is input, to predict the degree of eye opening with high accuracy even for low-quality images.

[0039] [Embodiment 2] A model learning device 20 according to a second embodiment of the present invention will be described with reference to the drawings. For convenience of description, the same reference numerals are used for components having the same functions as those described in the first embodiment, and the description thereof will not be repeated. The model learning device 20 is a device that trains a deep learning model ML2 so as to output an eye opening degree when an image including an eye as a subject is input. The deep learning model ML2 can be used in a prediction device 30 according to a third embodiment described later.

[0040] (Configuration of model learning device) Fig. 3 is a block diagram showing the functional configuration of model learning device 20. As shown in Fig. 3, model learning device 20 includes a control unit 21 and a storage unit 22. Control unit 21 comprehensively controls each unit of model learning device 20. Storage unit 22 stores various data used by control unit 21. For example, when model learning device 20 is configured by a computer including a processor and a memory, control unit 21 is configured by a processor, and storage unit 22 is configured by a memory.

[0041] The storage unit 22 stores a set of teacher data. The teacher data is generated by referring to two images of the same eye taken by two different photographing devices. For example, the teacher data may be generated by associating the degree of eye opening obtained from one of the two images, which has a higher accuracy of the degree of eye opening that can be obtained based on each image, with the other image. More specifically, the teacher data may be data in which the degree of eye opening obtained from a high-quality image is associated with a low-quality image. Here, it is assumed that the set of teacher data is generated by the teacher data generating device 10 according to the first embodiment of the present invention.

[0042] The storage unit 22 also stores an extraction model ML1 and a deep learning model ML2. The extraction model ML1 is a model that extracts a partial image near the eye region from an image of a subject. A known technique can be adopted as the extraction model ML1. The deep learning model ML2 is a model that receives an image including an eye as a subject as an input and outputs an eye opening degree. The deep learning model ML2 learns under the control of a learning unit 212, which will be described later.

[0043] The control unit 21 includes an acquisition unit 211 and a learning unit 212. The acquisition unit 211 acquires teacher data in which an image including an eye as a subject is associated with an eye opening degree. For example, the acquisition unit 211 acquires teacher data stored in the storage unit 22. The acquisition unit 211 also uses the extraction model ML1 to extract a partial image near the eye region from a low-quality image included in the teacher data and output the partial image to the learning unit 212. The learning unit 212 trains a deep learning model ML2, which receives an image including an eye as a subject as an input and outputs an eye opening degree, using the teacher data. The "image including an eye as a subject" input to the deep learning model ML2 is the partial image near the eye region extracted from the teacher data by the acquisition unit 211.

[0044] (Model learning method flow) The model learning device 20 configured as above executes a model learning method S20. Fig. 4 is a flow diagram showing the flow of the model learning method S20. As shown in Fig. 4, the model learning method S20 includes steps S21 to S24.

[0045] In step S21, the acquisition unit 211 acquires a set of teacher data from the storage unit 22. The acquisition unit 211 may read a part of the teacher data stored in the storage unit 22 for learning in step S23, which will be described later. In this case, a part or all of the remaining teacher data may be used for verification in step S24, which will be described later.

[0046] In step S22, the acquisition unit 211 uses the extraction model ML1 to extract a partial image near the eye region from the low-image-quality image included in the training data acquired in step S21. Hereinafter, the extracted partial image is also simply referred to as a low-image-quality image.

[0047] In step S23, the learning unit 212 trains the deep learning model ML2 using the set of teacher data. The training is performed, for example, by repeating a process of adjusting parameters constituting the deep learning model ML2 so that data output when the low-quality image extracted in step S22 is input to the deep learning model ML2 approaches the degree of eye opening associated with the low-quality image from which the low-quality image was extracted.

[0048] In step S24, the control unit 21 determines whether to end the learning. For example, the end condition may be, but is not limited to, that a predetermined number of epochs is reached, that the performance of the deep learning model ML2 exceeds a predetermined level, that the parameters of the deep learning model ML2 converge, or a combination of these. In step S24, in order to evaluate the performance of the deep learning model ML2 for determining the end condition, a part or all of the remaining teacher data stored in the storage unit 22 that is not used for learning may be used.

[0049] If it is determined in step S24 that the process is No (not to end), the control unit 21 repeats the process from step S21. If it is determined in step S24 that the process is Yes (to end), the control unit 21 ends the model learning method S20.

[0050] (Effects of this embodiment) The model learning device 20 and the model learning method S20 according to the present embodiment train the deep learning model ML2 using teacher data in which the degree of eye opening obtained based on the high-quality image captured by the first image capturing device 91, which has higher performance, of two images captured by the first image capturing device 91 and the second image capturing device 92 with the same eye as a subject, is associated with the low-quality image captured by the second image capturing device 92, which has lower performance. This allows the deep learning model ML2 to be trained so that the degree of eye opening can be directly predicted with high accuracy for images captured by an image capturing device with performance comparable to that of the second image capturing device 92.

[0051] [Embodiment 3] A prediction device 30 according to a third embodiment of the present invention will be described with reference to the drawings. For ease of explanation, the same reference numerals are used for components having the same functions as those described in the first and second embodiments, and the explanations thereof will not be repeated. The prediction device 30 is a device that directly predicts the degree of eye opening from a test image including an eye as a subject, using a deep learning model ML2 that has been trained to output the degree of eye opening when an image including an eye as a subject is input.

[0052] (Configuration of prediction device) Fig. 5 is a block diagram showing a functional configuration of the prediction device 30. As shown in Fig. 5, the prediction device 30 includes a control unit 31 and a storage unit 32. The control unit 31 controls each unit of the prediction device 30. The storage unit 32 stores various data used by the control unit 31. For example, when the prediction device 30 is configured by a computer including a processor and a memory, the control unit 31 is configured by the processor, and the storage unit 32 is configured by the memory.

[0053] The storage unit 32 stores an extraction model ML1 and a deep learning model ML2. The extraction model ML1 has been described in the second embodiment, and detailed description will not be repeated. The deep learning model ML2 is a model that has been trained to output an eye opening degree when an image including an eye as a subject is input. For example, the deep learning model ML2 is trained using teacher data in which an image including an eye as a subject is associated with an eye opening degree, and the teacher data is generated based on two images of the same eye taken by two different photographing devices. For example, the teacher data may be generated by associating the eye opening degree acquired from one of the two images, which has a higher accuracy of the eye opening degree that can be acquired based on each image, with the other image. More specifically, the teacher data may be data in which a low-image quality image is associated with an eye opening degree acquired from a high-image quality image. Here, the deep learning model ML2 is assumed to be a deep learning model ML2 trained by the model learning device 20.

[0054] The control unit 31 includes an acquisition unit 311, a prediction unit 312, and an output unit 313. The acquisition unit 311 acquires an examination image including an eye as a subject. The acquisition unit 311 also extracts a partial image near the eye region from the examination image using an extraction model ML1 and outputs the extracted partial image to the prediction unit 312. The prediction unit 312 predicts the degree of eye opening from the examination image using a deep learning model ML2 that has been trained to output the degree of eye opening when an image including an eye as a subject is input. The "image including an eye as a subject" input to the deep learning model ML2 is the partial image near the eye region extracted from the examination image by the acquisition unit 311. The output unit 313 outputs the predicted degree of eye opening.

[0055] For example, the test image is acquired from a third imaging device 93. As shown in FIG. 5, the prediction device 30 is communicably connected to the third imaging device. The third imaging device 93 is a device that captures a test image including the subject's eye as a subject. The performance of the third imaging device 93 is similar to that of the second imaging device 92. In other words, the performance of the third imaging device 93 may be such that the degree of eye opening cannot be obtained with high accuracy from the test image captured by the device using publicly known technology.

[0056] (Prediction method flow) The prediction device 30 configured as above executes a prediction method S30. Fig. 6 is a flow diagram showing the flow of the prediction method S30. As shown in Fig. 6, the prediction method S30 includes steps S31 to S34.

[0057] In step S31, the acquisition unit 311 acquires a test image of the subject's eye captured by the third image capturing device 93. Here, the third image capturing device 93 is assumed to have performance equivalent to that of the second image capturing device 92, and the test image is also referred to as a "low-quality test image."

[0058] In step S32, the acquisition unit 311 uses the extraction model ML1 to extract a partial image near the eye region from the low-image-quality test image acquired in step S31. Hereinafter, the extracted partial image is also simply referred to as a "low-image-quality test image." In step S33, the prediction unit 312 predicts the degree of eye opening by inputting the low-image-quality test image to the deep learning model ML2. In step S34, the output unit 313 outputs the predicted degree of eye opening. The output destination of the degree of eye opening may be, for example, a display device (not shown) connected to or built into the prediction device 30. Also, for example, the output destination of the degree of eye opening may be a speaker (not shown) connected to or built into the prediction device 30. Also, for example, the output destination of the degree of eye opening may be various output devices (not shown) provided in other devices connected to the network.

[0059] (Effects of this embodiment) The prediction device 30 and prediction method S30 according to the present embodiment predict the degree of eye opening using a deep learning model ML2 that has been trained to input an image including an eye as a subject and output the degree of eye opening, so that it is possible to directly predict the degree of eye opening from a test image without the need to detect eye feature points. Also, the calculation cost can be reduced compared to the case where the degree of eye opening is calculated by detecting eye feature points.

[0060] The prediction device 30 and prediction method S30 according to the present embodiment predict the degree of eye opening from a test image including the eye as a subject by using a deep learning model ML2 that is trained using teacher data associated with a low-quality image captured by the second image capture device 92, which has lower performance, and an eye opening degree obtained based on a high-quality image captured by the first image capture device 91, which has higher performance, of two images captured by the first image capture device 91 and the second image capture device 92, which have the same eye as a subject. As a result, even if the image quality of the test image including the eye as a subject is comparable to that of the low-quality image captured by the second image capture device 92, it is possible to accurately predict the degree of eye opening.

[0061] In addition, by accurately predicting the degree of eye opening using the prediction device 30, it becomes possible to estimate the subject's attention, alertness, and performance in a wide range of situations, such as everyday life, which can be useful in improving the performance of e-sports athletes, office workers, drivers, and the like.

[0062] [Example] An experiment was conducted to generate teacher data, generate a deep learning model ML2, and predict the eye opening degree by applying EOD (eyelid opening degree) as the eye opening degree in each of the above-mentioned embodiments. EOD is defined in the following reference 1, and is expressed as the value obtained by dividing the maximum vertical width of the eye region by the original diameter of the iris.

[0063] [Reference 1] David F Dinges, Malissa M Mallis, Greg Maislin, John Walker Powell, et al. Evaluation of techniques for ocular measurement as an index of fatigue and as the basis for alertness management. Technical report, United States. National Highway Traffic Safety Administration, 1998.

[0064] (Example 1 of generating training data) An experiment was conducted to generate teacher data by photographing 97 subjects using the above-mentioned teacher data generating device 10. In this experiment, a webcam was used as the first photographing device 91. In addition, a non-contact eye tracker FX3 with an infrared sensor manufactured by Eye Tracking Inc. (hereinafter, referred to as FX3) was used as the second photographing device 92. FX3 uses infrared rays to output measurement values ​​such as EOD at 61.6 Hz. In other words, the EOD output from FX3 is the degree of eye opening obtained based on an image photographed using infrared rays. FX3 has a higher resolution than a webcam. In this experiment, each subject was photographed for 40 minutes using the first photographing device 91 and the second photographing device 92, and 2,328,000 pieces of image data with EOD (i.e., teacher data) could be generated using the teacher data generating device 10.

[0065] Among the generated teacher data, the number of images with eyes open used in Examples 2 and 3 described later was limited to three times the number of images with eyes closed. Of the teacher data for 97 subjects, 77 were used for learning / verification in Example 2 described later, and the remaining 20 were used for testing in Example 3 described later. As a result, the teacher data for learning / verification was about 1600 images per subject on average, totaling 161,240 images. Of these, 128,992 images were used for learning and 32,248 images were used for verification. The teacher data for testing was 480,000 images for 20 subjects.

[0066] (Example 2 of deep learning model training) The above-mentioned model learning device 20 was used to learn four types of deep learning models ML2 using the above-mentioned learning / verification teacher data among the teacher data generated in Example 1. The model learning device 20 used the Blaze Face architecture (see Reference 2) as the extraction model ML1 that extracts a partial image near the iris from the teacher data. Here, VGG19 (see Reference 3), ResNet50 (see Reference 4, hereinafter RN50), EfficientNet (see Reference 5, hereinafter EffNet), and Swin Transformer (see Reference 6, hereinafter Swin) were adopted as the four types of deep learning models ML2. In addition, in these four types of models, only the output layer was replaced so as to output a one-dimensional EOD value. In addition, these four types of models were previously trained on ImageNet (see Reference 7) except for the last fully connected layer.

[0067] [Reference 2] Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. Blazeface: Sub-millisecond neural face detection on mobile gpus. CoRR, abs / 1907.05047, 2019. [Reference 3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. [Reference 4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. [Reference 5] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019. [Reference 6] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE / CVF international conference on computer vision, pages 10012-10022, 2021. [Reference 7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.

[0068] (Example 3 of predicting eye opening degree) An experiment was conducted to predict EOD by the prediction device 30 described above using each of the four types of deep learning models ML2 trained in Example 2. In addition, the PERCLOS of each subject was calculated based on the predicted EOD. For the prediction of EOD, the training data for 20 subjects for testing among the training data generated in Example 1 was used as the test images. In addition, the EOD associated with the training data was referred to as the correct answer in order to compare with the predicted value.

[0069] In addition, to evaluate the accuracy of the four types of deep learning models ML2, we compared them with six comparison systems. All of the six comparison systems were constructed to detect the contours (feature points) of the eyelids and iris from images and calculate the EOD based on the detected contours. More specifically, each of the six comparison systems uses the above-mentioned Blaze Face architecture to estimate the position and key points of the face from the image to obtain a trimmed image near the iris. Each comparison system also estimates the eyelid contour by performing quadratic curve approximation on the eyelid key points, and calculates the maximum vertical width of the eye region from the estimated eyelid contour. Each comparison system also inputs the trimmed image near the iris into a segmentation model to perform iris segmentation, and performs circumscribing circle approximation based on the segmentation result. The diameter of the circumscribing circle can be considered to be equivalent to the original diameter of the iris. Each comparison system then calculates the EOD from the original diameter of the iris and the above-mentioned maximum vertical width.

[0070] All six comparison systems adopted U-Net (see Reference 8) as the iris segmentation architecture, but used different models as encoders. The encoders used by the six comparison systems were ResNet18 (see Reference 4, hereafter RN18), the above-mentioned RN50, EfficientNet-b1 (see Reference 5, hereafter EffNet), Timm-EfficientNet-b4 (see Reference 5, hereafter timm-EffNet), Mix Vision Transformer-b1 (see Reference 9, hereafter mit-b1), and Mix Vision Transformer-b4 (see Reference 9, hereafter mit-b4).

[0071] [Reference 8] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015. [Reference 9] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 12135-12144, 2022.

[0072] EOD was predicted using these six comparison systems. In addition, PERCLOS for each subject was calculated based on the predicted EOD. For EOD prediction, the training data of the 20 subjects mentioned above was used as the test images. In addition, the EOD associated with the training data was referred to as the correct answer for comparison with the predicted value.

[0073] Fig. 7 shows the quantitative evaluation results of the prediction device 30 and the comparative systems. In Fig. 7, SM shows the quantitative evaluation of the six comparative systems (EffNet, timm-EffNet, RN18, RN50, mit-b1, mitb4) described above. RM shows the quantitative evaluation of the prediction device 30 using the four deep learning models ML2 (VGG19, EffNet, RN50, Swin) described above.

[0074] As an index of quantitative evaluation, the mean square error (MSE) and intraclass correlation coefficients (ICC) were calculated for EOD (hereinafter, MSE-EOD and ICC-EOD). In addition, the MSE and ICC were calculated for PERCLOS calculated from EOD (hereinafter, MSE-PERCLOS and ICC-PERCLOS). In addition, based on the EOD, it was determined whether the test image had the eyes open or closed, and the F-score calculated from the precision and recall was calculated as an index of the accuracy of the determination result. In addition, the frame rate (FPS) was calculated as an index of the processing speed. In FIG. 7, the MSE-EOD of each of the four types of prediction devices 30 is smaller than that of each of the six types of comparison systems, and it can be seen that the accuracy is high. However, since MSE-EOD indicates the average of the square error in the entire EOD, for example, in data in which the eyes are mostly open, the accuracy may be high even if the EOD of the closed eye area is not properly estimated. Therefore, it is necessary to refer to the values ​​of MSE-PERCLOS and / or F-score as well. It can be seen that the MSE-PERCLOS of each of the four types of prediction devices 30 is smaller than that of each of the six types of comparison systems, and has higher accuracy.

[0075] ICC is used to evaluate the degree of agreement of the predicted values, and its value ranges from -1 to 1. ICC approaches 1 in the case of perfect agreement and approaches 0 in the case of no agreement. In FIG. 7, it can be seen that the ICC-EOD of each of the four types of prediction devices 30 is larger than that of each of the six comparison systems, indicating high accuracy. It can also be seen that the ICC-PERCLOS of each of the four types of prediction devices 30 is larger than that of each of the six comparison systems, indicating high accuracy. When comparing ICC-EOD and ICC-PERCLOS, the latter is higher, which is thought to be due to the correct discrimination of open and closed eyes. In addition, the ICC-PERCLOS for each of the four types of prediction devices 30 is 0.8245 on average, which is higher than 0.75, indicating sufficient agreement.

[0076] In addition, here, in order to calculate the F-score, Positive is defined as closed eyes and Negative is defined as open eyes. This is because determining whether the eyes are closed is important for PERCLOS, which indicates the percentage of eyes closed during a certain period of time. In FIG. 7, the F-score of each of the four types of prediction devices 30 is larger than that of each of the six comparison systems, indicating higher accuracy. In addition, the FPS of each of the four types of prediction devices 30 is also larger than that of each of the six comparison systems, indicating higher processing speed. This is thought to be because each comparison system calculates EOD after extracting the contours of the eyelids and irises, while the prediction device 30 directly predicts EOD.

[0077] FIG. 8 is a diagram comparing the EOD predicted by the prediction device 30 and the comparison system. In FIG. 8, the upper part shows a partial image near the eye region extracted from the test image. The middle part shows the outline of the eyelid and iris extracted by the comparison system (ResNet18) from the partial image near the eye region. The numerical value GD in the lower part indicates the correct EOD included in the teacher data as the test image (i.e., the EOD output by FX3). The numerical value SM in the lower part indicates the EOD calculated by the comparison system (ResNet18). The numerical value RM in the lower part indicates the EOD predicted by the prediction device 30 (VGG19). In FIG. 8, it can be seen that the EOD (RM) predicted by the prediction device 30 (VGG19) is closer to the correct EOD (GD) than the EOD (SM) calculated by the comparison system (ResNet18).

[0078] FIG. 9 is a graph plotting the time series distribution of the correct and predicted values ​​of PERCLOS. In FIG. 9, the vertical axis indicates PERCLOS, and the horizontal axis indicates the time axis. The plot "True" indicates the correct PERCLOS output by FX3. The plot "Predicted" indicates the PERCLOS calculated based on the EOD output by the prediction device 30 (VGG19). As shown in FIG. 9, it can be seen that there is a high degree of agreement between the correct and predicted values, even from a visual point of view.

[0079] As can be seen from Figures 7 to 9, the prediction device 30 according to embodiment 3 of the present invention exhibited accuracy significantly exceeding the accuracy of the six comparison systems, regardless of which of the four types of deep learning model ML2 was used.

[0080] [Modifications] The performance of the first image capturing device 91 and the second image capturing device 92 in each of the above-mentioned embodiments does not necessarily have to be different. For example, the performance of the first image capturing device 91 and the second image capturing device 92 may not be significantly different or may be the same. In this case, the teacher data generating device 10 may determine from which of the first image and the second image the eye opening degree is to be obtained based on the accuracy of the eye opening degree obtainable from each eye opening degree, and obtain the eye opening degree from the determined image and associate it with the other image. For example, the teacher data generating device 10 may extract feature amounts for calculating the eye opening degree from each of the first image and the second image, and determine to obtain the eye opening degree from the image from which the feature amount can be extracted more accurately.

[0081] The teacher data generating device 10 does not necessarily have to determine from which of the first image and the second image the eye openness degree is to be obtained each time the teacher data generating process is performed. For example, the teacher data generating device 10 may determine from which of the first image and the second image the eye openness degree is to be obtained by obtaining the eye openness degree from each of a predetermined number of pairs of the first image and the second image and comparing the statistical values ​​of the accuracy. Then, after the determination, when performing the process of generating teacher data, the teacher data generating device 10 does not need to perform the process of determining from which image the eye openness degree is to be obtained, but can obtain the eye openness degree from one of the images determined in advance and associate it with the other image.

[0082] Furthermore, the teacher data in each of the above-described embodiments does not necessarily have to be generated by associating the eye opening degree obtained from one image with the other image. For example, the teacher data may be generated based on such two images. For example, the teacher data generating device 10 may generate teacher data by obtaining the eye opening degree from each of such two images, calculating a statistical value (e.g., an average value), and associating the statistical value with one or both of the two images.

[0083] In addition, the above-mentioned prediction device 30 is not necessarily limited to the deep learning model ML2 trained using the teacher data generated by the teacher data generating device 10, and may use other deep learning models. Such a deep learning model may be a model trained to output the degree of eye opening for an image including an eye as a subject. For example, such a deep learning model may be trained using a set of teacher data in which the degree of eye opening calculated by a human is associated with an image including an eye as a subject.

[0084] In addition, the prediction device 30 described above may be configured to predict any eye feature amount, not limited to the degree of eye opening. Such eye feature amounts include, but are not limited to, pupil diameter, gaze point position, gaze direction, gaze coordinates, eye movement speed, and the like. In this case, the teacher data may be generated by associating the eye feature amount obtained from one image, which has a higher accuracy of the eye feature amount obtainable based on each image, with the other image, among two images taken by two different imaging devices with the same eye as a subject. In addition, by reading the degree of eye opening as "eye feature amount" in the description of each of the above-mentioned embodiments, the configuration, operation, and effect thereof can be similarly described.

[0085] In addition, the prediction device 30 described above may be configured to predict any facial feature, not limited to eye feature. Examples of such facial feature include, but are not limited to, face direction and shaking, mouth opening, and facial muscle (wrinkle) state. In this case, the teacher data may be generated by associating the facial feature acquired from one image, which has a higher accuracy of the facial feature that can be acquired based on each image, with the other image, of two images taken by two different imaging devices with the same eye as a subject. In addition, by replacing the degree of eye opening in the description of each of the above-mentioned embodiments with "facial feature", the configuration, operation, and effect thereof can be similarly described.

[0086] [Software implementation example] The functions of the teacher data generation device 10, the model learning device 20, and the prediction device 30 (hereinafter, each referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, each part included in the control units 11, 21, 31).

[0087] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0088] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0089] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0090] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0091] 10. Teacher data generator 20 Model Learning Device 30 Prediction Device 11, 21, 31 Control section 12, 22, 32 storage section 111, 211, 311 Acquisition Department 112 Generation part 212 Learning Department 312 Prediction Department 313 Output section 91 First Camera 92 Second Camera 93 Third Camera ML1 Extraction Model ML2 Deep Learning Models S10 How to generate training data S20 model training method S30 Prediction Method

Claims

1. an acquisition unit for acquiring an examination image including an eye as a subject; A prediction unit that predicts an eye opening degree from the test image using a deep learning model that has been trained to output an eye opening degree when an image including an eye as a subject is input; A prediction device comprising:

2. The deep learning model is trained using training data in which an image including an eye as a subject is associated with an eye opening degree, The training data is generated based on two images of the same eye taken by two different imaging devices. The prediction device according to claim 1 .

3. The teacher data is generated by associating the degree of eye opening obtained from one of the two images, which has a higher degree of eye opening that can be obtained based on each image, with the degree of eye opening obtained from the other image. The prediction device according to claim 2 .

4. The two imaging devices have different performances, the one image is a first image captured by a first image capture device having higher performance of the two image capture devices, The other image is a second image captured by a second image capture device having lower performance of the two image capture devices. The prediction device according to claim 3 .

5. an acquisition unit that acquires two images captured by two different image capture devices of the same eye as a subject; A generating unit that generates teacher data in which an image including an eye as a subject is associated with an eye opening degree based on the two images; A teacher data generating device comprising:

6. An acquisition unit that acquires teacher data in which an eye opening degree is associated with an image including an eye as a subject; A learning unit that uses the teacher data to learn a deep learning model that receives an image including an eye as a subject and outputs an eye opening degree, The training data is generated by referring to two images of the same eye taken by two different imaging devices. Model learning device.

7. 1. A computer implemented prediction method comprising: An acquisition step in which the computer acquires an examination image including an eye as an object; A prediction step in which the computer predicts the degree of eye opening from the test image using a deep learning model that has been trained to input an image including an eye as a subject and output an eye opening degree; A method for predicting a prediction result.

8. A computer-implemented teacher data generation method, comprising: an acquisition step in which the computer acquires two images taken by two different image capture devices of the same eye as a subject; A generation step in which the computer refers to the two images and generates teacher data in which an image including an eye as a subject is associated with an eye opening degree; A method for generating teacher data, comprising:

9. 1. A computer-implemented method for model training, comprising: An acquisition step in which the computer acquires teacher data in which an image including an eye as a subject is associated with an eye opening degree; A learning step in which the computer learns a deep learning model that inputs the image and outputs an eye opening degree by using the teacher data, The training data is generated by referring to two images of the same eye taken by two different imaging devices. Model training methods.

10. A program causing a computer to function as the prediction device according to claim 1, the program causing the computer to function as the acquisition unit and the prediction unit.

11. A program causing a computer to function as the teacher data generating device according to claim 5, the program causing the computer to function as the acquisition unit and the generation unit.

12. 7. A program for causing a computer to function as the model learning device according to claim 6, the program causing the computer to function as the acquisition unit and the learning unit.

Images