Blinking estimation device, learning device, blinking estimation method, learning method, and program

Abstract

The present invention uses image information that indicates a motion of an eyelid, estimates the time during which the eyelid is performing the motion that has the spontaneous physiological characteristics of blinking, and outputs information that indicates the time.

Description

Technical Field

[0001] The present invention relates to a technique for detecting blinking (winking).

Background Art

[0002] A technique for detecting the time when blinking occurs from the image information obtained by an eye camera is known (see, for example, Non-Patent Document 1, etc.).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, it is difficult to accurately estimate the time when blinking occurs under various environments.

[0005] The present invention provides a technique for accurately estimating the time when blinking occurs even under various environments.

Means for Solving the Problems

[0006] Using image information representing the movement of the eyelids, estimate the time when the eyelids are performing a movement having the physiological characteristics of spontaneous blinking, and output information representing the time.

Effects of the Invention

[0007] This allows for accurate estimation of the time when blinking occurs, even under diverse environmental conditions. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1A is a block diagram illustrating the functional configuration of the learning device according to the embodiment. Figure 1B is a block diagram illustrating the functional configuration of the blink estimation device according to the embodiment. [Figure 2] Figure 2 is a block diagram illustrating the functional configuration of the blink estimation device according to the embodiment. [Figure 3] Figure 3 is a block diagram illustrating the hardware configuration of the device according to the embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. [First Embodiment] First, a first embodiment of the present invention will be described. <Structure> As illustrated in Figure 1A, the learning device 11 of this embodiment has memory units 111, 113 and a learning unit 112, and obtains an estimated model 113a by a learning process (machine learning process) using the learning data 111a.

[0010] As illustrated in Figure 1B, the blink estimation device 12 of this embodiment includes a confidence estimation unit 121, a blink estimation unit 122, and a memory unit 123. It uses the estimation model 113a obtained by the learning device 11 and the information representing the eyelid movement of the user 100 obtained by the acquisition device 13 to obtain and output the blink estimation result of the user 100.

[0011] The acquisition device 13 can be any device that acquires information representing the eyelid movements of user 100. For example, the acquisition device 13 may be an eye camera, eye tracker, or image sensor that acquires image information (video information) representing the eyelid movements of user 100; a biosensor that acquires biosignals (e.g., electromyographic signals) associated with the eyelid movements of user 100; or a sensor (e.g., a position sensor, velocity sensor, or acceleration sensor) that acquires the position, velocity, and acceleration of user 100's eyelids. Furthermore, the acquisition device 13 may be a device that acquires information representing the eyelid movements of both of user 100's eyes; or a device that acquires information representing the eyelid movements of one of user 100's eyes.

[0012] <Learning Process> Next, the learning process using the learning device 11 (Figure 1A) of this embodiment will be described. The learning device 11 of this embodiment is a device that learns an estimation model 113a for estimating the time it takes to perform a movement that has the physiological characteristics of spontaneous blinking from information representing eyelid movement.

[0013] Blinking is a type of eyelid opening and closing movement in animals (including humans), and it can be broadly classified into conscious and unconscious blinks. Conscious blinking is called "voluntary blinking." Unconscious blinking, on the other hand, can be further divided into two types: "reflexive blinking" (blinking when something is flying around the eyes, for example) and "spontaneous blinking" (blinking unconsciously and naturally). One of the physiological characteristics of blinking is the duration of the opening and closing movement, and the duration of the opening and closing movement during blinking is often within a certain range. Generally, the duration of reflexive blinking is shorter than that of spontaneous blinking. Also, the duration of voluntary blinking is at least as long as or longer than that of spontaneous blinking. There is no upper limit to the duration of voluntary blinking. By utilizing these characteristics, if the eyelids perform an opening and closing motion for a duration physiologically corresponding to blinking, it can be estimated that the motion is a blink; otherwise, it can be estimated that it is not a blink. This allows for accurate estimation of the time of blinking even under diverse environmental conditions. Here, the duration of reflexive blinks, spontaneous blinks, and short voluntary blinks all fall within the range of 40ms to less than 500ms. Assuming that reflexive blinks and short voluntary blinks do not occur, blinks with durations in the range of 40ms to less than 500ms can be extracted as spontaneous blinks. By processing the data to extract spontaneous blinks in this way, false detections due to image problems caused by ambient light and vibration can be suppressed, and the time of blinking can be accurately estimated under diverse environmental conditions. Note that the opening and closing motion of the eyelids is a series of movements that transition from an open eye state to a closed eye state, and then back to an open eye state. Closed eye state is when the eyelids are closed, and open eye state is when the eyelids are open. For example, the duration of the opening and closing motion is the duration of this series of movements from the point in time when the transition from the open eye state to the closed eye state begins, to the point in time when the transition from the closed eye state back to the open eye state is completed.

[0014] Another physiological characteristic of blinking is that both eyelids open and close simultaneously. This means that, for example, both eyelids may open and close at the same time or nearly simultaneously, or both eyes may close simultaneously or nearly simultaneously. By utilizing this characteristic, if both eyelids open and close simultaneously, it can be inferred that the movement is a blink; otherwise, it can be inferred that it is not a blink. This allows for accurate estimation of the time of blinking, even under diverse environmental conditions.

[0015] It is also possible to estimate blinking by combining the two physiological characteristics mentioned above. That is, if both eyelids perform an opening and closing movement for a duration that corresponds to a physiologically spontaneous blink, then that movement can be estimated as a blink; otherwise, it can be estimated as not a blink. By combining the two physiological characteristics mentioned above, it is possible to estimate the time when a blink occurred with greater accuracy, even under diverse environmental conditions.

[0016] The estimation model 113a of this embodiment is a model that estimates a degree of confidence representing the degree of certainty that the eyes are closed or open, based on information representing eyelid movement. As described later, such a degree of confidence can be used to determine whether or not the opening and closing movement of the eyelids has the physiological characteristics of blinking described above. Therefore, the estimation model 113a of this embodiment can be considered a model for estimating the time during which a movement having the physiological characteristics of blinking described above is performed, based on information representing eyelid movement. The degree of confidence may be a discontinuous value expressed as a binary value, a discontinuous value expressed as a triplicate or multi-value discontinuous value, or a continuous value. A higher degree of confidence may indicate a higher certainty (probability) that the eyes are closed, a lower degree of confidence may indicate a higher certainty that the eyes are closed, a higher degree of confidence may indicate a higher certainty that the eyes are open, and a lower degree of confidence may indicate a higher certainty that the eyes are open. The estimation model 113a can be any model that takes information representing eyelid movement as input, obtains such a degree of confidence, and outputs it. The estimation model 113a may be, for example, a model based on deep learning, a hidden Markov model, a support vector machine, or any other known classifier. For example, as the estimation model 113a, a Deep Convolutional Neural Network (DCNN) can be used, which obtains and outputs a confidence score representing the degree of certainty that the eyes are closed based on information representing each frame image of a video representing eyelid movement. However, this does not limit the present invention.

[0017] The memory unit 111 stores training data 111a for obtaining an estimated model 113a through a training process. The training data 111a depends on the estimated model 113a and includes at least information representing eyelid movements for training. The information representing eyelid movements for training is, for example, time-series information. For example, the information representing eyelid movements for training may be image information representing eyelid movements, biological signals associated with eyelid movements, or eyelid position, velocity, acceleration, etc. The training data 111a may be supervised training data or unsupervised training data. For example, the estimated model 113a is a pair of information representing eyelid movements for training and corresponding correct labels. For example, if the estimation model 113a is a DCNN that obtains and outputs a confidence score representing the degree of certainty that the eyes are closed, based on information representing each frame image of a video representing eyelid movement, then the training data 111a consists of each frame image of a video representing eyelid movement for training (image information representing eyelid movement for training) and the corresponding ground truth label. The ground truth label in this embodiment may be a label representing whether the eyes are closed or open, a label representing the confidence score, or a label representing a function value of the confidence score. Note that whether the eyes are closed or open and the confidence score are determined based on a unified standard. For example, if the pupil is completely hidden, the eyes may be considered closed, and otherwise, they may be considered open.

[0018] The learning unit 112 executes learning processing using the learning data 111a stored in the storage unit 111, obtains an estimation model 113a, and stores it in the storage unit 113. This learning processing may be of any kind. For example, the learning unit 112 may perform learning processing using only the learning data 111a stored in the storage unit 111, or may perform transfer learning using the learning data 111a stored in the storage unit 111 based on a large-scale pre-trained DCNN such as Resnet-50. In transfer learning, for example, information included in the learning data 111a is changed, noise is added, a part is removed, it is moved, rotated, etc., and the result is added as new learning data (so-called data augmentation), and the estimation model 113a is learned using the original learning data 111a and the new learning data. For example, when the learning data 111a is a set of image information representing eyelid movement for learning (for example, each frame image of a video representing eyelid movement) and corresponding correct labels, new image information (for example, frame image) obtained by changing the brightness or color of the image information (for example, the frame image), adding noise, removing a part, moving it, rotating it, etc., and the original image information (for example, frame image), and their corresponding correct labels may be added as new learning data.

[0019] The estimation model 113a obtained as described above is also stored in the storage unit 123 of the blink estimation device 12 (FIG. 1B).

[0020] <Blink Estimation Processing> Next, the blink estimation processing by the blink estimation device 12 (FIG. 1B) of the present embodiment will be described. The acquisition device 13 acquires information representing the movement of the eyelids of the user 100. The information representing the movement of the eyelids of the user 100 is, for example, time-series information. For example, the acquisition device 13 may acquire image information representing the movement of the eyelids of the user 100, may acquire a biological signal accompanying the movement of the eyelids of the user 100, or may acquire the position, velocity, acceleration, etc. of the eyelids of the user 100. However, the type of information representing the movement of the eyelids of the user 100 is the same as the type of information representing the movement of the learning eyelids included in the aforementioned learning data 111a. For example, if the estimation model 113a is a DCNN that obtains and outputs a confidence level representing the degree of confidence of being closed based on information (image information representing the movement of the eyelids) representing each frame image of a video representing the movement of the eyelids, the acquisition device 13 acquires each frame image of the video representing the movement of the eyelids of the user 100 (image information representing the movement of the eyelids of the user 100). The acquisition device 13 may acquire information representing the movement of the eyelids of both eyes of the user 100, or may acquire information representing the movement of the eyelids of one eye of the user 100. However, as will be described later, if the blink estimation unit 122 uses the physiological feature of blinking that both eyelids of both eyes perform opening and closing movements, the acquisition device 13 needs to acquire information representing the movement of the eyelids of both eyes of the user 100. The information representing the movement of the eyelids of the user 100 acquired by the acquisition device 13 is input to the confidence estimation unit 121 (step S13).

[0021] The confidence estimation unit 121 applies the information representing the eyelid movements of the input user 100 to the estimation model 113a extracted from the memory unit 113a, and obtains and outputs a confidence score representing the degree of certainty that user 100's eyes are closed or open. For example, if the information representing the eyelid movements of user 100 input to the confidence estimation unit 121 is time-series information, the confidence estimation unit 121 outputs time-series information of the confidence score corresponding to the time-series information representing the eyelid movements of user 100. If the information representing the eyelid movements of both of user 100's eyes is input to the confidence estimation unit 121, the confidence estimation unit 121 may obtain and output a confidence score corresponding to each eyelid of both eyes, or it may obtain and output a confidence score corresponding to the eyelid of either eye. However, as will be described later, if the blink estimation unit 122 utilizes the physiological characteristic of blinking, that both eyelids open and close, then the confidence estimation unit 121 needs to obtain and output a confidence level corresponding to each eyelid of each eye. When information representing the movement of one eyelid of the user 100 is input to the confidence estimation unit 121, the confidence estimation unit 121 obtains and outputs a confidence level corresponding to that eyelid. The confidence level output from the confidence estimation unit 121 is input to the blink estimation unit 122 (step S121).

[0022] The blink estimation unit 122 estimates the time during which the user's eyelids perform a movement with the physiological characteristics of spontaneous blinking (for example, the time interval during which this movement is performed, or any point in time within that time interval (for example, the start time, end time, center time, or a combination of the start and end times)) based on the input confidence level and the physiological characteristics of blinking described above, and outputs information representing this time as the estimation result (step S122). A specific example of this process is shown below. However, these are merely examples and do not limit the present invention.

[0023] <Specific Example 1> Specific example 1 assumes that the acquisition device 13 acquires information representing the movement of the eyelids of both eyes of the user 100, and the confidence estimation unit 121 outputs a confidence level corresponding to each eyelid of both eyes. The blink estimation unit 122 then estimates the time during which both eyelids are performing an opening and closing movement of a duration corresponding to a physiologically spontaneous blink, and outputs this time as the estimated result.

[0024] The blink estimation unit 122 receives the confidence levels of the right eye, CR(0),...,CR(T-1), and the confidence levels of the left eye, CL(0),...,CL(T-1), as input. Here, T is a positive integer, and t=0,...,T-1 are integer indices representing time, with larger t representing more recent time. First, the blink estimation unit 122 uses an appropriate threshold to binarize the confidence levels of the right eye, CR(0),...,CR(T-1), to obtain the binarized confidence levels of the right eye, CR'(0),...,CR'(T-1), and then binarizes the confidence levels of the left eye, CL(0),...,CL(T-1), to obtain the binarized confidence levels of the left eye, CL'(0),...,CL'(T-1). For example, the blink estimation unit 122 uses one threshold TH and sets CR'(t)=1 if CR(t)>TH, CR'(t)=0 if CR(t)≦TH, CL'(t)=1 if CL(t)>TH, and CL'(t)=0 if CL(t)≦TH. Alternatively, the blink estimation unit 122 may obtain binarization confidence CR'(0),...,CR'(T-1) and binarization confidence CL'(0),...,CL'(T-1) by threshold processing that prevents chattering using hysteresis. In this case, for example, the blink estimation unit 122 uses two thresholds, an upper limit THU and a lower limit THL (where THU>THL), and sets CR'(0)=1 if CR(0)>THU and CR'(0)=0 if CR(0)≦THU. Furthermore, the blink estimation unit 122 sets CR'(t)=1 for t=1,...,T-1 if CR'(t-1)=1 and CR(t)>THL, CR'(t)=0 if CR'(t-1)=1 and CR(t)≦THL, CR'(t)=1 if CR'(t-1)=0 and CR(t)>THU, and CR'(t)=0 if CR'(t-1)=0 and CR(t)≦THU. Similarly, in this case, for example, the blink estimation unit 122 uses two thresholds, an upper limit THU and a lower limit THL, and sets CL'(0)=1 if CL(0)>THU, and CL'(0)=0 if CL(0)≦THU.Furthermore, the blink estimation unit 122 sets CL'(t)=1 for t=1,...,T-1 if CL'(t-1)=1 and CL(t)>THL, CL'(t)=0 if CL'(t-1)=1 and CL(t)≦THL, CL'(t)=1 if CL'(t-1)=0 and CL(t)>THU, and CL'(t)=0 if CL'(t-1)=0 and CL(t)≦THU (step S1221-1).

[0025] Next, the blink estimation unit 122 extracts the time intervals IR(0),...,IR(R-1) from the binarization confidence CR'(0),...,CR'(T-1) of the right eye, which represent opening and closing movements of a duration corresponding to physiologically spontaneous blinking, and extracts the time intervals IL(0),...,IL(L-1) from the binarization confidence CL'(0),...,CL'(T-1) of the left eye, which represent opening and closing movements of a duration corresponding to physiologically spontaneous blinking. The duration of the opening and closing movements is determined based on predetermined criteria. However, R and L are positive integers less than or equal to T. For example, if the binarization confidence level is 1 representing eye opening and 0 representing eye closing, the duration of the opening and closing motion may be the time of a continuous interval of 0s (for example, the "00...000" interval in ...100...0001...) or the duration of the opening and closing motion may be the sum of the duration of the continuous interval of 0s and the duration of a predetermined number of 1s before and after it (for example, the "11100...000111111" interval in ...1111100...000111111...). The duration corresponding to a physiologically spontaneous blink is, for example, 40 ms or more and less than 500 ms (step S1222-1).

[0026] Next, the blink estimation unit 122 uses the time intervals IR(0),...,IR(R-1) in which the right eye performs an opening and closing motion for a duration corresponding to a physiologically spontaneous blink, and the time intervals IL(0),...,IL(L-1) in which the left eye performs an opening and closing motion for a duration corresponding to a physiologically spontaneous blink, to obtain the time intervals I(0),...,I(K-1) in which both eyes perform an opening and closing motion for a duration corresponding to a physiologically spontaneous blink, and outputs information representing these time intervals I(0),...,I(K-1) as an estimation result (an estimation result representing the time during which blinking occurs). Here, K is a positive integer less than or equal to T. Also, the time interval I(k) (where k∈{0,...,K-1}) may be a point in time or an interval. For example, if there exists a time interval IL(i) that coincides with or approximates the time interval IR(r) (where r∈{0,...,R-1} and i∈{0,...,L-1}), the blink estimation unit 122 may use the time interval IL(i) that coincides with or approximates the time interval IR(r) as time I(k) (where k∈{0,...,K-1}), or it may use any point in time belonging to the time interval IL(i) (for example, the start time, end time, central time, or a pair of the start and end times of the time interval IL(i)) as time I(k), or it may use the time interval IR(r) as time I(k), or it may use any point in time belonging to the time interval IR(r) as time I(k). Alternatively, for example, the blink estimation unit 122 may define time I(0),...,I(K-1) as the time interval in which time intervals IR(0),...,IR(R-1) and IL(0),...,IL(L-1) overlap, or it may define time I(k) as any point in time belonging to the overlapping time interval. That is, time I(0),...,I(K-1) may define time intervals belonging to both time intervals IR(0),...,IR(R-1) and time intervals IL(0),...,IL(L-1), or it may define time I(k) as any point in time belonging to both time intervals IR(0),...,IR(R-1) and time intervals IL(0),...,IL(L-1). If the time intervals IR(0),...,IR(R-1) or IL(0),...,IL(L-1) do not exist, the blink estimation unit 122 may output an estimation result indicating that there is no time during which blinking occurs.Alternatively, if the time intervals IR(0),...,IR(R-1) do not exist, the blink estimation unit 122 may use the time interval IL(i), as time I(k), or any point in time belonging to the time interval IL(i) as time I(k). Similarly, if the confidence levels CL(0),...,CL(T-1) do not exist, the blink estimation unit 122 may use the time interval IR(r) as time I(k), or any point in time belonging to the time interval IR(r) as time I(k) (step S1223-1).

[0027] It should be noted that, depending on environmental conditions such as lighting, it may not be possible to obtain highly reliable confidence levels CR(0),...,CR(T-1) or CL(0),...,CL(T-1). In such cases, the blink estimation unit 122 may use only the confidence levels CR(0),...,CR(T-1) and CL(0),...,CL(T-1) that meet the reliability criteria, and obtain time I(0),...,I(K-1) as described above. Alternatively, if the reliability of confidence levels CR(0),...,CR(T-1) meets the criteria, but the reliability of confidence levels CL(0),...,CL(T-1) does not, the blink estimation unit 122 may use the time interval IR(r) as time I(k), or any point in time within the time interval IR(r) as time I(k). Similarly, if the reliability of confidence levels CL(0),...,CL(T-1) meets the criterion, but the reliability of confidence levels CR(0),...,CR(T-1) does not, the blink estimation unit 122 may use the time interval IL(i) as time I(k), or any point in time within the time interval IL(i) as time I(k). Furthermore, if neither the reliability of confidence levels CR(0),...,CR(T-1) nor the reliability of confidence levels CL(0),...,CL(T-1) meets the criterion, the blink estimation unit 122 may perform error processing, such as not outputting the estimation result. Note that the reliability of the confidence levels can be anything, but for example, if the magnitude of the information representing the eyelid movement of the corresponding user 100 (e.g., the brightness of the image information) is below the saturation value, the reliability of the confidence level may be considered to meet the criterion, and otherwise, the reliability of the confidence level may be considered to not meet the criterion.

[0028] <Specific Example 2> In specific example 1, the blink estimation unit 122 binarized the confidence levels CR(0),...,CR(T-1) of the right eye and CL(0),...,CL(T-1) of the left eye (step S1222-1) before extracting the time I(0),...,I(K-1). However, the blink estimation unit 122 may obtain the time I(0),...,I(K-1) without binarizing either the confidence levels CR(0),...,CR(T-1) of the right eye or the confidence levels CL(0),...,CL(T-1) of the left eye.

[0029] In this case, the blink estimation unit 122 receives the confidence levels CR(0),...,CR(T-1) of the right eye and CL(0),...,CL(T-1) of the left eye as input. The blink estimation unit 122 extracts the time intervals IR(0),...,IR(R-1) from the confidence levels CR(0),...,CR(T-1) of the right eye, which represent opening and closing movements of a duration corresponding to physiologically spontaneous blinking, and extracts the time intervals IL(0),...,IL(L-1) from the confidence levels CL(0),...,CL(T-1) of the left eye, which represent opening and closing movements of a duration corresponding to physiologically spontaneous blinking. For example, the blink estimation unit 122 compares a predetermined threshold with confidence levels CR(0),...,CR(T-1) to detect time intervals IR(0),...,IR(R-1) in which opening and closing movements of a duration corresponding to physiologically spontaneous blinking are occurring, and compares a predetermined threshold with confidence levels CL(0),...,CL(T-1) to detect time intervals IL(0),...,IL(L-1) in which opening and closing movements of a duration corresponding to physiologically spontaneous blinking are occurring. For example, in cases where a higher confidence level indicates an open-eye state and a lower confidence level indicates a closed-eye state, the blink estimation unit 122 may extract time intervals in which the confidence level CR(t) is continuously below the threshold, and among these time intervals, those corresponding to a duration corresponding to physiologically spontaneous blinking may be designated as time intervals IR(0),...,IR(R-1). Similarly, in this case, the blink estimation unit 122 may, for example, extract time intervals in which the confidence level CL(t) is continuously below a threshold, and among these time intervals, those corresponding to the duration of physiologically spontaneous blinking may be designated as time intervals IL(0),...,IL(L-1) (step S1222-2).

[0030] Subsequently, the blink estimation unit 122 executes step S1223-1 and outputs information representing times I(0),...,I(K-1) as the estimation result. The rest is the same as in Specific Example 1.

[0031] <Specific Example 3> In specific example 3, the blink estimation unit 122 estimates the time during which one eyelid is performing an opening and closing movement that corresponds to a physiologically spontaneous blink, and outputs this time as the estimated result.

[0032] The blink estimation unit 122 receives at least one of the confidence levels of the right eye, CR(0),...,CR(T-1) and the confidence levels of the left eye, CL(0),...,CL(T-1). The blink estimation unit 122 uses an appropriate threshold to either binarize the confidence levels of the right eye, CR(0),...,CR(T-1), to obtain the binarized confidence levels of the right eye, CR'(0),...,CR'(T-1), or binarize the confidence levels of the left eye, CL(0),...,CL(T-1), to obtain the binarized confidence levels of the left eye, CL'(0),...,CL'(T-1), or obtain both the binarized confidence levels CR'(0),...,CR'(T-1) and the binarized confidence levels CL'(0),...,CL'(T-1) (step S1222-3).

[0033] The blink estimation unit 122 may extract a time interval IR(r) in which an opening and closing motion of a duration corresponding to a physiologically spontaneous blink occurs from the binarization confidence CR'(0),...,CR'(T-1) of the right eye as time I(k), or it may select any point in time within that time interval IR(r) as time I(k). Alternatively, the blink estimation unit 122 may extract a time interval IL(i) in which an opening and closing motion of a duration corresponding to a physiologically spontaneous blink occurs from the binarization confidence CL'(0),...,CL'(T-1) of the left eye as time I(k), or it may select any point in time within that time interval IL(i) as time I(k). Alternatively, the blink estimation unit 122 may use only confidence CR(0),...,CR(T-1) and confidence CL(0),...,CL(T-1) that satisfy the reliability criteria to obtain time I(0),...,I(K-1) as described above. Alternatively, if the reliability of confidence levels CR(0),...,CR(T-1) meets the criterion, but the reliability of confidence levels CL(0),...,CL(T-1) does not, the blink estimation unit 122 may use only the time interval IR(0),...,IR(R-1) to obtain time I(0),...,I(K-1) as described above. Similarly, if the reliability of confidence levels CL(0),...,CL(T-1) meets the criterion, but the reliability of confidence levels CR(0),...,CR(T-1) does not meet the criterion, the blink estimation unit 122 may use only the time interval IL(0),...,IL(L-1) to obtain time I(0),...,I(K-1) as described above. The blink estimation unit 122 outputs information representing the time I(0),...,I(K-1) obtained in this way as an estimation result. Furthermore, if neither the reliability of confidence levels CR(0),...,CR(T-1) nor the reliability of confidence levels CL(0),...,CL(T-1) meets the criteria, the blink estimation unit 122 may perform error processing such as not outputting the estimation result (step S1223-3).

[0034] <Specific Example 4> In specific example 3, the blink estimation unit 122 binarized the confidence scores CR(0),...,CR(T-1) of the right eye and / or the confidence scores CL(0),...,CL(T-1) of the left eye (step S1222-3) and then extracted the time I(0),...,I(K-1). However, the blink estimation unit 122 may obtain the time I(0),...,I(K-1) without binarizing the confidence scores CR(0),...,CR(T-1) of the right eye or the confidence scores CL(0),...,CL(T-1) of the left eye.

[0035] In this case, the blink estimation unit 122 is input with confidence levels CR(0),...,CR(T-1) for the right eye and / or confidence levels CL(0),...,CL(T-1) for the left eye. For example, the blink estimation unit 122 may compare a predetermined threshold with confidence levels CR(0),...,CR(T-1) and set time I(k) to a time interval IR(r) in which an opening and closing motion of a duration corresponding to a physiologically spontaneous blink is occurring, or set time I(k) to any point in time within that time interval IR(r). Alternatively, for example, the blink estimation unit 122 may compare a predetermined threshold with confidence levels CL(0),...,CL(T-1) and set time I(k) to a time interval IL(i) in which an opening and closing motion of a duration corresponding to a physiologically spontaneous blink is occurring, or set time I(k) to any point in time within that time interval IL(i) (see Specific Example 2). The rest is the same as in Specific Example 3.

[0036] <Specific Example 5> Specific example 5 assumes that the acquisition device 13 acquires information representing the movement of the eyelids of both eyes of the user 100, and the confidence estimation unit 121 outputs a confidence level corresponding to each eyelid of both eyes. The blink estimation unit 122 then estimates the time during which both eyelids are opening and closing, and outputs this time as the estimated result.

[0037] The blink estimation unit 122 receives the confidence levels of the right eye CR(0),...,CR(T-1) and the confidence levels of the left eye CL(0),...,CL(T-1) as input. First, the blink estimation unit 122 uses an appropriate threshold to binarize the confidence levels of the right eye CR(0),...,CR(T-1) to obtain the binarized confidence levels of the right eye CR'(0),...,CR'(T-1), and then binarizes the confidence levels of the left eye CL(0),...,CL(T-1) to obtain the binarized confidence levels of the left eye CL'(0),...,CL'(T-1) (step S1221-1).

[0038] Next, the blink estimation unit 122 extracts the time intervals IR(0),...,IR(R-1) in which the blinking motion is occurring from the binarization confidence CR'(0),...,CR'(T-1) of the right eye, and extracts the time intervals IL(0),...,IL(L-1) in which the blinking motion is occurring from the binarization confidence CL'(0),...,CL'(T-1) of the left eye. The difference from Specific Example 1 is that the time intervals in which the blinking motion is occurring are extracted as IR(0),...,IR(R-1) and IL(0),...,IL(L-1) regardless of whether or not the time length corresponds to a physiologically spontaneous blink (step S1222-5).

[0039] Next, the blink estimation unit 122 uses the time intervals IR(0),...,IR(R-1) in which the right eye is performing an opening and closing motion, and the time intervals IL(0),...,IL(L-1) in which the left eye is performing an opening and closing motion for a certain duration, to obtain the time intervals I(0),...,I(K-1) in which both eyes are performing an opening and closing motion, and outputs information representing these time intervals I(0),...,I(K-1) as an estimation result (an estimation result representing the time during which blinking is occurring). The difference from Specific Example 1 is that information representing the time intervals I(0),...,I(K-1) in which both eyes are performing an opening and closing motion is output as an estimation result, regardless of whether the duration corresponds to a physiologically spontaneous blink or not (step S1223-5).

[0040] <Specific Example 6> In specific example 5, the blink estimation unit 122 binarized the confidence scores CR(0),...,CR(T-1) of the right eye and / or the confidence scores CL(0),...,CL(T-1) of the left eye (step S1222-5) and then extracted the time I(0),...,I(K-1). However, the blink estimation unit 122 may obtain the time I(0),...,I(K-1) without binarizing either the confidence scores CR(0),...,CR(T-1) of the right eye or the confidence scores CL(0),...,CL(T-1) of the left eye.

[0041] In this case, the blink estimation unit 122 receives the confidence levels CR(0),...,CR(T-1) of the right eye and CL(0),...,CL(T-1) of the left eye as input. The blink estimation unit 122 extracts the time intervals IR(0),...,IR(R-1) in which the opening and closing motion is occurring from the confidence levels CR(0),...,CR(T-1) of the right eye, and extracts the time intervals IL(0),...,IL(L-1) in which the opening and closing motion is occurring from the confidence levels CL(0),...,CL(T-1) of the left eye. The difference from Specific Example 2 is that the time intervals in which the opening and closing motion is occurring are extracted as IR(0),...,IR(R-1) and IL(0),...,IL(L-1) regardless of whether the duration corresponds to a physiologically spontaneous blink (step S1222-6).

[0042] Subsequently, the blink estimation unit 122 uses the time intervals IR(0),...,IR(R-1) during which the right eye is performing an opening and closing motion, and the time intervals IL(0),...,IL(L-1) during which the left eye is performing an opening and closing motion, to obtain the time intervals I(0),...,I(K-1) during which both eyes are performing an opening and closing motion, and outputs information representing these time intervals I(0),...,I(K-1) as an estimation result (an estimation result representing the time during which blinking is occurring) (step S1223-6).

[0043] <Experimental Results> Next, the experimental results of this embodiment will be illustrated. In this experiment, 9576 eye frame images representing eyelid movement were used for videos A and B respectively as information representing eyelid movement in the training data 111a, and the blink estimation unit 122 performed the processing of specific example 1. In step S1221-1, the blink estimation unit 122 binarized the confidence level using a threshold TH.

[0044] In video A, 764 blinks were detected by human visual observation, while the blink estimation device 12 detected 760 blinks. The median time difference between the blink start time observed by human visual observation and the blink start time detected by the blink estimation device 12 was 30.4 milliseconds. In video B, 171 blinks were detected by human visual observation, while the blink estimation device 12 detected 174 blinks. The median time difference between the blink start time observed by human visual observation and the blink start time detected by the blink estimation device 12 was 25.6 milliseconds. Both videos were acquired outdoors under challenging ambient light conditions for image processing, with significant differences in brightness in both light and dark directions, and shadows from eyelashes even appearing within a single image. In such cases, classical image processing detection methods could only barely detect some blinks by adjusting parameters. In contrast, the method of this embodiment achieved these excellent results without adjusting any parameters.

[0045] <Features of this embodiment> In this embodiment, image information representing eyelid movement is used to estimate the time during which the eyelid is performing a movement with the physiological characteristics of spontaneous blinking, and information representing that time is output. In this way, because this embodiment takes into account the physiological characteristics of spontaneous blinking, it is possible to accurately estimate the time at which blinking occurs even under diverse environments (e.g., diverse ambient light). Furthermore, by using the machine learning-based estimation model 113a, it is possible to assume diverse environments and further improve the estimation accuracy. In addition, by using transfer learning to train the estimation model 113a, it is possible to improve the estimation accuracy in diverse environments even when the amount of training data 111a is small.

[0046] [Second Embodiment] Next, a second embodiment of the present invention will be described. The estimation model 113a of the first embodiment was a model that estimated the degree of confidence that the eyes were closed or open, based on information representing eyelid movement. However, instead, a model may be used that estimates the time during which the eyelids are performing movements that have the physiological characteristics of spontaneous blinking, based on information representing eyelid movement. In the following, the differences from the first embodiment will be explained in detail, and the explanation of matters already described will be simplified using the same reference numerals.

[0047] <Structure> As illustrated in Figure 1A, the learning device 21 of this embodiment has storage units 111, 113 and a learning unit 212, and obtains an estimated model 213a by a learning process using the learning data 211a.

[0048] As illustrated in Figure 2, the blink estimation device 22 of this embodiment has a blink estimation unit 222 and a memory unit 123, and uses the estimation model 213a obtained by the learning device 21 and the information representing the eyelid movement of the user 100 obtained by the acquisition device 13 to obtain and output the blink estimation result of the user 100.

[0049] <Learning Process> Next, the learning process using the learning device 21 (Figure 1A) of this embodiment will be described. The learning device 21 of this embodiment is a device that learns an estimation model 213a for estimating the time it takes to perform a movement that has the physiological characteristics of spontaneous blinking from information representing eyelid movement.

[0050] The estimation model 213a of this embodiment is a model that estimates the time during which the eyelids perform movements with the physiological characteristics of spontaneous blinking, based on information representing eyelid movement. Specific examples of the physiological characteristics of spontaneous blinking are as described in the first embodiment. The estimation model 213a can be any model that takes information representing eyelid movement as input and outputs the time during which the eyelids perform movements with the physiological characteristics of spontaneous blinking. The estimation model 213a may be, for example, a model based on deep learning, a hidden Markov model, a support vector machine, or any other known classifier.

[0051] The memory unit 111 stores learning data 211a for obtaining an estimated model 213a through a learning process. The learning data 211a depends on the estimated model 213a and includes at least information representing eyelid movements for learning. Specific examples of information representing eyelid movements are as described in the first embodiment. The learning data 211a may be supervised learning data or unsupervised learning data. For example, the estimated model 213a is a pair of information representing eyelid movements for learning and corresponding ground truth labels. The ground truth labels in this embodiment may represent whether or not the eyelids are performing movements with the physiological characteristics of spontaneous blinking, or they may represent the probability that the eyelids are performing such movements, or they may represent a function value of that probability. Such ground truth labels can be generated, for example, using the estimation results obtained by inputting information representing eyelid movements for learning into the blink estimation device 12 of the first embodiment.

[0052] The learning unit 212 performs a learning process using the learning data 211a stored in the memory unit 111, obtains an estimated model 213a, and stores it in the memory unit 113. As described in the first embodiment, this learning process may be of any type.

[0053] As described above, the estimated model 213a is also stored in the memory unit 123 of the blink estimation device 22 (Figure 2).

[0054] <Blink Estimation Processing> Next, the blink estimation process using the blink estimation device 22 (Figure 2) of this embodiment will be explained. The acquisition device 13 acquires information representing the eyelid movements of user 100. The information representing the eyelid movements of user 100 acquired by the acquisition device 13 is input to the blink estimation unit 222 (step S23).

[0055] The blink estimation unit 222 applies information representing the eyelid movements of the input user 100 to the estimation model 213a extracted from the memory unit 113a, estimates the time during which the user 100's eyelids are performing movements that have the physiological characteristics of spontaneous blinking, and outputs information representing that time as the estimation result (step S222).

[0056] <Features of this embodiment> In this embodiment, image information representing eyelid movement is used to estimate the time during which the eyelid is performing a movement with the physiological characteristics of spontaneous blinking, and information representing that time is output. In this way, because this embodiment takes into account the physiological characteristics of spontaneous blinking, it is possible to accurately estimate the time when blinking occurs even under diverse environments (e.g., diverse ambient light). In particular, this embodiment uses an estimation model 213a that estimates the time during which the eyelid is performing a movement with the physiological characteristics of spontaneous blinking based on information representing eyelid movement. As a result, the output from the estimation model 213a can be used directly as the blink estimation result, simplifying the blink estimation process. Furthermore, since the blink estimation process can be managed using only the estimation model 213a stored in the blink estimation device 22 without considering thresholds, updating the blink estimation process is also easy. Moreover, by using an estimation model 213a based on machine learning, diverse environments (e.g., diverse ambient light) can be assumed, and the estimation accuracy can be further improved. In addition, by using transfer learning to train the estimation model 213a, the estimation accuracy in diverse environments can be improved even when the amount of training data 211a is small.

[0057] [Hardware configuration] The learning devices 11, 21 and blink estimation devices 12, 22 in each embodiment are devices configured by a general-purpose or dedicated computer, for example, equipped with a processor (hardware processor) such as a CPU (central processing unit) and memory such as RAM (random-access memory) and ROM (read-only memory), executing a predetermined program. That is, the learning devices 11, 21 and blink estimation devices 12, 22 in each embodiment have, for example, processing circuits configured to implement the respective parts they each possess. This computer may have one processor and memory, or it may have multiple processors and memories. This program may be installed on the computer, or it may be pre-recorded in ROM, etc. Furthermore, some or all of the processing units may be configured using electronic circuits that realize processing functions independently, rather than electronic circuits that realize their functional configuration by being loaded with a program, such as a CPU. Also, the electronic circuits that constitute one device may include multiple CPUs.

[0058] Figure 3 is a block diagram illustrating the hardware configuration of the learning devices 11, 21 and blink estimation devices 12, 22 in each embodiment. As illustrated in Figure 3, the learning devices 11, 21 and blink estimation devices 12, 22 in this example have a CPU (Central Processing Unit) 10a, an input unit 10b, an output unit 10c, a RAM (Random Access Memory) 10d, a ROM (Read Only Memory) 10e, an auxiliary storage device 10f, a communication unit 10h, and a bus 10g. The CPU 10a in this example has a control unit 10aa, an arithmetic unit 10ab, and a register 10ac, and performs various arithmetic processing according to various programs loaded into the register 10ac. The input unit 10b is an input terminal, keyboard, mouse, touch panel, etc., to which data is input. The output unit 10c is an output terminal, display, etc., to which data is output. The communication unit 10h is a LAN card, etc., controlled by the CPU 10a which has loaded a predetermined program. Furthermore, RAM 10d is an SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), etc., and has a program area 10da where a predetermined program is stored and a data area 10db where various data is stored. Furthermore, auxiliary storage device 10f is, for example, a hard disk, MO (Magneto-Optical disc), semiconductor memory, etc., and has a program area 10fa where a predetermined program is stored and a data area 10fb where various data is stored. Furthermore, bus 10g connects CPU 10a, input unit 10b, output unit 10c, RAM 10d, ROM 10e, communication unit 10h, and auxiliary storage device 10f so that information can be exchanged. CPU 10a writes the program stored in the program area 10fa of auxiliary storage device 10f to the program area 10da of RAM 10d according to the loaded OS (Operating System) program. Similarly, CPU 10a writes various data stored in the data area 10fb of auxiliary storage device 10f to the data area 10db of RAM 10d.The addresses on RAM 10d where the program and data are written are then stored in register 10ac of CPU 10a. The control unit 10aa of CPU 10a sequentially reads these addresses stored in register 10ac, reads the program and data from the area on RAM 10d indicated by the read addresses, sequentially has the calculation unit 10ab execute the calculations indicated by the program, and stores the calculation results in register 10ac. This configuration realizes the functional configuration of the learning devices 11, 21 and the blink estimation devices 12, 22.

[0059] The above-mentioned program can be recorded on a computer-readable recording medium. Examples of computer-readable recording media are non-transitory recording media. Examples of such recording media include magnetic recording devices, optical discs, magneto-optical recording media, and semiconductor memory.

[0060] The distribution of this program can be carried out, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing it in the storage device of a server computer and transferring it from the server computer to other computers via a network. As described above, a computer executing such a program may, for example, first store the program recorded on the portable recording media or the program transferred from the server computer in its own storage device. Then, when processing is to be executed, this computer reads the program stored in its own storage device and executes the processing according to the program it reads. Alternatively, as another form of execution of this program, the computer may directly read the program from the portable recording media and execute the processing according to that program, or it may sequentially execute the processing according to the program received each time a program is transferred to this computer from the server computer. Furthermore, the above processing may be executed by a so-called ASP (Application Service Provider) type service, which does not transfer the program from the server computer to this computer, but realizes the processing function only by issuing execution instructions and obtaining results. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the computer's processing).

[0061] In each embodiment, the device is configured by executing a predetermined program on a computer; however, at least a part of these processes may be implemented in hardware.

[0062] [Other variations] It should be noted that the present invention is not limited to the embodiments described above. For example, user 100 may be an animal other than a human. Furthermore, the various processes described above may not only be executed sequentially according to the description, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the processes. Needless to say, other modifications can be made as appropriate without departing from the scope of the claims. [Explanation of Symbols]

[0063] 11,21 Learning device 12,22 Blinking Estimation Device

Claims

1. A blink estimation device having a blink estimation unit that obtains a confidence score representing the degree of certainty that the eyes are closed or open, uses only the information representing the movement of the eyelid on the side whose confidence score meets a standard from among the information representing the movement of the eyelids of both eyes, estimates the time during which the eyelid is performing a movement having the physiological characteristics of spontaneous blinking, and outputs information representing the said time.

2. A blink estimation device according to claim 1, A blink estimation device that estimates the time during which the eyelid on the side that meets the aforementioned reliability criteria is performing an opening and closing movement for a duration corresponding to a physiologically spontaneous blink.

3. A blink estimation device according to claim 1, The blink estimation unit is a blink estimation device that estimates the time during which both eyelids of the eyes are performing opening and closing movements, provided that the reliability meets the criteria.

4. A blink estimation device according to claim 1, The blink estimation unit is a blink estimation device that estimates the time during which both eyelids of the eyes that meet the reliability criteria are performing an opening and closing movement for a duration corresponding to a physiologically spontaneous blink.

5. An estimation model for obtaining a confidence score representing the degree of certainty that the eyes are closed or open, and estimating the time during which the eyelids are performing movements with the physiological characteristics of spontaneous blinking, using only the information representing the movement of the eyelid on the side whose confidence score meets a criterion, from among the information representing the movement of the eyelids of both eyes. A learning device having a learning unit that learns and outputs.

6. A blink estimation method using a blink estimation device, A blink estimation method that obtains a confidence score representing the degree of certainty that the eyes are closed or open, uses only the information representing the movement of the eyelid on the side whose confidence score meets a criterion from among the information representing the movement of the eyelids of both eyes, estimates the time during which the eyelid is performing a movement that has the physiological characteristics of spontaneous blinking, and outputs information representing the said time.

7. A learning method using a learning device, An estimation model for estimating the time during which the eyelids perform movements with the physiological characteristics of spontaneous blinking, using only the information representing the movement of the eyelid on the side where the confidence level of the aforementioned confidence level meets a certain criterion, based on the confidence level of the eyelid movement among the information representing the movement of both eyelids. A learning method that learns and outputs.

8. A program for causing a computer to function as a blink estimation device according to any one of claims 1 to 4, or as a learning device according to claim 5.

Images