Blink detection device

Abstract

Blink detection device comprising: an ocular potential measuring unit having a pair of electrodes, the ocular potential measuring unit being configured to sequentially measure an eyelid state index value indicating a state between an open state and a closed state of eyelids in an eye of a person; and a signal processing device (3), wherein the signal processing device (3) is designed to: Determining, for each measured eyelid condition index value, that a blink occurs when a change in the measured eyelid condition index value from a baseline, which is a reference value of the eyelid condition index value, in the case that the person's eyelids are in the open state, exceeds a magnitude of a threshold; Saving the sequentially measured eyelid condition index values; Counting a number of times of a preliminary occurrence of blinks with respect to each of a plurality of preliminary thresholds, wherein the number of times of the preliminary occurrence of blinks is the number of times when the change in the eyelid condition index value from the baseline exceeds an order of magnitude of a corresponding preliminary threshold, for each of the eyelid condition index values sequentially measured and stored over a predetermined period of time, the orders of magnitude of the thresholds being different from each other; Creating a histogram of the number of times of preliminary blink occurrence with respect to the preliminary thresholds; and Setting the threshold to a preliminary threshold from the preliminary thresholds at which a change curve of the histogram is minimal, the change curve representing a change in the number of times of preliminary occurrence of blinks measured between successively applied preliminary thresholds in the histogram.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention

[0001] The present invention relates to a device that detects blinking (of the eyes) of a person, and more particularly, to a device that detects blinking in a waveform indicating an eyelid state in an opening-closing movement of eyelids in an electrooculogram (EOG) or the like of a person. 2. Description of the state of the art

[0002] Technologies for detecting a person's blinking, detecting drowsiness, and similar things have been proposed, since a person's blinking frequency and its change are linked to the person's degree of drowsiness. As a method for detecting a person's blinking, there are known an EOG (electro-oculography) method in which electrodes are attached near an eye and a potential difference (ocular potential) between the retina and the cornea is detected in response to the opening and closing of eyelids, as well as a method for capturing an image of the person's eyelids and detecting the opening and closing of the eyelids from the captured image. In short, in the EOG method, since during an opening and closing movement of the eyelids, as schematically shown in Fig. 6A, a temporary change in the ocular potential occurs, this change in the ocular potential is obtained from chronologically measured ocular potential data, thereby detecting a blink. Regarding this point, in "Automatic Detection of Eyeblinks and Analysis of Eyeblink Waveforms (A Computerized Identification and Data Analysis of Eyeblink EOG Waves)" Hiroaki YUZE; Hideoki TADA, Ergonomics (Japan Ergonomics Society), Vol. 30, No. 5, pages 331-337, as an algorithm for automatically detecting such an ocular potential change due to eye blinks, it is proposed to calculate a differential value of chronological ocular potential data and, as in Fig. 6B, detecting as blinking a waveform portion in which the value continuously exceeds thresholds on the negative side and the positive side within a predetermined time (approximately 0.2 seconds) in the waveform of the differential value of the ocular potential. Japanese Patent Application Publication No. 2017-42269 proposes a device that detects eye blinking in a waveform of chronological data of the differential value of the ocular potential and detects, as an eye blink waveform, a change in the differential value of the ocular potential when the differential value of the ocular potential changes from an upper threshold to a lower threshold within a predetermined time after exceeding the upper threshold, or when it changes from the lower threshold to the upper threshold within a predetermined time after falling below the lower threshold.In order to improve the detection accuracy of the eye blink waveform, using as the upper threshold and the lower threshold numbers obtained by multiplying coefficients for the thresholds, the coefficients being set based on frequency characteristic values indicating frequency characteristics of the differential value of the ocular potential by a standard deviation of the differential value of the ocular potential.

[0003] Furthermore, US 2016 / 0 304 099 A1 discloses a device for detecting a driver's drowsiness, which is designed to determine, during a driving break, whether the threshold values for detecting drowsiness are reset due to their necessity when a driving break occurs. US 2016 / 0 310 060 A1 teaches an eye detection method for detecting the degree of opening and closing of a subject's eyes, wherein a reference value of an eyelid condition index value is the value at which the subject's eyelids are in an open state. Finally, US 6 717 518 B1 discloses the generation of a histogram to identify each opening and closing of the eye and to determine features that indicate the person has fallen asleep. SUMMARY OF THE INVENTION

[0004] In the blink detection technologies described above, a configuration is generally used to chronologically measure an index value indicating an eyelid state or a position in an opening-closing movement of eyelids using a sensor for detecting an ocular potential, a camera for capturing an image of the eyelids, or the like, and to determine that a blink is occurring when a change in the index value at the time the eyelids are closed relative to the index value (baseline) at the time the eyelids are opened exceeds a "threshold" (that is, when the index value is shifted upward beyond the "threshold" in the case where the index value at the time the eyelids are closed is shifted upward from the index value (baseline) at the time the eyelids are opened).or when the index value is shifted downward beyond the "threshold" in the case where the index value at the time the eyelids are closed is shifted downward from the index value (baseline) at the time the eyelids are opened. In these blink detection technologies, regarding the setting of the "threshold," which is a criterion for the occurrence of blinks, a visual observation of an opening-closing movement of eyelids with a change in the index value is usually recorded in advance, and a value previously specified as the index value at the time the eyelids transition from the open state to the closed state is set as the "threshold" to be used for the actual detection of a person's blinks, for example, in the case of detecting the blinks of a driver driving a vehicle.

[0005] However, in the case where the threshold preset in the above-mentioned manner is set to a certain value, even if it is optimal immediately after the blink detection processing is started, there may be a case where it cannot be used for correct detection of the occurrence of blinks if the baseline, which is the reference of the index value, varies or a change width of the index value of the occurrence of blinks varies because of a factor of change in physical conditions of the person and various other factors after the actual blink detection processing for the person is started.Therefore, in order to prepare for such a case, it is advantageous for a blink detection device to have a configuration capable of appropriately resetting the threshold even after the actual blink detection processing for a person has been started.

[0006] Therefore, the present invention provides a configuration capable of setting an appropriate threshold even after blink detection processing is started in a blink detection device that chronologically measures an index value indicating an eyelid state in an opening-closing movement of eyelids and determines that blinking occurs when the index value is shifted upward or downward beyond the threshold.

[0007] Furthermore, in the above-mentioned device, it is preferable that the threshold adjustment be performed at a suitable timing after the start of the blink detection processing. Therefore, the present invention provides the above-mentioned device configured to be able to adjust the threshold at a proper timing.

[0008] A blink detection device according to an aspect of the present invention recited in claim 1 comprises an ocular potential measuring unit having a pair of electrodes, the ocular potential measuring unit being configured to sequentially measure an eyelid condition index value indicating a state between an open state and a closed state of eyelids in an eye of a subject, and a signal processing device. The signal processing device is configured to: determine, for each measured eyelid condition index value, that blinking is occurring when a change width of the measured eyelid condition index value from a baseline, which is a reference value of the eyelid condition index value, in the case where the subject's eyelids are in the open state, exceeds an order of magnitude of a threshold; store the sequentially measured eyelid condition index values;Counting a number of times of preliminary blink occurrences with respect to each of a plurality of preliminary thresholds, the number of times the change in the eyelid condition index value from the baseline exceeds an order of magnitude of a corresponding preliminary threshold, for each of the eyelid condition index values sequentially measured and stored over a predetermined period of time, the orders of magnitude of the preliminary thresholds being different from each other; Creating a histogram of the number of times of preliminary blink occurrences with respect to the preliminary thresholds;and setting the threshold to a preliminary threshold from the preliminary thresholds at which a change curve of the histogram is minimal, the change curve representing a change in the number of times of preliminary occurrence of blinks measured between successively applied preliminary thresholds in the histogram;

[0009] In the aspect discussed above, the "eyelid condition index value" may typically be an ocular potential. The "ocular potential measuring unit" measures a potential difference between at least one pair of electrodes arranged near an eye so that a potential difference between the retina and the cornea in response to the opening and closing of the eyelids can be measured in each aspect. A signal of the potential difference between the electrodes, i.e., the ocular potential, is subjected to AC-DC conversion and is then used for processing.In this case, based on the arrangement of the electrodes on a person, it is determined to which of the positive and negative sides the change in the ocular potential is directed relative to the ocular potential value when the eyelids are in the open state when blinking occurs, that is, when the eyelids transition from the open state to the closed state. It is understood that this aspect includes the case where the change in the ocular potential at the time of blinking is on the positive side, as well as the case where it is on the negative side. Specifically, the "eyelid state index value" may be a time differential value of the ocular potential.Furthermore, in another unclaimed aspect, the "eyelid condition index value" may be an index value indicating the degree of eyelid opening (the distance between the upper eyelid and the lower eyelid, or the position of the upper eyelid relative to the lower eyelid) in an image of a person's eye. In this case, a camera that captures an image of the person's eye may be used, and the degree of eyelid opening may be detected by detecting the positions of the upper eyelid and the lower eyelid in the image of the eye through any image processing technique.It should be noted that in this case, the question of which of the positive and negative sides a change in the degree of eyelid opening is directed towards when the eyelids transition from the open state to the closed state depends on the definition of the degree of eyelid opening, and in this aspect includes the case where the change in the degree of eyelid opening at the time of blinking is on the positive side, as well as the case where it is on the negative side. Specifically, the "eyelid state index value" may be a time differential value of the degree of eyelid opening. The key point is that the "eyelid state index value" is shifted when the eyelids transition from the open state to the closed state.

[0010] Furthermore, in the above-mentioned embodiment, the "baseline" may be a reference value of the eyelid condition index value in the case where a person's eyelids are in the open state, as mentioned above, and may typically be a time-averaged value or the like of the eyelid condition index values measured in the case where the eyelids are in the open state. Here, the eyelid condition index values measured in the case where the eyelids are in the open state may normally be eyelid condition index values previously measured during a period where it was visually confirmed that the eyelids are in the open state.In particular, when the eyelid condition index value is an ocular potential or its time differential value, the baseline can simply be set to 0 V as the reference value of the eyelid condition index value in the case where the eyelids are in the open state, and it is understood that this case also falls within the scope of this aspect. As mentioned above, the aforementioned eyelid condition index value is generally kept close to the baseline when the eyelids are in the open state, and when the eyelids transition to the closed state, that is, when blinking occurs, it changes by a certain width upward (toward the positive side) or downward (toward the negative side) beyond the baseline. Furthermore, the "threshold magnitude from the baseline" corresponds to a magnitude of the difference between the baseline and the threshold.Accordingly, in the device of this aspect, it can be determined that blinking is occurring when the change width of the eyelid state index value from the baseline exceeds the magnitude of the threshold from the baseline, as mentioned above. Specifically, in one embodiment, since the eyelid state index value is shifted to the positive side from the baseline upon blinking occurrence, the change width of the eyelid state index value from the baseline exceeds the magnitude of the threshold from the baseline when the eyelid state index value exceeds the threshold, it can be determined that blinking is occurring when the eyelid state index value exceeds the threshold.Furthermore, since when the eyelid condition index value is shifted to the negative side from the baseline when blinking occurs, the change width of the eyelid condition index value from the baseline exceeds the magnitude of the threshold from the baseline when the eyelid condition index value falls below the threshold, it can be determined that blinking occurs when the eyelid condition index value falls below the threshold.

[0011] However, as already mentioned, with regard to the "threshold" that is the criterion for blink occurrence in the configuration of the blink detection device according to this aspect mentioned above, there may be a case where, if it remains fixed at a certain value, the blink occurrence cannot be detected with excellent accuracy if the baseline varies or the change width of the index value upon blink occurrence varies. Specifically, with regard to the eyelid condition index value, while as long as no abnormal event occurs when blinking occurs, the eyelid condition index value shifts from the baseline by a maximum of an approximately fixed width (normally, the eyelid condition index value does not deviate much from the value upon blink occurrence), with regard to such an eyelid condition index value that deviates as mentioned above,If the magnitude of the threshold from the baseline is too small relative to the width of change in the eyelid condition index value from the baseline, where this width of change corresponds to the occurrence of blinking, this leads to a false detection in which the change in the eyelid condition index value without the occurrence of blinking is interpreted as a blink, and if the magnitude of the threshold from the baseline is too large, this leads to an erroneous detection of the occurrence of blinking.

[0012] Therefore, the above-mentioned device of this aspect enables updating the threshold to a correct value at the correct time, as mentioned above. More specifically, as mentioned above, the "signal processing device" is configured to: store the chronologically measured eyelid condition index value; count, with respect to each of the plurality of provisional thresholds whose provisional thresholds are different from each other, the number of times the change width of the eyelid condition index value from the baseline exceeds the magnitude of the provisional threshold from the baseline (the magnitude of the difference between the baseline and the provisional threshold) over a predetermined period of time; and set the threshold to the provisional threshold at which the change in the number of times with respect to the provisional threshold is minimal.That is, in the "signal processing device," first, the number of times the change width of the eyelid condition index value from the baseline exceeds the magnitude of the preliminary threshold from the baseline—the "number of preliminary blink occurrences"—is counted for each preliminary threshold while changing the preliminary threshold in the eyelid condition index values recorded over the specified period. Next, the preliminary threshold at which the change in the number of preliminary blink occurrences is minimal is detected, and this preliminary threshold is updated for the "threshold" that is the criterion for blink occurrence.

[0013] According to the configuration of the above-mentioned signal processing apparatus, the “threshold” can be updated to a more appropriate value even after the blink detection processing has started by using the eyelid state index values measured up to that point.That is, in the configuration of detecting the occurrence of blinks by the change width of the eyelid state index value from the baseline exceeding the magnitude of the threshold from the baseline, as in this aspect, as mentioned before, if the magnitude of the threshold from the baseline is too small, it will cause many false detections of the occurrence of blinks, and if the magnitude of the threshold from the baseline is too large, it will cause many erroneous detections of the occurrence of blinks; but if the "threshold" is the correct value, it can be considered that both false detections and erroneous detections of the occurrence of blinks are reduced, and the number of detection times is stabilized.Accordingly, the "threshold" at which the number of detected times is stabilized, that is, the preliminary threshold at which the change in the number of preliminary blink occurrences is minimal, can be used as the correct "threshold." As will be described in the Embodiments section below, practically a blink can be detected with excellent accuracy when using the "threshold" selected in the manner mentioned above.

[0014] In the aspect discussed above, the signal processing device may be configured to count, with respect to each preliminary threshold, the number of times the change width of the eyelid condition index value from the baseline exceeds the magnitude of the preliminary threshold from the baseline while changing the preliminary threshold or scanning the preliminary threshold at each predetermined width. In this case, in the signal processing device, in one aspect, a preliminary threshold at which a difference in the number of times the preliminary threshold is changed by the predetermined width is minimal may be selected as the threshold among the preliminary thresholds.Furthermore, in the signal processing device, in another aspect, a preliminary threshold at which a standard deviation of the number of times with respect to a predetermined number (for example, five) of the adjacent preliminary thresholds is minimal may be selected as the threshold among the preliminary thresholds. The point is that in the case of creating a histogram of the number of times the change width of the eyelid condition index value from the baseline exceeds the magnitude of the preliminary threshold from the baseline with respect to the preliminary threshold, the preliminary threshold at which the histogram is the flattest is sufficient to be selected as the threshold.

[0015] Furthermore, in the above-mentioned aspect, the "predetermined period" for the eyelid condition index value used to adjust the threshold can be set to any value by a designer or a user of the device. For example, the "predetermined period" may be the length of a predetermined time. In this case, the "length of the predetermined time" may be set to be the length of time during which an amount of eyelid condition index values with which the adjustment of the threshold can be achieved with excellent accuracy is included. Otherwise, it may be a period until the number of blink occurrences determined by the signal processing device reaches a predetermined number of times. Here, the "predetermined number of times" may be the number of blinks with which the adjustment of the threshold can be achieved with sufficient accuracy.Furthermore, in the above-mentioned aspect, the signal processing device may be configured to reset the threshold for each elapse of the predetermined period of time and may be configured to be able to update the threshold at the appropriate time.

[0016] As mentioned above, in the above-mentioned aspect, in the blink detection device that chronologically measures the eyelid condition index value and determines that blinking occurs when the change width of the eyelid condition index value from the baseline exceeds the magnitude of the threshold from the baseline, if a proper threshold is set, false detections and erroneous detections of blink occurrence are reduced, and based on the statistical knowledge that near the proper threshold the counted number of blink occurrences is stabilized, the threshold can be updated to a proper value.According to such an aspect, even more accurate blink detection can be achieved because, when performing blink detection of a driver driving a vehicle, blink detection of a person working at a desk, or the like, the threshold can be updated to a correct value at the correct time even if a baseline varies or a change width of an index value upon blink occurrence varies after the blink detection processing is started. Furthermore, according to this aspect, blink detection processing does not need to be suspended due to threshold updating because the threshold update processing can be executed in parallel during the execution of the blink detection processing, and blink detection can be performed consecutively for a long time.

[0017] Further objects and advantages of this aspect will become more apparent from the following description of preferred embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Features, advantages and technical and industrial significance of exemplary embodiments of the invention are described below with reference to the accompanying drawings, in which like reference numerals designate like elements and in which: Fig. 1A is a graph schematically showing the change of an ocular potential during the occurrence of blinks obtained by an EOG method; Fig. 1B is a diagram schematically showing an embodiment of a blink detection device of the present invention; Fig. 2A is a diagram schematically showing a situation of counting the number of times of preliminary blinks (the number of times an eyelid condition index value exceeds a preliminary threshold) for each magnitude of the preliminary threshold with respect to the eyelid condition index value (ocular potential) recorded over a predetermined period of time, according to the teachings of the present invention; Fig. 2B shows a histogram (bar chart) of the number of times of preliminary blinks obtained by plotting the number of times of preliminary blinks with respect to the preliminary threshold, where a curve ft presents the number of times of preliminary blinks and a curve Δft presents a change in the number of times of preliminary blinks with respect to the preliminary threshold; Fig. Figure 2C shows a typical example of a time change of the ocular potential and exemplifies that blinks can be detected with excellent accuracy when a preliminary threshold is set to the minimum value in the change of the number of times of preliminary blinks with respect to the preliminary threshold in Fig. 2B is set as a threshold; Fig. Figure 3A shows a situation of detecting the occurrence of blinks with visual confirmation, where the occurrence of blinks is determined based on the ocular potential measured by a sensor. In the figure, "OK" indicates that the occurrence of blinks with visual confirmation can also be detected based on the ocular potential (correct response), and "NG" indicates that the occurrence of blinks with visual confirmation cannot be detected based on the sensor's ocular potential (erroneous detection) and that a blink is erroneously detected based on the sensor's ocular potential (false detection); Fig. 3B is a graph obtained by plotting detection rates with respect to false detection rates, calculating the detection rate (the ratio of the number of blinks detected from the ocular potential relative to the number of blinks with visual confirmation) and the false detection rate (the ratio of the number of false detections relative to the number of blinks detected from the ocular potential) while changing the preliminary threshold for the ocular potential measured by the sensor. The preliminary threshold of the curve graph closest to the point with detection rate 1.0 and false detection rate 0 is an optimal threshold; Fig. 4 is a diagram showing, in flowchart form, an example of blink detection processing in the present embodiment; Fig. 5A is a diagram showing, in flowchart form, an example of threshold setting monitoring processing in the present embodiment; Fig. Fig. 5B is a diagram showing, in flowchart form, an example of threshold setting processing in the processing of Fig. 5A shows; Fig. Figure 6A is a graph schematically showing the change in the ocular potential upon blinking, obtained by the EOG method in the prior art. The reason why a change direction of the ocular potential is reverse to that in Fig. 1A is that an arrangement of a positive electrode and a negative electrode reverse to that in the case of Fig. 1A is; Fig. Figure 6B is a graph schematically showing a differential value of the change in ocular potential upon blinking, obtained by the EOG method in the prior art. The reason why a change direction of the ocular potential is reverse to that in Fig. 1A is that the arrangement of the positive electrode and the negative electrode is reversed to that in the case of Fig. 1A is; Fig. Figure 7A is a graph showing a typical time change of the ocular potential along with a threshold, which is a criterion for blink detection. An example is shown where, although in the left stage, ocular potential changes B in which blinks occur can be detected with the set threshold, in the right stage, the baseline rises, and false detections X increase with the set threshold; and Fig. Figure 7B is a graph showing a typical time change of the ocular potential along with a threshold, which is a criterion for blink detection. An example is shown where, although ocular potential changes B in which blinks occur can be detected with the set threshold in the left stage, in the right stage the baseline drops, and erroneous detections Y increase with the set threshold. DETAILED DESCRIPTION OF EMBODIMENTS

[0019] Hereinafter, various preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the figures, the same reference numerals designate the same parts. Design of the device

[0020] In one of preferred embodiments of a blink detection device of the present invention, similar to the case of JP 2017-42269 A as a basic embodiment, an ocular potential of a person is measured by the EOG method. As schematically shown in Fig. 1A, it is known that the ocular potential changes in response to an eyelid state in an opening-closing movement of eyelids, and that a potential difference manifests itself as a substantially fixed width between the case where the eyelids are in a fully opened state and the case where they are in a fully closed state (hereinafter, the case where the eyelids are in the fully opened state is referred to as the “opened state,” and the case where the eyelids are in the fully closed state is referred to as the “closed state”), since it can be referred to as an index value indicating the eyelid state, that is, an eyelid state index value, when a potential for the case where the subject's eyelids are in the open state is used for chronological data of the ocular potential as a reference value, that is, a baseline,is set, and when a shift width of the ocular potential thereof exceeds an order of magnitude of a threshold set to be approximately equal to or slightly smaller than the potential difference between the open state and the closed state, it is determined that the subject's eyelids transition to the closed state and that blinking occurs, and thereby the blinking can be detected.

[0021] With reference to Fig. 1B, in one configuration of the blink detection device of the present embodiment, first, at least a pair of electrodes EL1, EL2 are adhered around an eye and the eyelids 2 of a person's face 1, and a potential difference between the electrodes is sequentially sent as an ocular potential signal to a signal processing device 3. In the signal processing device 3, the ocular potential signal from the electrodes sequentially undergoes processing, such as digitization, in a "pre-processing unit" to be converted into a form adaptable for subsequent processing, and the ocular potential signal that has undergone the processing is sent to a "blink detection unit" and a "storage unit."In the blink detection unit, the ocular potential signal and a threshold are compared with each other in an aspect described later, and the occurrence of blinks is detected. Furthermore, the ocular potential signal sent to the storage unit is temporarily stored and accumulated therein, and is then used by a threshold setting unit to set the threshold used by the blink detection unit. As described in detail later, the threshold setting unit determines (and sets) the optimal threshold for detecting the occurrence of blinks using the ocular potential signal obtained over a predetermined period of time, and the set threshold is used by the blink detection unit.Furthermore, in the blink detection device of the present embodiment, even during the execution of the blink detection processing, a configuration for updating the threshold at predetermined periods by the threshold setting unit may be employed, and a monitoring unit for monitoring the elapse of the predetermined period may be provided. The monitoring unit may be configured to use the time from a timer and / or the number of blinks detected in the blink detection unit and output an instruction for executing threshold setting processing (threshold setting flag) to the threshold setting unit in an aspect described later.Furthermore, the result of the blink detection may be sent, for example, to any device (not shown) for determining drowsiness, or may be sent, for example, to a display (not shown) for display thereon. The signal processing device 3 may typically be configured as a computer device, and in a normal aspect, a CPU, a memory device, and an input / output (I / O) device coupled to each other via a bidirectional common bus (not shown) are included. The operation of the individual parts of the blink detection device is accomplished by the CPU executing a program.

[0022] To be more precise, Fig. 1A, regarding the ocular potential used as the eyelid state index value, a direction in which the ocular potential changes in the closed state of the eyelids relative to the opened state of the eyelids is on a positive side because the direction of the change is determined depending on an arrangement of the electrodes EL1, EL2 stuck on the face 1 of the person, but the direction in which the ocular potential changes in the closed state of the eyelids may also be on a negative side depending on the arrangement (see Fig. 6A; in such a case, the threshold is set on the lower side of the baseline). The eyelid state index value used for blink detection can be chronological data of ocular potentials themselves, as shown in the figure, or can be chronological data of their time differential values. In this case, since the change in the eyelid state index value upon blink occurrence occurs either on the positive side or the negative side (see Fig. 6B), the output line may be set to be substantially zero, and the threshold(s) may be set to the positive side or the negative side, or both. Furthermore, the eyelid condition index value may be a distance between an upper eyelid and a lower eyelid obtained by detecting positions of the upper eyelid and the lower eyelid by any image processing technique from an image of a person's eye captured by a camera (not shown). In this case, in the post-processing unit, processing up to measuring the distance between the upper eyelid and the lower eyelid from the camera image is performed, and chronological data of the distances between the upper eyelid and the lower eyelid are sent to the "blink detection unit" and the "storage unit."In the case where the distance between the upper eyelid and the lower eyelid is used as the eyelid state index value, the distance in the eyelid open state is set as the baseline, and the threshold is set as the distance (=0) corresponding to the eyelid closed state or a value slightly larger than it. It should be understood that each of these cases falls within the scope of the present invention. Principles of blink detection and threshold setting

[0023] In the blink detection by the device of the present embodiment, as mentioned above, when the change width of the sequentially measured eyelid condition index value from the baseline exceeds the magnitude of the threshold from the baseline, the occurrence of blinking is determined.Regarding this point, since in the ocular potential or the like of a person, which is practically used as the eyelid condition index value, the change width of the eyelid condition index value upon occurrence of blinking varies to a certain degree, and in addition to the change in the value due to blinking, a change due to another factor such as eyeball movement also occurs, the threshold is preferably set so that the risk of a change other than a blink being erroneously detected as a blink is as small as possible, that is, so that false detections are as small as possible, and so that an actually occurring blink can be detected as much as possible without missing a blink, that is, so that all blinking movements can be detected with fewer erroneous detections as much as possible.

[0024] Furthermore, as mentioned in "SUMMARY OF THE INVENTION", even if a certain threshold can be appropriately used for detecting the occurrence of blinks immediately after the start of the blink detection processing, there may be a case that it cannot be used for detecting the occurrence of blinks with excellent accuracy because of subsequent changes in the person's physical conditions and other changes in various circumstances. For example, as in waveforms of the ocular potential exemplified in Fig. 7A (waveforms in which the value in the eyelids closed state is shifted to the positive side of the value in the eyelids opened state), while on the left in the figure, only the waveform parts B in which blinks occur can be selectively detected, since all the waveform parts B in which blinks occur exceed the set threshold and variations of the ocular potential other than these fall below the threshold, as on the right in the figure, when the output line rises due to a factor during the execution of the blink detection processing, variations X of the ocular potential other than blinks also exceed the threshold, and false blink detections occur. In addition, as in waveforms of the ocular potential exemplified in Fig. 7B, while on the left side of the figure, only the waveform parts B in which blinking occurs can be selectively detected, on the right side of the figure, when the output line drops due to some factor during the execution of the blink detection processing, waveforms Y having slightly small amplitudes among the waveform parts B in which blinking occurs will fall below the threshold, resulting in erroneous detections. In order to select only the waveform parts B in which blinking occurs with the greatest possible accuracy, even if such circumstances change during the execution of the blink detection processing, it can be considered that the threshold should be updated to an approximate value at the appropriate time without being fixed to a specific value.

[0025] Now, it is assumed that in the waveforms of the ocular potential used as the eyelid condition index value, as exemplified in Fig. 7A and Fig. As shown in Figure 7B, scanning is performed from near the baseline in a direction where the magnitude of the threshold becomes large. While the magnitude of the threshold from the baseline is small, many false detections occur because variations in the ocular potential other than blinking exceed the magnitude of the threshold, as well as the waveform portions B where blinking occurs.

[0026] Nevertheless, when the magnitude of the threshold from the baseline becomes large to a certain extent, variations exceeding the magnitude of the threshold from the baseline only affect the waveform parts B where blinking occurs. Even if the threshold increases or decreases to a certain extent, the change in the number of times variations exceeding the magnitude of the threshold from the baseline becomes small. When the magnitude of the threshold from the baseline also becomes large, even the waveform parts B where blinking occurs do not exceed the magnitude of the threshold from the baseline, and many erroneous detections occur.The reason is that characteristics of ocular potential waveforms and the like used as the eyelid condition index value are that the amplitudes of the waveform parts B in which blinking occurs are substantially uniform (even with certain variations) even when the baseline varies, and are larger than ocular potential variation amplitudes other than blinking that overlap with the baseline by a certain width or more.Accordingly, focusing on the characteristics of the waveforms of the eyelid condition index value, in a waveform of the eyelid condition index value over a certain period of time, while the magnitude of a temporary threshold (provisional threshold) is gradually increased from the baseline, the number of times the magnitude of the provisional threshold is exceeded from the baseline is counted, a provisional threshold at which a change in the number of times the magnitude of the provisional threshold is exceeded from the baseline is most stable is detected, and thus it can be considered that this provisional threshold can be used as a correct threshold.

[0027] Therefore, the threshold setting in the present embodiment is based on the above-mentioned knowledge. More specifically, and as schematically shown in Fig. As shown in Figure 2A, first, in a waveform of the chronological data of the eyelid condition index value, such as the ocular potential (in the example shown, a waveform in which the value in the eyelid closed state is shifted to the positive side of the value in the eyelid open state), the preliminary threshold is increased from 0 mV at predetermined distances (the distances can be set to wide when the preliminary threshold is small and large, and the distances can be set to fine within a range in which the preliminary thresholds are assumed to be close to the correct threshold), the preliminary occurrence of blinking is determined when the magnitude of the eyelid condition index value from the baseline exceeds the magnitude of the preliminary threshold from the baseline for each preliminary threshold, and its number of times is counted. Then, as in Fig. 2B, for the number of times of preliminary blink occurrence obtained for each preliminary threshold, a histogram of the number of times of preliminary blink occurrence with respect to the preliminary threshold is prepared, and a preliminary threshold which gives the flattest point in a curve ft indicating the number of times of preliminary blink occurrence, that is, the smallest minΔft in a change curve Δft of the number of times of preliminary blink occurrence, can be selected as the correct threshold.

[0028] The flattest point in the curve ft indicating the number of times of preliminary blink occurrence, or the smallest minΔft in the change curve Δft of the number of times of preliminary blink occurrence, can be detected by any technique. In one aspect, a difference Δft of the number of times ft(at) of preliminary blink occurrence, where the preliminary thresholds a0, a1, ..., a t , ..., a n are defined as Δft=ft(at)−ft(at−1).

[0029] The preliminary threshold at that yields the minimum value minΔft of Δft may be selected as the appropriate threshold. In a further aspect, various standard deviations centered at ft(at), for example, standard deviations of ft(a t-2 ), ft(a t-1 ), ft(at), ft(a t+1 ), ft(a t+2), can be calculated for each of all preliminary thresholds, and the preliminary threshold at which the standard deviation is smallest can be selected as the correct threshold (since the smaller the change, the smaller the standard deviation).

[0030] In particular, the histogram in Fig. 2B is a histogram obtained using a real example of an ocular potential waveform in which the value in the closed state of the eyelids is shifted to the positive side of the value in their open state as the eyelid state index value, and the threshold giving minΔft was 76 mV. Fig. 2C is a part of the ocular potential waveform used to create the histogram in Fig. 2B was used, and as can be seen from the figure, it can be confirmed that only the waveform of the occurrence of blinks selectively exceeds the threshold when the preliminary threshold determined by the above-mentioned threshold setting technique is set as the threshold.

[0031] Further, in order to check the validity of the threshold determined by the above-mentioned threshold setting technique according to the present embodiment, as mentioned below, the occurrence of blinking detected from the chronological data of a person's ocular potential (data in which the value shifted from the baseline to the positive side at the occurrence of blinking) was detected, and the occurrence of blinking was visually confirmed from a video image around an eye of the person, this video image being recorded by a video camera, to check that a threshold correctly used for detecting the occurrence of blinking from chronological data of an ocular potential substantially agreed with the threshold determined by the above-mentioned threshold setting technique.

[0032] More specifically, the procedure of checking was as follows: (i) an image of the surroundings of a person's eye was taken by a video camera to record this video image simultaneously with the measurement of chronological data of an ocular potential of the person; (ii) periods in which blinking occurred (eye-closed periods) were visually checked and detected from the recorded video image of the video camera (upper part of Fig. 3A); (iii) while in the chronological data of the subject's ocular potential, the threshold was changed from 0 mV to 300 mV, for each threshold, periods of exceeding the threshold were detected as periods in which blinking occurred (eye-closed periods) (lower part of Fig. 3A); (iv) the eye-closed periods detected from the chronological data of the ocular potential were recorded, whereby the eye-closed periods were detected by visual inspection of the video camera image for each threshold; if both eye-closed periods overlapped at least partially with each other, a correct response (OK) was detected; if the eye-closed period detected by visual inspection of the video camera image was not detected from the chronological data of the ocular potential, an erroneous detection (NG) was detected; if the eye-closed period was not detected from the chronological data of the ocular potential in any detection of the eye-closed period by visual inspection of the video camera image, a false detection (NG) was detected, and they were counted individually (upper and lower parts of Fig. 3A); (v) a ratio of the number of correct responses (OK) relative to the number of eye-closed periods detected by visual inspection of the video camera image was calculated as a detection rate, and a ratio of the number of false detections relative to the number of eye-closed periods detected from the chronological data of the ocular potential was calculated as a false detection rate, for each threshold, meaning that a higher detection rate is preferred and a lower false detection rate is preferred; and (vi) as in Fig. As shown in Figure 3B, points were plotted for each threshold, with the horizontal axis representing the false detection rate and the vertical axis representing the detection rate. Furthermore, the curve point at the shortest distance from the point with a detection rate of 1.0 and a false detection rate of 0.0 was specified. It can be seen that the threshold at the curve point at the shortest distance from the point with a detection rate of 1.0 and a false detection rate of 0.0 is a proper value at which the false detection rate can be reduced as low as possible and the detection rate can be increased as high as possible.

[0033] Regarding the chronological data of the ocular potential, which is used to calculate the above-mentioned histogram in Fig. 2B, the threshold at the curve point at the shortest distance from the point with a detection rate of 1.0 and a false detection rate of 0.0 obtained by the above-mentioned verification method was 75 mV. This value is substantially equal to the threshold obtained by the threshold adjustment technique of the present embodiment, 76 mV, and accordingly, it could be shown that the threshold obtained by the threshold adjustment technique of the present embodiment is the correct threshold at which the false detection rate can be reduced as low as possible and the detection rate can be increased as high as possible.

[0034] It is understood that the threshold setting technique of the present embodiment is a technique obtained by focusing on the feature of the waveform of the eyelid condition index value as mentioned above, that is, the feature that the amplitudes of the waveform parts B in which blinking occurs are substantially uniform even when the baseline varies and are larger than variation amplitudes of the ocular potential other than blinking that overlap with the baseline by a certain width or more, and further that the processing of setting the threshold to the preliminary threshold at which a change in the number of times of preliminary blinking occurrence is minimal can be executed even when the blink detection processing is not suspended.Accordingly, in the apparatus of the present embodiment, the threshold can be updated at the appropriate timing after the start of the blink detection processing and even during its execution. Operation of the device

[0035] In the apparatus of the present embodiment, as mentioned above, blink detection processing ( Fig. 4), in the “blink detection unit”, comparing the eyelid condition index value with the threshold to detect the occurrence of blinking, and threshold setting monitoring processing ( Fig. 5A), in the “monitoring unit”, for counting the time at which the threshold is updated in the “monitoring unit”, are executed in parallel, and the threshold setting processing ( Fig. 5B) Setting the threshold in the “threshold setting unit” according to the threshold setting principle described above can be carried out at the right time.

[0036] The blink detection processing, which is presented in the form of a flowchart in Fig. 4 and executed in the "blink detection unit" can be executed each time the eyelid condition index value is sequentially read after a user instructs the device to start processing. Specifically, in the example shown in the figure, the case is described where the eyelid condition index value in the closed state of the eyelids rises above the baseline (the average of the eyelid condition index values in the open state of the eyelids), as shown in Fig. 1A, and the baseline is essentially 0 mV. Since the eyelid condition index value is shifted to the positive side from the baseline when blinking occurs, when an eyelid condition index value V i a threshold V th In this case, the range of change of the eyelid condition index value V i starting from the baseline, the magnitude of the threshold V th starting from the baseline (it is understood that the skilled person can equally bring about the case that the eyelid condition index value falls below the baseline when the eyelids are closed; in particular, in this case, if the eyelid condition index value V i the threshold V th falls below, the range of change of the eyelid condition index value V i starting from the baseline, the magnitude of the threshold V th starting from the starting line).

[0037] In the process of Fig. 4, the eyelid condition index value V i read from the preprocessing unit (step 1), and it is determined whether the threshold V th has already been set or not (step 2). If the threshold V th has not yet been set, the read eyelid condition index value V i recorded unchanged in the storage unit (step 7). On the other hand, if the threshold V th has been set, as described in detail later, with respect to the eyelid condition index value Vi-1 read in the previous cycle and the eyelid condition index value V i read in the current cycle determines whether the eyelid condition index value Vi−1<die Schwelle Vth undder Augenlidzustands−Indexwert Vi> the threshold Vth are met or not (step 3). If condition (1) is not met, the eyelid state is considered unchanged, and the process proceeds rapidly. If condition (1) is met, since this indicates that the eyelid state index value exceeds the threshold and the eyelids are changing from the open state to the closed state (that the change width of the eyelid state index value from the baseline exceeds the magnitude of the threshold from the baseline), it is determined that blinking is starting (step 4). Next, with respect to the eyelid state index value Vi-1 read in the previous cycle and the eyelid state index value V i , which is read in the current cycle, determines whether the eyelid condition index value Vi−1>the threshold Vth and the eyelid condition index value Vi <die Schwelle Vth are met or not (step 5). If condition (2) is not met, the eyelid state is considered unchanged, and the process proceeds rapidly. If condition (2) is met, since this indicates that the eyelid state index value changes from the state of exceeding the threshold to the state of falling below the threshold and the eyelids transition from the closed state to the open state, it is determined that the blinking ends (step 6). Specifically, at this stage, the number of blinks C t is incremented by one. Then the eyelid condition index value V i recorded in the storage unit (step 7).

[0038] According to the series of processes described above, the eyelid condition index value is only recorded until the threshold is set. Furthermore, after the threshold is set, if the eyelids are still in the open state, step 3 and step 5 are passed with NO. Then, when the eyelids transition from the open state to the closed state and blinking begins, step 3 and step 5 are passed with YES and NO, respectively. While the eyelids continue to be closed, step 3 and step 5 are passed with NO. When the eyelids transition from the closed state back to the open state and blinking ends, step 3 and step 5 are passed with NO and YES, respectively, thereby detecting the occurrence of blinking once. Then, the processing is repeatedly executed, thereby sequentially recording the eyelid condition index values.In addition, each time a blink occurs, it is detected, and the results of the number of times and frequency of blinking, variations in the intervals of occurrence of blinks, and the like can be used for drowsiness determination and the like.

[0039] The execution of the threshold setting monitoring processing, which is shown in the form of a flowchart in Fig. 5A can be performed simultaneously with the start of blink detection processing in Fig. 4. In the threshold setting monitoring processing mentioned above, in brief, the instruction execution processing of the processing of setting the threshold used for detecting blinks, which was mentioned above in an aspect to be discussed later, is executed for each lapse of a predetermined period of time. More specifically, in one aspect of the threshold setting monitoring processing, it is first determined whether the threshold V th has already been set or not (step 11). At the level where the threshold V th has not yet been set, a threshold setting time T CHECK , which corresponds to a recording time of the eyelid condition index value used to set the threshold, is set (step 12), and a measurement start time is set in T set recorded (step 13). Subsequently, the time T timerof the timer and the process remains in standby mode until Ttimer−Tset>TCHECK is fulfilled (step 14). During this time, the eyelid condition index value V i in the blink detection processing of Fig. 4. More specifically, if the eyelid condition index value is the ocular potential, as mentioned above, the threshold setting time T CHECK for example, 2200 seconds or the like. Then, if the condition (3) is satisfied, as described later, the execution of the setting processing of the threshold V th (step 15). If the threshold V th has been set, the threshold setting time in T set recorded (step 16), and the set threshold V th is used in blink detection processing in Fig. 4. In addition, if the threshold V thwas set, then in the threshold setting monitoring processing to the time T timer of the timer; the execution of the setting processing of the threshold V th can be instructed each time condition (3) is met, creating a new threshold V th can be determined sequentially, and the threshold required for blink detection processing in Fig. 4 can be updated sequentially.

[0040] The threshold setting processing, which is presented in the form of a flowchart in Fig. 5B, in response to the instruction to execute (step 15) the setting processing of the threshold V th by the threshold setting monitoring processing. In the threshold setting processing, an initial value V so to a provisional threshold V sset (step 21), and for the eyelid condition index values V i , which were sequentially accumulated in the storage unit during the period where the threshold setting time T CHECK in Fig. 5A, the number of times of preliminary blinking is taken as the number of times that the eyelid condition index value V i the provisional threshold V s exceeds, that is, the number of times N s , the Vi−1<Vs und Vi> vs is satisfied, is counted and recorded (step 22). More specifically, in the example shown in the figure, since the eyelid condition index value is shifted from the baseline to the positive side when blinking occurs, when the eyelid condition index value V i the provisional threshold V s exceeds, the range of change of the eyelid condition index value V istarting from the baseline, the magnitude of the preliminary threshold V s starting from the baseline. Then, until the preliminary threshold V s a final value V se (>V so ) (step 23), while the preliminary threshold V s at any given value ΔV s is increased (step 24), the number of times N s , which satisfies condition (4), for each provisional threshold V s counted and recorded (step 22). More specifically, if the eyelid condition index value V i the eyepiece potential is, the settings for example V so = 0 mV; ΔV s = 1 mV to 10 mV; V se = 300 mV and the like. Then, if V s ≥V se is fulfilled, as described in the description regarding Fig. 2B, a histogram of the number of times N sof the preliminary occurrence of blinks for the preliminary threshold V s be created, the provisional threshold V ss , where the change in the number of times N s of the preliminary occurrence of blinks is minimal, can be selected by one of the above-mentioned techniques (step 25), and the detected preliminary threshold V ss can be considered as the new threshold V th (step 26) and can be used for blink detection processing in Fig. 4 can be used.

[0041] Regarding the execution of the setting processing of threshold V th for each elapse of the threshold setting time T CHECK in the above-mentioned threshold setting monitoring processing of Fig. 5A, the frequencies of blinking depend on individual differences or differences within a person, and the number of times blinks occur for the threshold setting time TCHECK also depends on differences or differences within a person. Furthermore, in the process of creating the histogram, the number of times N s of the preliminary occurrence of blinks for the preliminary threshold V s in the chronological data of the eyelid condition index value and selecting the preliminary threshold V ss , where the change in the number of times N s of the preliminary occurrence of blinks is minimal, the accuracies of the histogram and the selected preliminary threshold V ssbetter because the number of occurrences of blinks in the chronological data of the eyelid condition index value to be referred to is greater, and in order to obtain sufficient accuracies, it is preferable that blinks occur more frequently than a predetermined number of times in the chronological data of the eyelid condition index value to be referred to. Accordingly, in the threshold setting monitoring processing of Fig. 5A, instead of updating the threshold V th for each elapse of the threshold setting time T CHECK , the threshold V th be updated each time the number of blink occurrences exceeds a predetermined number. It should be noted that in the case of this aspect, it is necessary that the number of blink occurrences can be detected with good accuracy to a certain extent before the threshold V this updated (for example, if the threshold is too large, it may take a very long time for the number of blink occurrences to exceed the predetermined number, and blinking may then be difficult to detect). Therefore, if the threshold is set first, the threshold setting processing can be executed at the stage after the threshold setting time T CHECK has elapsed, so that the threshold is set once, and then the threshold V th be updated each time the number of blinks occurs exceeds the specified number.

[0042] Accordingly, with further reference to Fig. 5A, in another aspect of the setting monitoring processing, when the threshold V th has not yet been set, the processing is similar to the above, and if the threshold V thhas already been set, as indicated by dashed lines in the figure, the process remains in standby until the detected number of blinks C t a given number of times C ts (step 17), and during this time the eyelid condition index value V i in the blink detection processing of Fig. 4. More specifically, if the eyelid condition index value is the ocular potential, as mentioned above, the specified number of times C ts for example 500 or something like that. Then, if C t > C ts is met, the execution of the setting processing of the threshold V th , as mentioned above, is instructed (step 15), and the threshold is determined by processing in Fig. 5B using the chronological data of the eyelid condition index value accumulated until C t > C tsis met, and after the threshold V th is set, the number of times of blinking C t reset to zero (step 16). Then, the execution of the setting processing of the threshold V th be instructed every time C t > C ts is met. This allows a new threshold V th be determined, and the threshold used in the blink detection processing of Fig. 4 can be updated.

[0043] As mentioned above, in the present embodiment, in brief, in the configuration of detecting, as the occurrence of blinking, a situation in which a change width of the eyelid condition index value from the baseline exceeds the magnitude of the threshold from the baseline as described above, the number of times of preliminary blinking occurrence is counted while changing a preliminary threshold in the chronological data of the eyelid condition index value; a histogram of the number of times of preliminary blinking occurrence with respect to the preliminary threshold is prepared; and in the histogram, a preliminary threshold at which a change in the number of times of preliminary blinking occurrence is minimal is set as the threshold for detecting blinking.The reason for the availability of this threshold setting technique is that the waveform of the eyelid state index value has the feature that amplitudes of waveform parts where blinking occurs are substantially uniform even when the baseline varies, and are larger than variation amplitudes of the ocular potential other than blinking that overlap with the baseline by a certain width or more. Furthermore, it is expressly noted that in the threshold setting technique in the present embodiment, a threshold does not need to be preset before the blink detection processing starts, and the threshold can be set and updated at the appropriate timing after the blink detection processing starts.According to such a configuration, since when the detection of blinking of a driver driving a vehicle, the detection of blinking of a person working at a desk, or the like is performed, the threshold can be updated to a correct value at the correct timing even if a baseline varies or a change width of an index value upon blinking occurrence varies after the blink detection processing is started, more accurate detection of blinking can be performed consecutively for a long time without having to suspend the blink detection processing due to the threshold update.

Claims

[1] Blink detection device comprising: an ocular potential measuring unit having a pair of electrodes, the ocular potential measuring unit being configured to sequentially measure an eyelid state index value indicating a state between an open state and a closed state of eyelids in an eye of a person; and a signal processing device (3), wherein the signal processing device (3) is designed to: Determining, for each measured eyelid condition index value, that a blink occurs when a change in the measured eyelid condition index value from a baseline, which is a reference value of the eyelid condition index value, in the case that the person's eyelids are in the open state, exceeds a magnitude of a threshold; Saving the sequentially measured eyelid condition index values; Counting a number of times of a preliminary occurrence of blinks with respect to each of a plurality of preliminary thresholds, wherein the number of times of the preliminary occurrence of blinks is the number of times when the change in the eyelid condition index value from the baseline exceeds an order of magnitude of a corresponding preliminary threshold, for each of the eyelid condition index values sequentially measured and stored over a predetermined period of time, the orders of magnitude of the thresholds being different from each other; Creating a histogram of the number of times of preliminary blink occurrence with respect to the preliminary thresholds; and Setting the threshold to a preliminary threshold from the preliminary thresholds at which a change curve of the histogram is minimal, the change curve representing a change in the number of times of preliminary occurrence of blinks measured between successively applied preliminary thresholds in the histogram. [2] Blink detection device according to claim 1, wherein: the specified period is a length of a specified time; and the signal processing device is designed to reset the threshold for each elapse of the predetermined period of time. [3] Blink detection device according to claim 1, wherein: the signal processing device (3) is designed to reset the threshold for each elapse of the predetermined period of time. [4] Blink detection device according to one of claims 1 to 3, wherein: the preliminary threshold at which a change curve of the histogram is minimal is a preliminary threshold at which a standard deviation of the number of times of preliminary occurrence of blinks with respect to a predetermined number of adjacent preliminary thresholds in the histogram is minimal among the preliminary thresholds. [5] Blink detection device according to one of claims 1 to 4, wherein: the ocular potential measuring unit is designed to measure an ocular potential of the person; and the eyelid condition index value is the ocular potential. [6] Blink detection device according to one of claims 1 to 4, wherein: the ocular potential measuring unit is configured to take an image of the person's eye and detect the degree of opening of the eyelids in the image of the eye; and the eyelid condition index value is an index value indicating the degree of opening of the eyelids in the image of the eye. [7] Blink detection device according to claim 3, wherein: the predetermined period of time is a period of time until the number of times of blink occurrence determined by the signal processing device (3) reaches a predetermined number of times.

Images