Detection device and detection method

Abstract

To provide a detector capable of highly precisely detecting activities of eyes by a simple detector.SOLUTION: A detector 100 for detecting activities of eyes, includes sensors 31-1 to 32-12 capable of detecting movements of a plurality of portions of the skin near the eyes, and an operation part 10 for calculating the activities of eyes using signals from the sensors.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a detection device and a detection method. [Background technology]

[0002] Eye activity can be an indicator of mental and physical activity. Mental and physical activity includes the activity of a person's mind and body. Therefore, the results of sensing eye activity can be useful information in many fields. In recent years, there has been a demand for eye activity sensing technology to be available to the general public at all times in many situations in their daily lives.

[0003] Eye activity includes eyelid movement and eyeball movement, and the number and interval of blinks, the degree of eye opening, eyeball movement and direction, etc. As a technology for sensing eye activity, for example, International Publication WO2016 / 080042 (hereinafter, Patent Document 1) discloses a glasses-type detection device having a frame with multiple infrared sensors attached, which detects eyeball movement by measuring the eyeballs of a user wearing the glasses with infrared rays. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. WO2016 / 080042 Summary of the Invention

[0005] Infrared sensors such as those disclosed in Patent Document 1 directly detect eye movements to detect the number of blinks, eye vibrations, eye direction when the eyes are closed, and the like, but are known to have low recognition and detection rates.

[0006] On the other hand, although it is possible to improve detection accuracy by using a high-precision detection device, since eye activity is widely useful information, there is a demand to detect and obtain it on a daily basis, and therefore a device that is simple both mechanically and in terms of cost is desired. For this reason, a detection device and a detection method that can detect eye activity with high accuracy using a simple device are desired.

[0007] According to one embodiment, the detection device detects eye activity and includes a sensor capable of detecting movement of the skin at multiple locations near the eye, and a calculation unit that calculates eye activity using a signal from the sensor.

[0008] According to one embodiment, the detection method senses the movement of the skin at multiple locations near the eye and determines the eye activity based on the sensing results.

[0009] Further details will be described in the following embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a specific example of the configuration of a detection device according to an embodiment. [Diagram 2] FIG. 2 is a schematic block diagram showing an example of the functional configuration of a calculation device that functions as a calculation unit of the detection device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a detection method according to the embodiment. [Figure 4] FIG. 4 is a diagram showing a method for learning learning data when the eye activity to be detected is a blink. [Diagram 5] FIG. 5 is a graph showing the sensing results obtained by the sensor during blinking. [Figure 6] FIG. 6 is a diagram for explaining an outline of an evaluation experiment conducted by the inventor to verify the blink detection accuracy. [Figure 7] FIG. 7 is a graph showing the results of an evaluation experiment conducted by the inventor to verify the blink detection accuracy. [Figure 8]FIG. 8 is a diagram showing a method of learning learning data when the eye activity to be detected is the direction of the eyeball. [Figure 9] FIG. 9 is a diagram for explaining an outline of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball direction. [Figure 10] FIG. 10 is a graph showing the results of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball direction. [Figure 11] FIG. 11 is a graph showing the results of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball direction. [Figure 12] FIG. 12 is a diagram showing a method of learning learning data when the eye activity to be detected is the movement direction of the eyeball. [Figure 13] FIG. 13 is a diagram for explaining an outline of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball movement direction. [Figure 14] FIG. 14 is a graph showing the results of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball movement direction. [Figure 15] FIG. 15 is a graph showing the results of an evaluation experiment conducted by the inventor to verify the detection accuracy of the eyeball movement direction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <1. Overview of the detection device and detection method>

[0012] (1) A detection device according to an embodiment detects eye activity, and includes a sensor capable of detecting the movement of skin at multiple locations near the eye, and a calculation unit that calculates the eye activity using a signal from the sensor. The eye activity includes eyelid movement, eyeball movement, and the like. Specifically, the eye activity includes blinking, eyeball direction, eyeball movement direction, and the like. The multiple locations near the eye are, for example, around the eye, such as above and below the eye, or multiple locations above and below the eye.

[0013] The skin at multiple locations near the eye is affected by eye activity. Therefore, the inventors found that it is possible to detect eye activity by detecting the movement of the skin. The movement of the skin can be obtained, for example, by measuring the distance to the skin. In this case, the sensor required for detecting eye activity is a sensor for measuring the distance to the skin at multiple locations near the eye, such as an infrared distance sensor. This makes it possible to make the device less expensive than a device that detects eye activity by using an electric potential or performing image analysis. In addition, the fact that the device can be detected with high accuracy has been verified by an evaluation experiment conducted by the inventors, which will be described later.

[0014] (2) Preferably, the calculation of the eye activity by the calculation unit includes using one or more detection results of the skin movements from the skin movements at multiple locations detected by the sensor for use in the calculation of the eye activity, thereby making it possible to detect the eye activity with high accuracy.

[0015] (3) Preferably, the eye activity includes eye movement, and the calculation unit determines eye movement from the skin movements at multiple locations detected by the sensor using the detection results of the multiple skin movements.

[0016] (4) Preferably, determining eye movement includes inputting feature values ​​obtained from the detection results of multiple skin movements into a computation model and obtaining an output from the computation model, and the computation model is a machine learning model obtained by learning data including feature values ​​representing multiple skin movements as input values ​​and eye movements corresponding to the multiple skin movements as output values. The eye movement is, for example, the direction of the eye and the direction of movement of the eye. By using the machine learning model, the eye movement can be easily detected with high accuracy.

[0017] (5) Preferably, the sensor includes a distance sensor that detects the distance to the skin. The distance sensor is, for example, an infrared distance sensor. This makes it possible to configure the detection device at low cost.

[0018] (6) Preferably, the plurality of locations near the eye include above the eye and below the eye. This allows eye activity to be detected with high accuracy. This allows eye activity to be detected with high accuracy.

[0019] (7) Preferably, the plurality of locations near the eyes includes a plurality of locations above the eyes and a plurality of locations below the eyes, thereby enabling eye activity to be detected with high accuracy.

[0020] (8) Preferably, the detection device is a glasses-type device that can be worn on the subject's face so that the detection range of the sensor is near the eyes. By using glasses, the subject can easily wear the device and the burden of wearing it for a long time can be reduced. This makes it possible to detect eye activity over a long period of time.

[0021] (9) A detection method according to an embodiment senses the movement of the skin at multiple points near the eyes and determines eye activity based on the sensing results. By using this detection method, the detection device can be made inexpensive, and the fact that eye activity can be detected with high accuracy has been verified through evaluation experiments performed by the inventors described below.

[0022] <2. Examples of detection devices and detection methods>

[0023] The detection device 100 according to this embodiment is a detection device that detects eye activity. Eye activity includes eyelid movement, eyeball movement, etc. Specifically, eye blinking, eyeball direction, eyeball movement direction, etc.

[0024] 1, the detection device 100 according to the present embodiment has a plurality of sensors 31-1 to 32-12. These sensors 31-1 to 32-12 configure a glasses-type sensor. Specifically, as shown in FIG. 1, the sensors 31-1 to 31-12 are attached to the rim of the right-eye lens LA of the glasses 30, and the sensors 32-1 to 32-12 are attached to the rim of the left-eye lens LB of the glasses 30. The glasses-type sensors are easy for the user to wear and can reduce the burden of wearing them for long periods of time. This makes it possible to detect eye activity over a long period of time.

[0025] The sensors 31-1 to 32-12 are sensors capable of detecting the movement of the skin at multiple locations near the eyes. The multiple locations near the eyes include at least the areas above the eyes and below the eyes. Preferably, the multiple locations near the eyes include multiple locations above the eyes and multiple locations below the eyes. As an example, as shown in FIG. 1, the sensors 31-1 to 31-12 and the sensors 32-1 to 32-12 are arranged on the rims of the right-eye lens LA and the left-eye lens LB of the glasses 30 so as to be approximately evenly arranged around the eyes. Specifically, the sensors 31-1 to 31-5 are arranged evenly from the inner corner of the right eye to the iris above the right eye, the sensors 31-7 to 31-11 are arranged evenly from the inner corner of the right eye to the inner corner below the right eye, the sensor 31-6 is arranged beside the iris, and the sensor 31-12 is arranged beside the inner corner of the eye.

[0026] The detection of the movement of the skin by the sensors 31-1 to 32-12 includes, for example, measuring the distance to the skin. As a specific example, the sensors 31-1 to 32-12 are infrared distance sensors. By using the infrared distance sensors to detect eye activity using the measured distance as described below, it is possible to configure a device at a lower cost than when using electric potential or performing image analysis.

[0027] The sensors 31-1 to 32-12 are disposed near the eyes to measure the distance to the skin at multiple points near the eyes. As a specific example, as shown in the AA cross-sectional view of Fig. 1, the sensor 31-3 disposed above the right eye measures the distance to the upper eyelid of the right eye, and the sensor 31-9 disposed below the right eye measures the distance to the lower eyelid of the right eye.

[0028] The detection device 100 further includes a calculation device 10. The calculation device 10 is connected to the sensors 31-1 to 32-12 wirelessly or via a wire, and receives sensor signals indicating sensing results from the sensors 31-1 to 32-12. The sensor signals indicate the distance to the skin at each position. The calculation device 10 functions as a calculation unit that calculates eye activity using the signals from the sensors 31-1 to 32-12.

[0029] 1, or may be a server connected to the sensors 31-1 to 32-12 via a communication network such as the Internet. The computing device 10 may be a terminal device such as a smartphone, or may be a small computing device that can be mounted on the glasses 20.

[0030] 2, the arithmetic device 10 is configured as a computer having a processor 11 and a memory 12. The processor 11 is, for example, a CPU. The memory 12 includes a flash memory, an EEPROM, a ROM, a RAM, etc. Alternatively, the memory 12 may be a primary storage device or a secondary storage device. The memory 12 stores a program 122 to be executed by the processor 11.

[0031] The arithmetic device 10 further includes a communication device 13 for receiving sensor signals from the sensors 31-1 to 32-12. The communication device 13 receives the sensor signals from the sensors 31-1 to 32-12 and passes them to the processor 11. That is, the communication device 13 functions as an input unit for the sensor signals from the sensors 31-1 to 32-12.

[0032] Preferably, the memory 12 stores user data 121. The user data 121 includes identification information of a learning model 115 used in a determination process 114 described later for each user (hereinafter, subject) who wears the glasses 30 to detect eye activity. In this case, the processor 11 can use the learning model 115 corresponding to the subject when executing the determination process 114.

[0033] The processor 11 executes a program 122 stored in the memory 12 to execute a detection process for detecting eye activity of the subject. The detection process includes a feature extraction process 111. The feature extraction process 111 is a process for extracting features from sensor signals received by the communication device 13 from the sensors 31-1 to 32-12. The processor 11 extracts features according to the type of eye activity to be detected. At this time, the processor 11 uses a plurality of sensor signals from the sensors 31-1 to 32-12, and as an example, uses sensor signals from all the sensors. Details will be described later.

[0034] The detection process further includes a determination process 114. The determination process 114 is a process for determining the presence or absence and content of eye activity of the detection target. In detail, when the eye activity of the detection target is blinking, the determination of the eye activity includes determining the blink, and the determination of the blink includes determining whether a blink has occurred and determining the number of blinks based on whether a blink has occurred. When the eye activity of the detection target is eye movement, the determination of the eye activity includes determining the eye movement, and the determination of the eye movement includes determining the orientation of the eye and the direction in which the eye moves.

[0035] Preferably, the determination process 114 uses a learning model 115. Specifically, the determination process 114 includes inputting the features extracted in the feature extraction process 111 to the learning model 115 and obtaining an output from the learning model 115.

[0036] The learning model 115 is a machine learning model that has been machine-learned using teacher data of input data T1 and output data T2, as shown in FIG. 4 and the like. The input data T1 is a feature obtained from the sensor signals of the sensors 31-1 to 32-12, and is a feature corresponding to the eye activity of the detection target, as described later. Details will be described later for each eye activity of the detection target. Preferably, the learning model 115 has been machine-learned using teacher data corresponding to the subject. This is expected to improve the detection accuracy.

[0037] Preferably, the detection process includes an output control process 117. The output control process 117 is a process for controlling the output device 14 included in the arithmetic device 10 so as to output the determination result of the determination process 114 from the output device 14. The output device 14 is a display, a speaker, a transmission device to other devices, etc. This allows the detection result of the detection device 100 to be grasped from outside.

[0038] In the detection device 100, the eye activity of the subject is detected by the detection method shown in the flowchart of Fig. 3. In detail, referring to Fig. 3, the distances to the skin of a plurality of points near the eyes of the subject wearing the glasses 30 are measured by the sensors 31-1 to 32-12 (step S1). The sensor signals from the sensors 31-1 to 32-12 indicating the measured distances are input to the calculation device 10 (step S2).

[0039] Here, the detection device 100 is capable of detecting multiple types of eye activity such as blinking, eye direction, and eye movement direction, and has verified that each of these can be detected with high accuracy in evaluation experiments performed by the inventors described below. Therefore, preferably, the calculation device 10 accepts a designation of the eye activity to be detected, and varies the calculation in the subsequent processing depending on the designation.

[0040] In the arithmetic device 10 that has received the input of the sensor signal, the processor 11 executes the program 122 to perform a process of extracting features from the sensor signal (step S3). Preferably, in step S3, the processor 11 extracts features using a plurality of sensor signals from the sensors 31-1 to 32-12. As described above, when a designation of the type of eye activity to be detected is received, in step S3 the processor 11 extracts features according to the type of eye activity to be detected.

[0041] The processor 11 uses the feature extracted in step S3 to determine the eye activity of the subject (step S4). Specifically, in step S4, the feature extracted in step S3 is input to the learning model 115 (step S41), and an output value is obtained from the learning model 115 (step S42). At this time, when a designation of the type of eye activity to be detected is accepted, the process of step S4 includes selecting the learning model 115 that has been machine-learned according to the type of eye activity to be detected. Also, when a learning model 115 is prepared for each subject, the process of step S4 includes selecting the learning model 115 according to the subject.

[0042] (Blink detection)

[0043] Here, a case where blinking is detected as eye activity will be described. With reference to Fig. 4, when blinking is detected as eye activity, as an example of teacher data used for machine learning of the learning model 115, the feature amount obtained from the sensor signals of the sensors 31-1 to 32-12 sensing the surroundings of the eyes at the time of blinking is used as input data T1, and the determination result of "blinking" is used as output data T2.

[0044] Curve L1 in Fig. 5 represents the sensing result by one of sensors 31-1 to 32-12 during blinking. In Fig. 5, the vertical axis represents a normalized value of the measurement result of the distance to the skin, and the horizontal axis represents time. In other words, Fig. 5 represents the change in the distance to the skin over time at a position near the eye during blinking. Curve L2 represents the differential value of curve L1.

[0045] By analyzing the curves L1 and L2, the inventor found that in the section where the differential value shown by the curve L2 is negative, the eyelids are closed, and in the section where the differential value is positive, the eyelids are opened. From this, it was found that the point where the differential value changes from positive to negative is the turning point of the eyelids opening and closing movement. Specifically, in Fig. 5, point P1 corresponds to the start of the blink, point P2 corresponds to the end of the blink, and point P3 corresponds to the turning point, that is, the point where the eyelids open.

[0046] In the detection device 100 according to the present embodiment, the learning model 115 is machine-learned by utilizing this principle discovered by the inventor. That is, the curve L1 itself of the sensing result or the value of the curve L2 which is its differential value is used as the input data T1 as the feature amount, and the point where the differential value changes from positive to negative is the blink start point P1, the point where it changes from negative to positive is the blink turning point P3, and the point where the positive change becomes smaller than a predetermined value is the blink end point P2, which are machine-learned as one blink.

[0047] In machine learning, a threshold may be set for the value of curve L1 in the range where the value of curve L1 is in region R, and points detected as turning points of blinks that are equal to or greater than the threshold may be removed as noise. This is because the range where the value of curve L1 is in region R is when the eyelids are open, and therefore there are no turning points of blinks. This can improve the accuracy of blink detection.

[0048] Preferably, among the sensors 31-1 to 32-12, a sensor in which the difference in sensed distance is equal to or greater than a threshold is used for machine learning. This makes it possible to use the sensing results of a position near the eye where the distance to the skin is likely to be affected by blinking, thereby improving the blink detection accuracy. The sensor used may be one or more positions experimentally determined for blinking, or may be experimentally determined for each subject. In the latter case, the user data 121 includes identification information of the learning model 115 corresponding to the subject.

[0049] When detecting a blink, processor 11 extracts a feature amount, which is a differential value, from the sensor signal of a specific sensor among sensors 31-1 to 32-12, and inputs it to learning model 115 trained as shown in FIG. 4. Processor 11 detects a blink by obtaining an output value of "blink occurred" from learning model 115. Processor 11 also detects the number of blinks by inputting a series of feature amounts obtained from a series of sensor signals and counting the output values ​​of "blink occurred" obtained from learning model 115.

[0050] The inventors conducted an evaluation experiment to verify the blink detection accuracy of the detection device 100. Specifically, as shown in Fig. 6, a subject H sat facing the display 200 and looked straight ahead, gazing at each of 15 points displayed on the display 200 without changing the direction of his face, blinked 20 times at a pace of once per second, and the blinking period was sensed.

[0051] At this time, the sensor having the highest blink detection accuracy among the sensors 31-1 to 32-12 was selected as the detection sensor based on the sensing results when the subject H blinked. The learning model 115 was trained using the sensing results of the subject H by the detection sensor.

[0052] FIG. 7 shows the accuracy rate for each of the 15 points as a result of the evaluation experiment. The accuracy rate is the ratio of the number of blinks detected to the number of blinks (20), with 100% indicating detection of all 20 blinks. As shown in FIG. 7, the detection device 100 had an accuracy rate of 100% or close to 100% at almost all points. Therefore, it was verified that blinks can be detected with high accuracy by using the detection device 100.

[0053] (Detection of eye movement (eye direction))

[0054] Next, a case where eye movement, particularly the direction of the eye, is detected as eye activity will be described. The direction of the eye corresponds to the direction of the gaze. The inventors discovered that when the eye is facing a specific direction, the skin near the eye changes in a way that is specific to that direction, and therefore the direction of the eye is expressed by the sensing result of the distance to the skin.

[0055] Therefore, when detecting the direction of the eye as eye activity, referring to FIG. 8, the teacher data used for the machine learning of the learning model 115 uses the feature amount obtained from the sensor signals of the sensors 31-1 to 32-12 that sense the area around the eye with the eye pointed in a specific direction as input data T1, and the specific direction as output data T2.

[0056] As an example, referring to Fig. 8, the eye direction is divided into nine parts by directing the eye to each of a plurality of marks 1-1 to 3-3 arranged on the display 200, and the feature amount in each direction is set as input data T1. The feature amount used in this case represents the state of the skin in a plurality of places near the eye, and as an example, is the sensor value obtained at the same time from each of the sensors 31-1 to 32-12 as shown in the line graph of Fig. 8. Preferably, the sensor signal is smoothed to reduce noise before use. The graph of Fig. 8 shows the sensor value from each sensor for each direction.

[0057] When detecting the direction of the eyeball, the processor 11 uses the sensor values ​​obtained simultaneously from the sensors 31-1 to 32-12 as features and inputs them to the learning model 115 that has been machine-learned as shown in Fig. 8. The processor 11 obtains an output value of the direction from the learning model 115 to detect the direction of the eyeball.

[0058] The inventors conducted an evaluation experiment to verify the detection accuracy of the eyeball direction by the detection device 100. Specifically, as shown in Fig. 9, a subject H sat facing the display 200 and faced forward, and gazed at each of 15 points displayed on the display 200 for 30 seconds without changing the direction of his face, and the period was sensed. The angle change between each point was approximately 11 degrees. The sensing result of the subject H by the detection sensor was used for learning the learning model 115.

[0059] Fig. 10 shows the recognition accuracy for each of the 15 points as a result of the evaluation experiment. The recognition accuracy is expressed as the F-score when using each of the classifiers, K Nearest Neighbor (Knn), support vector machine (SVM), decision tree, and random forest. The right side of Fig. 10 shows the average of the 15 points. Fig. 11 shows the average F-score of the 15 points for each classifier, and is an enlarged view of the results at the right side of Fig. 10.

[0060] From the results of Fig. 10, the F-measure was about 0.9 for the highly accurate classifier for each direction. It was also found from the results of Fig. 11 that there was variation in the evaluation by the classifier. Therefore, this evaluation experiment verified that the eye direction can be detected with high accuracy by using the detection device 100.

[0061] (Detection of eye movement (direction of eye movement))

[0062] Next, a case where eye movement, particularly the direction of eye movement, is detected as eye activity will be described. From the viewpoint that the direction of eye movement is represented by a change in the orientation of the eye, the inventor has found that the direction of eye movement can be obtained from the amount of change in the sensor value over a short time interval based on the principle discovered by the inventor that the orientation of the eye is represented by the sensing result of the distance to the skin.

[0063] Therefore, when detecting the movement direction of the eyeball as the eye activity, referring to FIG. 12, the teacher data used for the machine learning of the learning model 115 is, for example, a feature amount when the eyeball is moved in a specific direction such as up, down, left, right, etc., as input data T1. The feature amount used in this case represents a change in the state of the skin at a plurality of places near the eye, and is, for example, a difference between the first sensor value and the last sensor value for a certain period for each of the sensors 31-1 to 32-12 as shown in the line graph of FIG. 12. Preferably, the sensor signal is smoothed to reduce noise before use. The certain period is, for example, about 0.3 seconds. The graph of FIG. 12 shows the difference between the first sensor value and the last sensor value for a certain period from each sensor for each of the up, down, left, and right directions.

[0064] When detecting the movement direction of the eyeball, the processor 11 extracts the above-mentioned feature amounts from the sensor values ​​obtained from the sensors 31-1 to 32-12 for a certain period of time, and inputs them to the learning model 115 that has been machine-learned as shown in Fig. 12. The processor 11 obtains an output value of the movement direction from the learning model 115, thereby detecting the movement direction of the eyeball.

[0065] The inventors conducted an evaluation experiment to verify the detection accuracy of the eye direction by the detection device 100. Specifically, as shown in FIG. 13, a subject H, who sat facing the display 200 and faced forward, moved his / her gaze between two of four points displayed on the display 200 without changing the direction of his / her face, and the period of movement was sensed. Each movement took 2 to 3 seconds, and was performed a total of 20 seconds in each direction. The sensing results of the subject H by the detection sensor were used for learning the learning model 115.

[0066] Fig. 14 shows the recognition accuracy for the downward, leftward, rightward, and upward directions as a result of the evaluation experiment. The recognition accuracy is shown as the F-score when using each of the classifiers: K Nearest Neighbor (Knn), support vector machine (SVM), decision tree, and random forest. Fig. 15 shows the average F-score for each classifier in the four directions.

[0067] From the results of Fig. 14, the F-value of the highly accurate classifier for each direction was about 0.9. Also, from the results of Fig. 15, it was found that there was not much variation in the evaluation by the classifier when looking at the average of the four directions. Therefore, this evaluation experiment verified that the eye movement direction can be detected with high accuracy by using this detection device 100.

[0068] <3. Notes> The present invention is not limited to the above-described embodiment, and various modifications are possible. [Explanation of symbols]

[0069] 10: Arithmetic device 11: Processor 12: Memory 13: Communication equipment 14: Output device 30: Glasses 31-1: Sensor 31-2: Sensor 31-3: Sensor 31-4: Sensor 31-5: Sensor 31-6: Sensor 31-7: Sensor 31-8: Sensor 31-9: Sensor 31-10: Sensor 31-11: Sensor 31-12: Sensor 32-1: Sensor 32-2: Sensor 32-3: Sensor 32-4: Sensor 32-5: Sensor 32-6: Sensor 32-7: Sensor 32-8: Sensor 32-9: Sensor 32-10: Sensor 32-11: Sensor 32-12: Sensor 100: Detection device 111: Feature extraction processing 114: Judgment process 115: Learning model 117: Output control processing 121: User data 122: Program 200: Display H: Subject LA: Lens for the right eye LB: Lens for the left eye P1: Starting point P2: End point P3: Turning point R: area T1: Input data T2: Output data

Claims

1. 1. A detection device for detecting eye activity, comprising: a plurality of infrared distance sensors for detecting distances to a plurality of skin locations near the eye; a calculation unit that calculates the eye activity using signals from the plurality of infrared distance sensors; The eye activity calculated by the calculation unit includes a gaze movement direction, The calculation unit is receiving a plurality of sensor signals indicative of the sensing results for each of the distances to the plurality of skin locations proximate the eye; extracting features representing changes in skin condition at the plurality of locations from the plurality of sensor signals; the feature amount includes a difference between an initial sensor signal and a final sensor signal during a certain period from the plurality of infrared distance sensors; The feature amount representing the change in the condition of the skin at the plurality of locations is input to a computation model, and a determination result of the movement direction of the gaze is obtained as an output of the computation model, The computational model is a machine learning model that has, as learning data, feature values ​​representing changes in skin condition at the multiple locations as input values ​​and gaze movement directions corresponding to the feature values ​​as output values; the determination result includes a determination result of an upward direction, a downward direction, a leftward direction, and a rightward direction as a determination result of a movement direction of the line of sight, The plurality of locations near the eye include a plurality of locations above the eye and below the eye in the left and right directions, and locations on the left and right sides of the eye, The feature amount represents a state change of a plurality of locations above and below the eye in the left and right direction, and each location on the left and right sides of the eye. Detection device.

2. The eye activity calculated by the computing unit further includes blinking and gaze direction. The detection device according to claim 1 .

3. The feature values ​​input to the computational model are feature values ​​corresponding to the type of eye activity to be determined.

3. The detection device according to claim 2.

4. The infrared distance sensor is a glasses type sensor that can be worn on the subject's face so that the detection range of the infrared distance sensor is in the vicinity of the eyes. A detection device according to any one of claims 1 to 3.

5. sensing distances to multiple points of skin near the eye using multiple infrared distance sensors; a calculation unit receiving a plurality of sensor signals indicating the results of the sensing by the plurality of infrared distance sensors; the calculation unit extracts, from the plurality of sensor signals, feature amounts that represent changes in skin condition at the plurality of locations; the calculation unit inputs the feature amounts representing the changes in the condition of the skin at the plurality of locations into a calculation model, and obtains a determination result of the gaze movement direction as an output of the calculation model; Prepare for this. the feature amount includes a difference between an initial sensor signal and a final sensor signal during a certain period from the plurality of infrared distance sensors; The computational model is a machine learning model that has, as learning data, feature values ​​representing changes in skin condition at the multiple locations as input values ​​and gaze movement directions corresponding to the feature values ​​as output values; the determination result includes a determination result of an upward direction, a downward direction, a leftward direction, and a rightward direction as a determination result of a movement direction of the line of sight, The plurality of locations near the eye include a plurality of locations above the eye and below the eye in the left and right directions, and locations on the left and right sides of the eye, The feature amount represents a state change of a plurality of locations above and below the eye in the left and right direction, and each location on the left and right sides of the eye. Detection methods.

Images