Classification device

Abstract

The invention relates to a sorting device. And classifying the state of the person according to the information of the eyes. A state of a person is classified on the basis of eye information by using an imaging device, a feature extraction unit, and a classifier. The imaging device has a function of generating an image group by performing continuous imaging, the image group preferably including an image of an eye region. The eyes have black-eye areas and white-eye areas. The feature extraction unit has a step for extracting an eye region from the image group, a step for subsequently extracting a transient eye amplitude, a step for subsequently detecting a start transient image, and a step for subsequently storing the end transient image as first data. And a step of storing the image after any time after the first data as second data. The feature extraction unit has a step of extracting a white eye region from the first data and the second data. The classifier may use the white eye region as learning data.

Description

This invention application is a divisional application of the invention patent application with international application number PCT / IB2020 / 058111, international application date September 1, 2020, application number 202080059927.6 entering the Chinese national phase, and name “Classification Method”. Technical Field

[0001] One embodiment of the present invention is a classification device and a classification method for classifying a person's condition based on eye information.

[0002] In addition, one embodiment of the present invention relates to a method for generating a classifier capable of classifying a person's state based on eye information using a computing device. In addition, one embodiment of the present invention relates to a method for generating learning data for learning eye information. In addition, one embodiment of the present invention relates to a method for extracting an eye region from a group of images continuously captured using a camera device and extracting a feature value from eye information that can be derived from the eye region. In addition, one embodiment of the present invention relates to a method for causing a classifier to learn by providing the feature value as learning data to the classifier. Background Art

[0003] In recent years, there has been a demand in society to improve the quality of life. For example, people are less likely to feel their own overworked state or symptoms of mental illness. If people can grasp their own state early, they can consider appropriate methods such as taking measures (rest, etc.) before the state worsens. For example, in the field of self-counseling, research on methods for self-examination of changes that people themselves have not noticed has been carried out.

[0004] For example, it is known that emotions, mental states, or physical fatigue appear in physical reactions such as eye movements, facial expressions, sounds, or heartbeats. It is believed that by measuring these physical reactions, people can discover changes that they would not notice. Therefore, various studies on the eyes have been conducted as an element that expresses emotions, mental states, or physical fatigue. This is because the eyes directly receive instructions from the brain, which is the part that controls a person's mood. Therefore, the information obtained by analyzing the changes that appear in the eyes (hereinafter, eye information) is considered to be an important element for brain research.

[0005] As information about the eyes, images obtained by a camera can be used, so information about the eyes can be obtained without interfering with a person's movements or work. As an example of a method for self-consultation, research has been conducted on detecting sleepiness, one type of physical fatigue, from pupil changes. For example, Patent Document 1 discloses a method for determining and detecting a person's state by determining the pupil state based on an image of the eyes taken using infrared rays.

[0006] Patent Document 2 discloses a system for detecting micro-saccades, which are one of eye movements, to detect neurological diseases. [Prior technical literature] [Patent Document]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 7-249197 [Patent Document 2] Japanese PCT International Application Translation No. 2016-523112 Summary of the invention Technical problem to be solved by the invention

[0008] One of the methods for obtaining eye information is the EOG (electrooculogram) detection method, which detects the electrical signal generated by eye movement. However, the EOG detection method has the following problems. Although its detection is very accurate, it requires multiple electrodes to be set around the eyes, so it is difficult to perform on a daily basis. For example, when using eye information for self-consultation, it is preferred to obtain eye information without adding burden to daily life. Therefore, the acquisition of eye information is required to be minimally invasive and low-contact.

[0009] For example, by using a camera, information about the eye can be obtained as an image. And, due to the recent progress in machine learning, the eye region can be identified and extracted from the obtained image. Note that the eye has a black eye region and a white eye region. And, the black eye region has an iris and a pupil constituting the black eye. It is known that the color of the iris is affected by genetics.

[0010] It is said that the pupil is related to a person's emotions or physical fatigue, and the pupil is also of interest in the information about the eye. However, when detecting the pupil state, there is a problem of being affected by the color of the iris. For example, in the case of an iris whose color brightness is close to that of the pupil, there is a problem of difficulty in distinguishing the pupil from the iris. However, a camera device that can use infrared rays to take pictures can obtain an image that makes it easy to distinguish the iris from the pupil. However, when using stronger infrared rays, there is a problem of affecting the cornea, iris, lens, retina, etc. of the eye. In addition, the pupil also reacts to the brightness of the environment, etc. Therefore, although the pupil is related to a person's emotions or physical fatigue, there is a problem of containing a lot of environment-dependent noise.

[0011] In addition, when obtaining information about the eye, the eye blinks (hereinafter referred to as blinking) irregularly. For example, blinking includes an unconscious blink reflex that is used to protect the eyeball from drying. In addition to the blink reflex, there are various types of blinking. Therefore, there is a problem that blinking is included in the information about the eye as a noise component. In addition, when using the information about the eye for analysis, there is the following problem: the image of the eye as the object is affected by the interval between irregular blinks or the brightness of the surrounding environment.

[0012] In view of the above problems, one of the purposes of one embodiment of the present invention is to provide a method for classifying a person's state based on eye information using a computing device. One of the purposes of one embodiment of the present invention is to provide a method for generating learning data for learning eye information. One of the purposes of one embodiment of the present invention is to provide a method in which an eye region is extracted from a group of images continuously captured using a camera device, and a feature quantity is extracted from the eye information. One of the purposes of one embodiment of the present invention is to provide a method for providing the feature quantity as learning data to a classifier to enable the classifier to learn.

[0013] Note that the recording of these purposes does not prevent the existence of other purposes. Note that one mode of the present invention does not need to achieve all of the above purposes. Purposes other than the above purposes are obvious from the description of the specification, drawings, claims, etc., and can be extracted from the description. Solutions to technical problems

[0014] One embodiment of the present invention is a classification method using an imaging device, a feature extraction unit, and a classifier. The classifier includes a classification model. The imaging device has a function of generating an image group by performing continuous shooting. The image group includes an image of an eye region. The eye has a black eye region and a white eye region. The black eye region has an area composed of an iris and a pupil, and the white eye region is an area of ​​the eyeball covered by a white membrane. The classification method has the following steps: a step in which the feature extraction unit extracts the eye region from the image group; a step in which the blink amplitude is extracted from the image group; a step in which an image determined to be a start blink is detected from the image group; a step in which an image determined to be a finish blink in the image group is stored as first data; and a step in which an image after an arbitrary time has passed after the first data in the image group is stored as second data. The classification method has a step in which the feature extraction unit extracts area information of the white eye region from the first data and the second data. The classification method has a step in which the feature extraction unit provides the area information of the white eye region to the classifier as learning data. The classification method has a step in which the classifier generates a classification model using the learning data.

[0015] In the above structure, the images of the eye regions included in the first data and the second data all have a black eye region and a white eye region. When the area information of the detected white eye region has an independent first region and a second region, the classification method has a step in which the feature extraction unit outputs a ratio of the first region to the second region. When the area information of the detected white eye region is detected as a third region, the classification method has the following steps: a step in which the feature extraction unit detects a circular region from the third region; a step in which the center of the circle is obtained from the circular region; a step in which the third region is divided into a first region and a second region using the x-coordinate of the center of the circle; and a step in which the ratio of the first region to the second region is output. The classification method has a step in which the feature extraction unit calculates the vibration width of the white eye based on the ratio of the first region to the second region. The classifier preferably uses the vibration width of the white eye as learning data.

[0016] In the above configurations, the classification method includes a step in which the feature extraction unit provides the classifier with the sclera vibration width and the blink amplitude as learning data. The classification method preferably includes a step in which the classifier generates a classification model using the sclera vibration width and the blink amplitude.

[0017] In each of the above structures, the classification method includes a step in which the classifier uses the learning data for learning. The classification method includes a step in which the feature extraction unit is provided with new first data and second data. The classification method preferably includes a step in which the classifier uses the classification model to classify the state of a person's emotions, changes in health status, etc.

[0018] In each of the above configurations, the classification method includes a step of adding a supervisory label to the learning data. The classification method preferably includes a step of performing learning by the classifier using the learning data to which the supervisory label is added. Effects of the Invention

[0019] One embodiment of the present invention may provide a method for classifying a person's state based on eye information using a computing device. One embodiment of the present invention may provide a method for generating learning data for learning eye information. One embodiment of the present invention may provide a method in which an eye region is extracted from a group of images continuously captured using a camera device, and a feature quantity is extracted from the eye information. One embodiment of the present invention may provide a method for generating a classifier by providing the feature quantity as learning data to a classifier.

[0020] Note that the effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not prevent the existence of other effects. Other effects are the effects that are not described in this section and are recorded below. Those skilled in the art can derive and appropriately extract the effects that are not described in this section from the description of the specification or drawings, etc. Note that one embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention sometimes does not have the effects listed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram for explaining a method of classifying a person's state based on eye information. Figure 2A This diagram explains the structure of the eye. Figure 2B This is a diagram for explaining a method of generating learning data. Figure 3 This is a flowchart illustrating a method for generating learning data. Figure 4 This is a flowchart illustrating a method for generating learning data. FIG. 5A to FIG. 5C This is a diagram explaining information about the eyes. Fig. 6A This is a diagram explaining information about the eyes. Figure 6B and Figure 6C FIG. 1 is a diagram for explaining the extracted white eye region. Figure 7 is a flow chart illustrating a method of detecting the white eye area. FIG. 8A to FIG. 8C This is a diagram for explaining a method of generating learning data. Fig. 9 is a block diagram illustrating a classification device. Modes for Carrying Out the Invention

[0022] The embodiments are described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following description, and a person skilled in the art can easily understand that the method and details can be transformed into various forms without departing from the purpose and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the contents described in the embodiments shown below.

[0023] Note that in the invention structure described below, the same reference numerals are used in common in different drawings to represent the same parts or parts with the same function, and repeated description is omitted. In addition, when representing parts with the same function, the same hatching is sometimes used without adding a reference numeral.

[0024] In addition, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent its actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings. Implementation In this embodiment, using Figures 1 to 9 A method for classifying a person's state based on information about the eyes will be described. Note that in this embodiment, the description is focused on either the left or right eye. However, the structure and method described in this embodiment can be applied to both the left and right eyes.

[0026] The method for classifying a person's state based on the information of the eyes described in this embodiment is controlled by a program running on a computing device. Therefore, the computing device can be replaced by a classification device having a method for classifying a person's state based on the information of the eyes. Fig. 9 The classification device for classifying the state of a person based on the information of the eyes is described in detail in the text. The program is stored in the memory or storage space (storage) of the computing device. Alternatively, the program is stored in a computer connected via a network (LAN (Local Area Network, Local Area Network), WAN (Wide Area Network, Wide Area Network), Internet, etc.) or a server computer including a database.

[0027] A classification device having a method for classifying a person's state based on information from the eyes includes a camera, a feature extraction unit, and a classifier. The camera can generate an image group and store it in a memory or storage space of a computing device. An image group refers to continuously captured images or dynamic images, etc., that is, a plurality of images. Therefore, the classification device can use an image group stored in a memory or storage space of a computing device. For example, when the classification device is assembled into a portable terminal such as a mobile device, the classification device preferably includes a camera. Note that the image group can also be provided to the classification device by a camera connected to a network such as a network camera (including a surveillance camera).

[0028] The image group preferably includes a human face as an object. Machine learning can be used to extract an image of the eye region from an image recording a human face. The machine learning process preferably uses artificial intelligence (AI: Artificial Intelligence). For example, in order to extract an image of the eye region, an artificial neural network (ANN: Artificial Neural Network) can be used in particular. Note that in one embodiment of the present invention, an artificial neural network is sometimes referred to as a neural network (NN: Neural Network). The computational processing of the neural network is performed using a circuit (hardware) or a program (software).

[0029] For example, when retrieving an image of the eye region from an image recording a human face, an image of a typical eye region can be provided as a query image. An area having a high similarity to the query image is extracted from the image recording the human face included in the image group. As an example, when retrieving an area having a high similarity, a convolutional neural network (CNN), a pattern matching method, etc. can be used as an image retrieval method. In one embodiment of the present invention, as long as the query image is recorded, it is not necessary to provide the query image every time a search is performed. The extracted eye area is stored as an image.

[0030] Detailed information about the eye is extracted from the image of the extracted eye area. The eye has a black eye area and a white eye area. The black eye area has an area consisting of the iris and the pupil, and the white eye area refers to the area of ​​the eyeball covered by the white membrane (sometimes called the sclera). In addition, the maximum width of the area sandwiched between the upper eyelid and the lower eyelid is the blink amplitude. In addition, the minimum value of the area sandwiched between the eyelids is the minimum value of the blink amplitude. Various eye states are recorded in the image group. The blink amplitude can be obtained by selecting multiple blink amplitudes that start from the front when the blink amplitude is large and calculating the average value.

[0031] Here, the function of the feature extraction unit is described. The feature extraction unit has the function of extracting detailed information about the eye from the image of the extracted eye region. The feature extraction unit performs the following processing on each image of the recorded face included in the image group. The following processing is a method for creating learning data of the eye that reduces the influence of noise caused by blinking.

[0032] The above method includes a step in which the feature extraction unit extracts an image of an eye region from an image recording a human face.

[0033] In addition, the method includes a step in which the feature extraction unit extracts the blink amplitude from the image of the eye region. The feature extraction unit can set a determination threshold for detecting a blink in a subsequent step by extracting the blink amplitude. As an example, half the width of the blink amplitude can be set as the determination threshold. However, it is preferred to set different determination thresholds for each person as the object. In addition, by converting the image group into an image group of the eye region, the size of the image processed by the computer can be reduced. Therefore, the memory usage can be reduced and the power consumption can be reduced.

[0034] In addition, the method includes a step in which the feature extraction unit detects an image determined to be a start of blinking from an image of an eye region. The start of blinking refers to a case where an image determined to be a closed eyelid is detected. In addition, the end of blinking refers to a case where an image determined to be an open eyelid is detected. In order to determine whether the eyelid is open or closed, the determination threshold can be used.

[0035] Blinking is determined based on the blink amplitude extracted from the image of the start of blinking, the blink amplitude extracted from the image of the end of blinking, and an image with an amplitude smaller than the blink amplitude extracted from the start and end of blinking. Although it varies from person to person, the blink period is considered to be about 300ms. The camera device preferably obtains at least three images of the eye region with a blink amplitude smaller than the determination threshold during the blink period.

[0036] In addition, the method has a step in which the feature extraction unit stores an image of the eye region after an arbitrary time after the image determined to have ended blinking as first data. The first data represents the state of the eye after blinking. Therefore, when blinking occurs due to blink reflex, the first data roughly and appropriately represents the state of the person. In addition, the arbitrary time is preferably freely set.

[0037] In addition, the method includes a step in which the feature extraction unit stores an image obtained after an arbitrary time has passed since the first data as second data. The second data generally represents a state such as a change in a person's emotions or health status. In addition, the arbitrary time is preferably freely set.

[0038] For example, when a person is emotionally excited, the eyes recorded in the second data are larger or smaller than the eyes recorded in the first data. In addition, when a person is sleepy, the amplitude of the eyes in the second data is sometimes smaller than the amplitude of the eyes recorded in the first data. Note that when judging sleepiness, the blink frequency can be added as a judgment. The blink frequency indicates how many times a person blinks per minute.

[0039] In addition, the method includes a step in which the feature extraction unit extracts area information of the white eye region from the first data and the second data. In one embodiment of the present invention, the state of eye movement can be grasped by using information that can be obtained from the white eye region. For example, the area information of the white eye region refers to the area ratio when the white eye region is divided into the first region and the second region using the center coordinates of the black eye, or the change in the area of ​​the first region and the second region due to the opening and closing of the eyelids.

[0040] For example, it is known that the black eye often rotates even when the line of sight stops at the same place. This is called fixation micromotion, and the largest rotation in the fixation micromotion is called micro-saccade. Micro-saccade plays a role in preventing the light incident through the pupil from always irradiating the same place of the retina. For example, if there is no micro-saccade and the light incident through the pupil always irradiates the same place of the retina, the retina cannot recognize the change of light. It is known that the retina cannot recognize the image incident through the pupil. In one embodiment of the present invention, in order to detect micro-saccade, the vibration width of the black eye caused by micro-saccade can be detected according to the area ratio of the white eye. Note that sometimes the vibration width of the black eye caused by micro-saccade is replaced by the vibration width of the white eye for explanation. In addition, the vibration width of the black eye and the white eye can be replaced by the vibration width of the eyeball.

[0041] It is said that when the movement of micro-saccade is large and fast, the eyes are in a state of high ability to produce vision. Therefore, when the movement of micro-saccade is large and fast, the person is in a state of high concentration on the object. In addition, when the movement of micro-saccade is small and slow, it can be said that the person is in a state of losing concentration on the object. This state can be judged as the person feeling sleepy. Note that in order to classify the state of the person more accurately, it is preferred to use multiple conditions such as the area ratio of the white eye, the blink amplitude or the vibration width of the white eye to make a judgment.

[0042] In addition, the method includes a step in which the feature extraction unit provides information on the white of the eye region as learning data to the classifier. For example, as the information on the white of the eye region, preferably, the area ratio of the white of the eye, the blink amplitude, the vibration width of the white of the eye, or the change in the eyelid position are provided as the learning data.

[0043] In addition, the above method has a step in which the classifier generates a classification model using the learning data.

[0044] The above learning data can be used to perform classification through unsupervised machine learning. For example, algorithms such as K-means or DBSCAN (density-based spatial clustering of applications with noise) can be used as classification models.

[0045] As another example, supervised data may be provided to the classification model. For example, emotional classification or a threshold for determining the width of the white eye vibration may be provided to the supervised data. Classification may be performed using supervised machine learning using learning data to which supervised data is added. For example, machine learning algorithms such as decision trees, naive Bayes, KNN (k Nearest Neighbor), SVM (Support Vector Machines), perceptrons, logistic regression, and neural networks may be used as classification models.

[0046] Next, use Figure 1 The method of classifying a person's state based on the eye information described in this embodiment will be described. Hereinafter, the method of generating the classifier will sometimes be referred to as the classification device 10 for description.

[0047] The classification device 10 includes a camera 20, a feature extraction unit 30, and a feature inference unit 40. The feature extraction unit 30 includes a blink detection unit 31, a memory management unit 32, a storage unit 33, a detection unit 34, and an information detection unit 35. The feature inference unit 40 includes a classifier 40a. The storage unit 33 includes a storage device 33a and a storage device 33b. Note that the number of storage devices in the storage unit 33 is not limited to two. The storage device may be one or more than three.

[0048] The camera device 20 can obtain images 21a to 21n of a person using the camera element in the camera device 20. The image includes the eye region of the person. The eye region includes the white eye region 101, the black eye region 102a and the eyelid 104. The black eye region includes the pupil region and the iris region. The images 21a to 21n are provided to the blink detection unit 31. a or n is a positive integer. In addition, n is greater than a.

[0049] The blink detection unit 31 may extract the eye region from each of the images 21a to 21n. The blink detection unit 31 stores the extracted eye region as images 22a to 22n. It is preferable to convert the images 22a to 22n into images having the same number of pixels. In addition, it is preferable to convert the images so that the eye widths are consistent. Hereinafter, when describing any of the images 22a to 22n, the name "image 22 of the eye region" may be used for simplicity of description. The blink detection unit 31 preferably uses CNN to extract the eye region.

[0050] The blink detection unit 31 can extract the blink amplitude from the image of the eye region. The blink amplitude is preferably determined as the maximum value of the distance between the upper eyelid and the lower eyelid in the state of opening the eyes or the average value of the maximum value of the distance extracted from multiple images. The blink detection unit 31 can use the blink amplitude to determine a determination threshold value, and the determination threshold value is used to determine whether the image of the eye region is in a blink state.

[0051] In addition, the blink detection unit 31 can detect the image 22p that is determined to be the start of blinking from the images 22a to 22n using the determination threshold. For example, when the blink detection unit 31 detects that the blink amplitude detected from the image 22p is smaller than the determination threshold, it can be determined that the eye has started blinking. Next, the blink detection unit 31 detects the image that is determined to be the end of blinking from the images after the image 22p. For example, when the blink detection unit 31 detects that the blink amplitude detected from the images after the image 22p is larger than the determination threshold, it can be determined that the eye has ended blinking. p is a positive integer.

[0052] As an example, when the image 22p +2 When the image is judged as the image of the end of blinking, the image 22p +3 The memory management unit 32 provides the image 22p as the first data. +3 In one embodiment of the present invention, although the image 22p which is determined to be the end of the blink is +2 After the image 22p +3 The first data is stored in the storage device 33a, but the image stored as the first data is not limited to the image 22p. +2 The image after that. The image 22q after any time can also be stored. q is a positive integer. In addition, q is greater than p.

[0053] Furthermore, the blink detection unit 31 may convert the image 22p into +3 The image 22r after an arbitrary time is provided as the second data to the memory management unit 32. The memory management unit 32 stores the image 22r in the storage device 33b. r is a positive integer. In addition, r is greater than q.

[0054] Note that the number of storage devices is not limited to two. Images of the eye region at different times after a blink may be stored in multiple storage devices. Figure 3 and Figure 4 The processing contents of the blink detection unit 31 will be described in detail.

[0055] The detection unit 34 can extract the area information of the white eye region from the first data stored in the storage device 33a and the second data stored in the storage device 33b. When the CNN is used to extract the area information of the white eye region, the efficiency is high.

[0056] The information detection unit 35 can detect more detailed information of the white eye region from the area information of the white eye region extracted by the detection unit 34. For example, the white eye region can be divided into a first region and a second region by using the center coordinates of the black eye. In addition, the area ratio of the first region and the second region can be extracted. Alternatively, the area of ​​the first region and the second region can be extracted, which changes due to the opening and closing of the eyelid. The method of extracting the area information of the white eye region will be described in detail using FIG. 6. The detection unit 34 and the information detection unit 35 can extract the size of the pupil from the black eye region.

[0057] The feature estimation unit 40 is provided with information on the white eye region extracted by the feature extraction unit 30 as learning data. For example, the area ratio of the white eye, the blink amplitude, the vibration width of the white eye, or the position change of the eyelid are provided to the classifier 40a as learning data. In addition, the size of the pupil may also be provided.

[0058] The classifier 40a in the feature estimation unit 40 can generate a classification model using the learning data. The classification model can also be provided with supervisory data 41. For example, the emotion classification, the threshold for determining the width of the vibration of the white of the eye, etc. can be provided as the supervisory data 41. The classification model is learned by being provided with the above-mentioned learning data and supervisory data. Therefore, the classifier 40a including the classification model can estimate the change of the emotion or health status of a person based on the information of the eyes.

[0059] Next, the new first data and the second data are provided to the feature extraction unit 30. The new learning data is provided to the feature estimation unit 40. The feature estimation unit 40 classifies the state such as the change of the emotion or health state of the person using the classifier 40a including the learned classification model and outputs the classification result Cout.

[0060] Figure 2A 1 is a diagram for explaining the structure of the eye. The components of the eye include the white eye region, the black eye region, and the eyelid 104. The white eye region includes the white eye region 101A and the white eye region 101B which are divided with the black eye as the center. The black eye region includes the pupil 102 and the iris 103. Next, the dimensions of the eye structure are defined. For example, in the eye, the horizontal width x of the eye, the width k of the iris, the width m of the pupil, and the vertical width (blink amplitude y) of the eye can be defined.

[0061] then, Figure 2B is a diagram for explaining the method of generating learning data. Blinking occurs irregularly in the eye. As an example, Figure 2BIn the figure, time BT1 to time BT6 represent the time when blinking occurs. When generating learning data of the eye, it is important to grasp the information of the stable eye. For example, it is necessary to remove the blinking image as noise from images 22a to 22n. When an image group including blinking images is used as learning data, in the learning of the classification model, it is necessary to let the classification model learn more images. In addition, in the classification device 10, the power consumption increases as the amount of images to be learned increases, and more image processing time is required.

[0062] In one embodiment of the present invention, a blink is used to trigger an event. As an example, a blink at time BT2 is described in detail.

[0063] Time T1 is the state before the blink starts (the state where the event has not occurred).

[0064] Time T2 is the time when the blink amplitude y is detected to be smaller than the determination threshold value. Specifically, time T2 is the time when the blink detection unit 31 detects the image 22p determined to be the start of blinking.

[0065] Time T3 is the time when the blink amplitude y is detected to be smaller than the determination threshold and smaller than the blink amplitude y detected from the image 22p. Specifically, time T3 is the time when the blink detection unit 31 detects the image 22p where the blink amplitude y is determined to be the smallest. +1 time.

[0066] Time T4 is the time when the blink amplitude y is detected to be larger than the determination threshold. Specifically, time T4 is the time when the blink detection unit 31 detects the image 22p determined to be the end of the blink. +2 time.

[0067] Time T5 is the image 22p after the end of the blink is detected. +3 Time. Image 22p +3 The image 22q is stored in the storage device 33a as data Data1. The image 22q obtained after an arbitrary time has passed since the time T4 may be used as data Data1.

[0068] Time T6 is when image 22p is detected +3 The image 22r is stored as data Data2 in the storage device 33b.

[0069] The learning data obtained through the steps from time T1 to time T6 can generate the following learning data: reduce noise content; reduce power consumption; and reduce the number of required storage devices.

[0070] Data1 and data Data2 may be processed as independent learning data, respectively. Alternatively, data Data1 and data Data2 may be processed as one learning data. Alternatively, third data may be generated by extracting the difference of data from data Data2, and the third data may be used as learning data. As such, data Data1, data Data2, and the third data may be used as learning data for other machine learning. The frequency of blinking or the width of the pupil may be added to data Data1, data Data2, and the third data. The frequency of blinking or the width of the pupil may also be used as one of the learning data representing changes in a person's emotions or health status.

[0071] Figure 3 2 is a flowchart for explaining a method for generating learning data. First, the blink amplitude is extracted using the image group of images 22a to 22n. Next, the blink amplitude is used to set a determination threshold for determining the start and end of a blink.

[0072] Step ST01 is a step of extracting the blink amplitude y from any number of images in the image group. Note that in step ST01, the maximum value y_max of the blink amplitude y and the minimum value y_min of the blink amplitude y are also extracted. The minimum value y_min of the blink amplitude y represents the minimum value detected from the image. By frequently summing and averaging the maximum value y_max of the blink amplitude y and the minimum value y_min of the blink amplitude y, characteristics such as the degree of eye opening of the target person can be extracted.

[0073] Step ST02 is a step of calculating the average value of the blink amplitude y. As an example, the average value y_ave is calculated using the first three or more digits of the blink amplitude y extracted in step ST01. It is preferable that the number of averaging can be set.

[0074] Step ST03 is a step of setting a determination threshold for determining the start and end of a blink. As an example, when the start and end of a blink are used as the determination conditions, 50% of the average value y_ave is set as the determination threshold. Note that it is preferred that the determination conditions for the start and end of a blink are not limited to 50% and can be set arbitrarily.

[0075] Step ST04 is a step of detecting and storing data Data1 and data Data2. Figure 4 The detection and storage of data Data1 and data Data2 will be described in detail.

[0076] Step ST05 is a step of ending the generation of learning data.

[0077] Figure 4 is a flowchart illustrating a method for generating learning data. Figure 4In the method of detecting and storing the data Data1 and the data Data2 of the learning data, a detailed description is given. Figure 4 In the example, a pixel group of images 22a to 22n is used to store the eye region as an image.

[0078] Step ST11 is a step of selecting an arbitrary image 22p from the image group.

[0079] Step ST12 is a step of confirming whether there is a new image. +1 If it exists, proceed to step ST13. In image 22p +1 If it does not exist, the process proceeds to step ST1B (return), and then proceeds to step ST05 to end the generation of learning data.

[0080] Step ST13 is a step of extracting the eye region from the image.

[0081] Step ST14 is a step of extracting the blink amplitude y from the eye region.

[0082] Step ST15 is a step of determining whether the blink amplitude y is less than a determination threshold. When the blink amplitude y is less than the determination threshold, it is determined that the blink has started, and the process proceeds to step ST16. When the blink amplitude y is greater than the determination threshold, it is determined that the blink has not started, and the process proceeds to step ST12.

[0083] Step ST16 is a step of confirming whether there is a new image. +2 If it exists, proceed to step ST17. In image 22p +2 If it does not exist, the process proceeds to step 1B, and then to step ST05 to end the generation of learning data. Although step ST16 includes the processing of step ST13 and step ST14, due to the blank space on the drawing, the processing is indicated by the symbol "*1". Hereinafter, the steps with the symbol "*1" include the processing of step ST13 and step ST14.

[0084] Step ST17 is a step of determining whether the blink amplitude y is greater than a determination threshold. When the blink amplitude y is greater than the determination threshold, it is determined that the blink has ended, and the process proceeds to step ST18. When the blink amplitude y is less than the determination threshold, it is determined that the blink has not ended, and the process proceeds to step ST16.

[0085] Step ST18 is to convert the image 22p +3 The step of storing the data Data1 in the storage device 33a.

[0086] Step ST19 is a step of checking whether there is a new image. If the image 22r exists, the process proceeds to step ST1A. If the image 22r does not exist, the process proceeds to step ST1B, and then proceeds to step ST05 to end the generation of learning data.

[0087] Step ST1A is a step of storing the image 22r as data Data2 in the storage device 33b. Next, the process proceeds to step ST12.

[0088] As described above, blinking can be detected as an event, and images taken at any time after the blinking are collected as learning data. Since the learning data is collected under the condition of "after blinking", the noise component can be reduced. In addition, information such as the number of blinks can be easily collected by setting a counter in step ST16.

[0089] FIG. 5A to FIG. 5C This is a diagram that explains information about the eyes. Figure 5A , the state of the eye after blinking is shown as an example. Figure 5A This corresponds to data Data 1. In many cases, the eyes after blinking indicate changes in the emotions or health status of the person in the image, which the person himself or herself does not notice. Figure 5A The blink amplitude y1 is preferably greater than -20% and less than +20 of the average value y_ave. The blink amplitude y1 is preferably greater than -10% and less than +10% of the average value y_ave. More preferably, the blink amplitude y1 is greater than -5% and less than +5% of the average value y_ave.

[0090] The detection unit 34 and the information detection unit 35 can detect, as eye information, the area of ​​the white eye region 101A, the area of ​​the white eye region 101B, the total area of ​​the white eye region 101A and the white eye region 101B, and the area ratio of the white eye region 101A to the region 101B. The detection unit 34 and the information detection unit 35 preferably use CNN to extract each area.

[0091] exist Figure 5B In FIG. 1 , the state of the eye after an arbitrary time after blinking is shown as an example. When an arbitrary time has passed after blinking, the expression muscles of the eye are sometimes affected by changes in the person's emotions or health status. When the expression muscles are affected, the blink amplitude y2 or the pupil changes. Figure 5B , as an example, the change in eye shape that occurs when a person is surprised or when the person is emotionally excited is shown. For example, the upper eyelid 104 rotates upward as shown in the direction EL1. This change can be detected as a change in the total area of ​​the white eye region 101A and the region 101B.

[0092] As another example, Figure 5C Showing the state of the eye at any time after blinking Figure 5B Different eye states. In many cases, the state of the eyes at any time after a blink indicates changes in a person's emotions or health status from blink to blink. Figure 5C , as an example, the change in eye shape that occurs when a person feels sleepy is shown. When the facial muscles are affected, the blink amplitude y3 or the pupil changes. For example, the upper eyelid 104 turns downward as shown in the direction EL1. The blink amplitude y3 is preferably greater than the determination threshold.

[0093] This change can be detected as a change in the area of ​​the white eye region 101A and the region 101B. Figure 5C As shown, the amount of change in direction EL2 or direction EL3 is sometimes different. Such subtle changes are sometimes affected by human emotions and are very effective when people give themselves counseling.

[0094] Therefore, in many cases, Figure 5B and Figure 5C Equivalent to Data2. However, Figure 5B or Figure 5C Sometimes it is equivalent to the state of the eyes after blinking. Figure 5B or Figure 5C In such a state, changes in a person's emotions or health status may become apparent. These classifications are performed using a classification model learned using learning data.

[0095] As another example, in a situation such as Figure 5C In such an eye state, the person may be in a state of high concentration. As a method of differentiating this situation, there is a method of utilizing the vibration width of micro-saccades.

[0096] In one embodiment of the present invention, the width of the vibration of the white of the eye (eyeball) caused by microsaccade can be detected from the area ratio of the white of the eye. A method of detecting the width of the vibration of the white of the eye will be described in detail with reference to FIG.

[0097] exist Fig. 6A In order to explain the method of detecting the vibration width of the sclera, Figure 5B To detect the vibration width of the white of the eye, the area of ​​the white of the eye must be determined. Fig. 6A When the white eye region is detected as one object as shown, it is necessary to set a dividing line SL for dividing the white eye region.

[0098] Figure 6B This is to explain the information extracted by the detection unit 34. Figure 5B 1. Note that the white eye region 101 is detected as an object. When there is only one object, the information detection unit 35 extracts a circular region substantially equal to the black eye region from the white eye region. Next, the center coordinates C (a, b) of the extracted circle are detected. Next, the x coordinate of the center coordinates can be used to set a dividing line SL for dividing the white eye region 101 into the white eye region 101A and the white eye region 101B.

[0099] Figure 6C This is to explain the information extracted by the detection unit 34. Figure 5C The white eye region 101 is extracted as two objects representing the white eye region 101A and the white eye region 101B.

[0100] exist Figure 6B and Figure 6C , the image is divided into a white eye region 101A and a white eye region 101B with the dividing line SL as the center. For example, when detecting the vibration width of the black eye, it is preferable to detect the change in the center of the black eye. Note that the detection accuracy of the vibration width of the black eye depends on the resolution of the number of pixels of the image. However, regarding the vibration width of the white eye, the change amount is based on the area comparison, and there is an inverse relationship that the white eye region 101A becomes larger and the white eye region 101B becomes smaller. Therefore, the detection accuracy of the vibration width of the white eye is higher than that of the vibration width of the black eye.

[0101] Figure 7 1 is a flowchart illustrating a method for detecting the white eye area.

[0102] Step ST20 is a step of extracting the white eye region from the image of the eye region using CNN.

[0103] Step ST21 is a step of determining whether the white eye region is one object. When the white eye region is detected as one object, the process proceeds to step ST22. When the white eye region is detected as multiple objects, the process proceeds to step ST25.

[0104] Step ST22 is a step of detecting a circle (black eye (iris and pupil)) area from the white eye area detected as one object.

[0105] Step ST23 is a step of detecting the center coordinates C (a, b) of the circular region.

[0106] Step ST24 is a step of dividing the white eye region around the x coordinate of the central coordinate C (a, b). The white eye region 101 is divided into the white eye region 101A and the white eye region 101B.

[0107] Step ST25 is a step of calculating the area of ​​each of the detected white eye region 101A and the white eye region 101B.

[0108] FIG. 8A to FIG. 8C is a diagram illustrating the method of generating learning data. FIG. 8A to FIG. 8C, time T11, time T21, and time T31 are the times before the start of blinking. Time T12, time T22, and time T32 are the times when the feature extraction unit 30 detects the start of blinking. Time T13, time T23, and time T33 are the times when the blink detection unit 31 detects the image determined to have the smallest blink amplitude y. Time T14, time T24, and time T34 are the times when the feature extraction unit 30 detects the end of blinking. Time T15, time T25, and time T35 are the times when an arbitrary time has passed after the end of blinking. Time T16, time T26, and time T36 are the times when an arbitrary time has passed after the end of blinking. Time T17, time T27, and time T37 are the times when an arbitrary time has passed after the end of blinking, which are different from time T16, time T26, and time T36.

[0109] exist FIG. 8A to FIG. 8C In the example, the feature extraction unit 30 stores images at different times after the time T14, the time T24, or the time T34 at which the blink ends as data Data1, data Data2, and data Data3, respectively.

[0110] Fig. 8A This is an example of the sclera vibrating laterally. When the sclera vibrates in the x-axis direction, the width of the sclera vibration can be easily detected.

[0111] Figure 8B This is an example of the white eye vibrating in the tilt direction. When the white eye vibrates in the tilt direction, the change in the x-axis direction can be used instead of the vibration in the tilt direction to detect it as the vibration width of the white eye. When judging the vibration in the tilt direction, the white eye area can be further divided with the y coordinate of the center coordinate C (a, b) as the center, and the areas of the four areas from the white eye area 101A to the white eye area 101D can be compared. Since the processing of detecting the movement amount in the tilt direction can be reduced, the power consumption can be reduced.

[0112] Figure 8C This is the case where the eyelid moves up and down. Since it is difficult to detect using the area ratio of the white eye, it is preferable to detect the blink amplitude y. Figure 8C When the white of the eye fluctuates in the x-axis direction or in the tilt direction, the width of the fluctuation of the white of the eye can be easily detected. By combining the width of the fluctuation of the white of the eye, the state of the person can be classified more accurately.

[0113] Fig. 9 This is a block diagram for explaining a classification device 100 having a method for classifying a person's state based on eye information.

[0114] The classification device 100 includes a computing unit 81, a memory 82, an input / output interface 83, a communication device 84, and a storage space 85. In other words, the method of using the classification device 100 to classify the state of a person based on the information of the eyes is provided by a program including the camera 20, the feature extraction unit 30, and the feature estimation unit 40. In addition, the program is stored in the storage space 85 or the memory 82, and the computing unit 81 is used to perform parameter exploration.

[0115] The input / output interface 83 is electrically connected to a display device 86a, a keyboard 86b, a camera 86c, etc. Fig. 9 Although not shown in the figure, a mouse etc. may also be connected.

[0116] The communication device 84 is electrically connected to other networks via a network interface 87. In addition, the network interface 87 includes communication via wired or wireless. The network is electrically connected to a monitoring camera 88, a network camera 89, a database 8A, a remote computer 8B or a remote computer 8C, etc. Note that the monitoring camera 88, the network camera 89, the database 8A, the remote computer 8B or the remote computer 8C electrically connected via the network can also be set in different buildings, different regions, and different countries.

[0117] Part of the configuration and method described in one embodiment of the present invention may be appropriately combined and used. [Explanation of symbols]

[0118] : Data1: data, Data2: data, Data3: data, T1: time, T2: time, T3: time, T4: time, T5: time, T6: time, T11: time, T12: time, T13: time, T14: time, T15: time, T16: time, T17: time, T21: time, T22: time, T23: time, T24: time, T25: time, T26: time, T27: time, T31: time, T32: time, T33: time, T34: time, T35: time, T36: time, T37: time, y1: blink amplitude, y2: blink amplitude, y3: blink amplitude, 8A: database, 8B: remote computer, 8C: remote computer, 10: classification device, 20: camera device, 21a: image, 21n: image, 2 2: image, 22a: image, 22n: image, 22p: image, 22q: image, 22r: image, 30: feature extraction unit, 31: blink detection unit, 32: memory management unit, 33: storage unit, 33a: storage device, 33b: storage device, 34: detection unit, 35: information detection unit, 40: feature inference unit, 40a: classifier, 41: supervision data, 81: operation unit, 82: memory, 83: input and output interface, 84: communication device, 85: storage space, 86a: display device, 86b: keyboard, 86c: camera, 87: network interface, 88: surveillance camera, 89: network camera, 100: classification device, 101: white eye area, 101A: white eye area, 101B: white eye area, 102: pupil, 102a: black eye area, 103: iris, 104: eyelid.

Claims

1. A classification device, comprising: A camera device, a feature extraction unit and a feature estimation unit, The camera device is configured to capture a group of images. The image group includes images of eye regions, The eye region has a white eye region, The white eye area is the area of ​​the eyeball covered by the white membrane. The feature extraction unit is configured to extract the eye region from the image group, extract the blink amplitude from the image group, detect an image determined to be the start of a blink from the image group, store an image in the image group determined to be the end of the blink as first data, and store an image in the image group after an arbitrary time from the first data as second data, The feature extraction unit is configured to extract area information of the white eye region from the first data and the second data. The feature extraction unit is configured to provide the area information of the white eye region to the feature estimation unit, and The feature inference unit is configured to output a classification result.

2. A classification device, comprising: A computing unit, a storage device and a camera device, wherein the storage device includes a program having a feature extraction unit and a feature estimation unit, The computing unit is configured to execute the program. The camera device is configured to capture a group of images. The image group includes images of eye regions, The eye region has a white eye region, The white eye area is the area of ​​the eyeball covered by the white membrane. The feature extraction unit is configured to extract the eye region from the image group, extract the blink amplitude from the image group, detect an image determined to be the start of a blink from the image group, store an image in the image group determined to be the end of the blink as first data, and store an image in the image group after an arbitrary time from the first data as second data, The feature extraction unit is configured to extract area information of the white eye region from the first data and the second data. The feature extraction unit is configured to provide the area information of the white eye region to the feature estimation unit, and The feature inference unit is configured to output a classification result.

3. The classification device according to claim 1 or 2, The feature extraction unit includes a blink detection unit, And the blink detection unit is configured to extract the eye region from each of the image groups.

4. The classification device according to claim 1 or 2, The feature extraction unit includes an information detection unit, And the information detection section is configured to extract an area ratio between a first area of ​​the white eye region and a second area of ​​the white eye region.

5. The classification device according to claim 1 or 2, The feature estimation unit is configured to classify human emotions.

6. The classification device according to claim 1 or 2, The feature estimation unit is configured to classify the health status.

Images