Fatigue degree evaluation system

Abstract

To provide a fatigue degree evaluation system.SOLUTION: A fatigue degree evaluation system comprises an accumulation unit, a generation unit, a storage unit, an acquisition unit and a measurement unit. The accumulation unit has a function for accumulating a plurality of first images and a plurality of second images, the first images being the images of eyes and around the eyes acquired from a side face or an oblique direction, and the second images being the images of the eyes and around the eyes acquired from a front face. The generation unit has a function for performing learning with a teacher to generate a learned model. The storage unit has a function for storing the learned model. The acquisition unit has a function for acquiring third images, the third images being the images of the eyes and around the eyes acquired from the side face or the oblique direction. The measurement unit has a function for measuring a fatigue degree from the third images on the basis of the learned model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] 1. Field of the Invention An aspect of the present invention relates to a method for evaluating a fatigue level, a system for evaluating a fatigue level, and an apparatus for evaluating a fatigue level. [Background technology]

[0002] In modern society, proper management of workers' health is an important issue, as it leads not only to the health of workers but also to improved labor productivity and the prevention of accidents. Of course, proper management of health is not limited to workers, but is also an important issue for students, housewives, and others.

[0003] Deterioration of health is caused by the accumulation of fatigue. Fatigue can be divided into physical fatigue, mental fatigue, and nervous fatigue. Symptoms caused by the accumulation of physical fatigue are relatively easy to notice. On the other hand, symptoms caused by the accumulation of mental fatigue or nervous fatigue are often difficult to notice. Recently, there has been an increase in visually demanding VDT (Visual Display Terminal) work, creating an environment in which nervous fatigue can easily accumulate.

[0004] One of the causes of fatigue is psychological stress (also simply called stress). It is said that chronic fatigue leads to autonomic nervous system disorders. In recent years, therefore, methods of measuring fatigue levels and stress states using machine learning and the like have been attracting attention. Patent Document 1 discloses a method of detecting mental fatigue using blinking light. Patent Document 2 discloses an autonomic nervous function and stress level evaluation device equipped with a machine learning device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-301841 A [Patent Document 2] JP 2008-259609 A Summary of the Invention [Problem to be solved by the invention]

[0006] When the detection device and evaluation device disclosed in Patent Document 1 and Patent Document 2 are used to evaluate fatigue and stress levels, if the user is a worker, he or she needs to interrupt work related to the labor, which may reduce labor productivity. In addition, the user may visually recognize the detection device, which may cause further mental fatigue to accumulate in addition to the mental fatigue accumulated before using the detection device. Therefore, it is difficult to detect mental fatigue correctly.

[0007] In view of the above, an object of one embodiment of the present invention is to evaluate a degree of fatigue. Another object of one embodiment of the present invention is to evaluate a degree of fatigue while suppressing a decrease in labor productivity.

[0008] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0009] In view of the above problems, one aspect of the present invention provides a system (fatigue level evaluation system) that performs machine learning using information about the eyes and their surroundings as learning data to generate a pre-trained model and evaluates a fatigue level based on information about the eyes and their surroundings acquired from a position that is difficult for a user to visually recognize. Another aspect of the present invention provides an instrument and an electronic device equipped with the fatigue level evaluation system.

[0010] One aspect of the present invention is a fatigue evaluation system having an accumulation unit, a generation unit, a memory unit, an acquisition unit, and a measurement unit. The accumulation unit has a function of accumulating a plurality of first images and a plurality of second images. The plurality of first images are images of the eye and its surroundings acquired from the side or an oblique direction. The plurality of second images are images of the eye and its surroundings acquired from the front. The generation unit has a function of performing supervised learning and generating a trained model. The memory unit has a function of storing the trained model. The acquisition unit has a function of acquiring a third image. The third image is an image of the eye and its surroundings acquired from the side or an oblique direction. The measurement unit has a function of measuring the fatigue level from the third image based on the trained model.

[0011] In the above-described fatigue evaluation system, it is preferable that at least one of pupils and blinking is provided as teaching data in the supervised learning.

[0012] In the fatigue evaluation system, it is preferable that one of the plurality of first images and one of the plurality of second images are acquired at the same time.

[0013] In the above-mentioned fatigue evaluation system, the side or oblique direction is preferably an angle of 60° to 85° horizontally with respect to the line of sight.

[0014] In addition, the fatigue evaluation system preferably has an output unit. In addition, the output unit preferably has a function of providing information.

[0015] Another aspect of the present invention is a fatigue assessment device including, among the fatigue assessment systems, glasses having a memory unit, an acquisition unit, and a measurement unit, and a server having an accumulation unit and a generation unit. Effect of the Invention

[0016] According to one embodiment of the present invention, the degree of fatigue can be evaluated. In addition, according to one embodiment of the present invention, the degree of fatigue can be evaluated while suppressing a decrease in labor productivity.

[0017] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from descriptions in the specification, drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a fatigue level evaluation system. [Diagram 2] FIG. 2 is a flow chart showing an example of a method for evaluating a fatigue level. [Diagram 3] 3A to 3C are diagrams for explaining a method for photographing the eye and its surroundings. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a CNN. [Diagram 5] 5A and 5B are diagrams for explaining a method for photographing the eye and its surroundings. [Figure 6] 6A and 6B are schematic diagrams of a human visual field. [Figure 7] 7A and 7B are schematic diagrams showing changes in pupil diameter over time. [Figure 8] 8A and 8B are diagrams illustrating an instrument and an electronic device in which the fatigue evaluation system is incorporated. [Figure 9] Fig. 9A is a diagram illustrating an instrument in which a part of the fatigue evaluation system is incorporated, and Fig. 9B is a diagram illustrating an electronic device in which a part of the fatigue evaluation system is incorporated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be easily understood by those skilled in the art that the modes and details of the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0020] In the configuration of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and the repeated explanations are omitted. In addition, when referring to similar functions, the same hatch pattern may be used and no particular reference numeral may be used.

[0021] In addition, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent the actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0022] It should also be noted that the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and are not intended to limit the numbers.

[0023] (Embodiment 1) In this embodiment, a fatigue evaluation system and a method for evaluating a fatigue level, which are one embodiment of the present invention, will be described with reference to FIGS. 1 to 7B.

[0024] <Example of fatigue evaluation system configuration> First, a configuration example of a fatigue evaluation system will be described with reference to FIG.

[0025] 1 is a diagram showing an example of the configuration of a fatigue level assessment system 100. The fatigue level assessment system 100 includes an accumulation unit 101, a generation unit 102, an acquisition unit 103, a storage unit 104, a measurement unit 105, and an output unit 106.

[0026] The accumulation unit 101, the generation unit 102, the acquisition unit 103, the storage unit 104, the measurement unit 105, and the output unit 106 are connected to each other via a transmission path. The transmission path includes a network such as a local area network (LAN) or the Internet. The network can use either or both of wired and wireless communication.

[0027] In addition, when wireless communication is used in the above network, in addition to short-range communication means such as Wi-Fi (registered trademark) and Bluetooth (registered trademark), various communication means can be used, such as communication means compliant with the third generation mobile communication system (3G), communication means compliant with LTE (sometimes called 3.9G), communication means compliant with the fourth generation mobile communication system (4G), or communication means compliant with the fifth generation mobile communication system (5G).

[0028] The storage unit 101 stores learning data.

[0029] The generation unit 102 has a function of performing machine learning.

[0030] The acquisition unit 103 has a function of acquiring information. Here, the information acquired by the acquisition unit 103 is information on the eyes and their surroundings. The acquisition unit 103 is, for example, one or more selected from a camera, a pressure sensor, a distortion sensor, a temperature sensor, a gyro sensor, and the like.

[0031] The storage unit 104 stores the information acquired by the acquisition unit 103. In addition, a trained model is also stored.

[0032] There are cases where the storage unit 104 does not need to be provided. For example, this is the case when the trained model and the information acquired by the acquisition unit 103 are stored in the accumulation unit 101.

[0033] The measurement unit 105 has a function of measuring the degree of fatigue. The function of measuring the degree of fatigue includes a function of calculating the degree of fatigue and a function of determining whether the degree of fatigue is abnormal.

[0034] The output unit 106 has a function of providing information. The information is the fatigue level calculated by the measurement unit 105, the determination result as to whether the fatigue level is abnormal, etc. Components included in the output unit 106 include a display, a speaker, etc.

[0035] The above is a description of an example of the configuration of the fatigue evaluation system 100.

[0036] <How to assess fatigue level> Next, an example of a method for evaluating the fatigue level will be described with reference to FIGS. 2 to 7B.

[0037] As mentioned above, chronic fatigue is said to lead to autonomic nervous system disorders. The autonomic nervous system consists of the sympathetic nervous system, which is active during physical activity, during the day, and when under stress, and the parasympathetic nervous system, which is active at rest, at night, and when relaxed. When the sympathetic nervous system is dominant, the pupils dilate, the heart beats faster, and blood pressure rises. On the other hand, when the parasympathetic nervous system is dominant, the pupils constrict, the heart beat is suppressed, blood pressure drops, and drowsiness occurs.

[0038] When the balance of the autonomic nervous system becomes poor, it can cause hypothermia, decreased blinking and tear production, etc. In addition, slouching or arching your back for long periods of time can also lead to autonomic nervous system disorders.

[0039] From the above, if the disturbance or balance of the autonomic nervous system can be evaluated, the degree of fatigue can be evaluated objectively. In other words, the degree of fatigue can be evaluated objectively by evaluating the changes over time in the pupils (pupil diameter or pupil area), heartbeat or pulse, blood pressure, body temperature, blinking, posture, etc.

[0040] Fig. 2 is a flow diagram showing an example of a method for evaluating a fatigue level. The method for evaluating a fatigue level has steps S001 to S006 shown in Fig. 2. Steps S001 and S002 are steps for generating a trained model, and steps S003 to S006 are steps for measuring a fatigue level. In other words, the method for evaluating a fatigue level includes a method for generating a trained model and a method for measuring a fatigue level.

[0041] [How to generate a trained model] First, an example of a method for generating a trained model will be described. The method for generating a trained model includes steps S001 and S002 shown in FIG.

[0042] In step S001, learning data used to generate a trained model is prepared. For example, information on the eyes and their surroundings is acquired as the learning data. In other words, step S001 can be rephrased as a process of acquiring information on the eyes and their surroundings. As will be described later, it is preferable that the information on the eyes and their surroundings is acquired, for example, from the side and the front.

[0043] The information on the eyes and their surroundings is acquired using one or more selected from a camera, a pressure sensor, a distortion sensor, a temperature sensor, a gyro sensor, etc. The information on the eyes and their surroundings may be acquired using a publicly available data set.

[0044] The learning data may include teacher data (also called teacher signals, correct labels, etc.) such as pupils (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, bloodshot eyes, etc. In particular, pupils (pupil diameter or pupil area) and blinking tend to change over time due to mental fatigue, and are therefore preferable as teacher data.

[0045] The information on the eyes and their surroundings prepared as learning data is stored in storage unit 101. After the learning data is stored in storage unit 101, the process proceeds to step S002.

[0046] In step S002, machine learning is performed based on the learning data stored in the storage unit 101. The machine learning is performed by the generation unit .

[0047] For the above machine learning, it is preferable to use, for example, supervised learning, and it is more preferable to use supervised learning that utilizes a neural network (particularly, deep learning).

[0048] As deep learning, it is preferable to use, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder (AE), a variational autoencoder (VAE), or the like.

[0049] A trained model is generated by the above machine learning. The trained model is stored in the storage unit 104.

[0050] In addition, the pupils (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, bloodshot eyes, etc., which are given as training data, vary from person to person depending on age, body type, sex, etc. Therefore, the trained model may be updated to suit the user.

[0051] The above is an example of a method for generating a trained model.

[0052] [How to measure fatigue level] Next, an example of a method for measuring a fatigue level will be described. The method for measuring a fatigue level has steps S003 to S006 shown in Fig. 2. The method for measuring a fatigue level includes a method for calculating a fatigue level and a method for determining whether the fatigue level is abnormal.

[0053] In step S003, information on the eyes and their surroundings is obtained for use in calculating the degree of fatigue.

[0054] It is preferable that the information on the eyes and their surroundings used for calculating the degree of fatigue is obtained, for example, from a side view or an oblique direction. By obtaining the information on the eyes and their surroundings from a side view or an oblique direction, the information can be obtained from a position that is difficult for the user to visually recognize. Therefore, the information can be obtained without the user being aware of it.

[0055] The information on the eyes and their surroundings used to calculate the degree of fatigue is obtained using one or more sensors selected from a camera, a pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, and the like.

[0056] Furthermore, information on the eyes and their surroundings to be used in calculating the degree of fatigue is acquired in chronological order.

[0057] Information about the eyes and their surroundings to be used in calculating the degree of fatigue is stored in the storage unit 104. After the information is stored in the storage unit 104, the process proceeds to step S004.

[0058] In step S004, the degree of fatigue is calculated. The fatigue degree is calculated using the trained model generated in step S002 and the information on the eye and its surroundings acquired in step S003. The fatigue degree is calculated by the measurement unit 105.

[0059] Calculating the degree of fatigue refers to quantifying an index for evaluating the degree of fatigue, such as at least one of pupil (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, and bloodshot eyes.

[0060] The calculation of the degree of fatigue is not limited to the quantification of an index for evaluating the degree of fatigue. For example, the degree of fatigue may be quantified from the information on the eye and its surroundings acquired in step S003 using a trained model.

[0061] Before proceeding to step S005, steps S003 and S004 are repeatedly performed for a certain period of time, thereby obtaining time-series data for determining whether an abnormality has occurred in the index for evaluating the fatigue level.

[0062] In step S005, it is determined whether an abnormality has occurred in the index for evaluating the fatigue level.

[0063] If it is determined that an abnormality has occurred in the index for evaluating the degree of fatigue, the degree of fatigue is determined to be high. If it is determined that the degree of fatigue is high, the process proceeds to step S006. On the other hand, if it is determined that no abnormality has occurred in the index for evaluating the degree of fatigue, the degree of fatigue is determined to be not high. If it is determined that the degree of fatigue is not high, the process proceeds to step S003.

[0064] When the fatigue level is quantified in step S004, a determination is made as to whether or not there is an abnormality in the fatigue level numerical value. If it is determined that there is an abnormality in the fatigue level numerical value, the fatigue level is determined to be high. If it is determined that the fatigue level is high, the process proceeds to step S006. On the other hand, if it is determined that there is no abnormality in the fatigue level numerical value, the fatigue level is determined to be not high. If it is determined that the fatigue level is not high, the process proceeds to step S003.

[0065] In step S006, information is output. The information is an index for evaluating the degree of fatigue calculated by the measurement unit 105, a quantified degree of fatigue, a determination result as to whether the degree of fatigue is abnormal, etc. The information is output as visual information such as character strings, numerical values, graphs, and colors, and auditory information such as voice and music, for example.

[0066] After outputting the above information, the process ends.

[0067] The above is a description of an example of a method for calculating a fatigue level.

[0068] This concludes the explanation of one example of the method for evaluating fatigue level.

[0069] <<Specific examples of methods for assessing fatigue level>> In this section, a specific example of a method for evaluating the degree of fatigue will be described with reference to Figures 3A to 7B. Here, a change in pupil (pupil diameter or pupil area) over time is selected as an index for evaluating the degree of fatigue.

[0070] As the learning data prepared in step S001, for example, image data of the eyes and their surroundings is used. In this case, the image data of the eyes and their surroundings is preferably image data of the eyes and their surroundings acquired from the front and image data of the eyes and their surroundings acquired from the side or oblique direction. Compared with image data acquired from the side or oblique direction, image data acquired from the front allows the pupil (pupil diameter or pupil area) to be detected with high accuracy. Therefore, by using image data acquired from the front and image data acquired from the side or oblique direction as learning data, a trained model with higher accuracy can be generated than when only image data acquired from the side or oblique direction is used as learning data.

[0071] Images of the eyes and their surroundings for use as learning data may be obtained, for example, by photographing them from the front, side, or obliquely using a camera or the like.

[0072] 3A to 3C show examples of photographing from the front, side, or oblique direction using cameras 111a to 111d. FIG. 3A is a view of the subject of photographing from above. FIG. 3B is a view of the subject of photographing from the right side. FIG. 3C is a view of the subject of photographing from the front. For clarity of the figures, cameras 111a, 111b, and 111d are omitted in FIG. 3B, and cameras 111c and 111d are omitted in FIG. 3C. The subject of photographing does not have to be limited to the person (user) whose fatigue level is to be evaluated.

[0073] 3A to 3C, the eye and its surroundings from the front are photographed using camera 111c and camera 111d, while the eye and its surroundings from the side or oblique direction are photographed using camera 111a and camera 111b.

[0074] Before machine learning, image data for learning data may be processed or corrected. Examples of image data processing or correction include cutting out parts unnecessary for machine learning, converting to grayscale, using a median filter, a Gaussian filter, etc. Processing or correcting image data can reduce noise generated during machine learning.

[0075] It is preferable to prepare a plurality of combinations of image data of the eye and its surroundings acquired from the front and image data of the eye and its surroundings acquired from the side or oblique direction, which are acquired at the same time, as the learning data. By photographing the eye and its surroundings from a plurality of angles at the same time, the above processing or correction can be facilitated. Thus, a trained model with high accuracy can be generated. For example, processing or correction is performed on the image data of the eye and its surroundings acquired from the side or oblique direction, taking into consideration the image data of the eye and its surroundings acquired from the front. This enhances the pupil outline in the image data of the eye and its surroundings acquired from the side or oblique direction, and the pupil (pupil diameter or pupil area) can be detected with high accuracy.

[0076] In addition, when many images obtained from the side or oblique direction can be obtained, only the image data obtained from the side or oblique direction may be used as learning data. Also, when image data of the eye and its surroundings obtained from the side or oblique direction is processed or corrected in consideration of image data of the eye and its surroundings obtained from the front, only the image data obtained from the side or oblique direction may be used as learning data.

[0077] As described above, since the learning data is image data, it is preferable to use a convolutional neural network for the machine learning performed in step S002.

[0078] [Convolutional Neural Network] Here, we will explain convolutional neural networks (CNNs).

[0079] FIG. 4 shows an example of the configuration of a CNN. The CNN is composed of a convolution layer CL, a pooling layer PL, and a fully connected layer FCL. Image data IPD is input to the CNN, and feature extraction is performed. In this embodiment, the image data IPD is image data of the eye and its surroundings.

[0080] The convolution layer CL has the function of performing convolution on image data. Convolution is performed by repeating product-sum operations between a part of the image data and the filter value of a weight filter (also called a kernel). Image features are extracted by the convolution in the convolution layer CL.

[0081] The above-mentioned product-sum calculation may be performed on software using a program, or may be performed by hardware. When the product-sum calculation is performed by hardware, a product-sum calculation circuit may be used. As the product-sum calculation circuit, a digital circuit or an analog circuit may be used.

[0082] The product-sum circuit may be formed using transistors having Si in a channel formation region (also referred to as Si transistors) or transistors using a metal oxide in a channel formation region (also referred to as OS transistors). In particular, OS transistors have an extremely small off-state current and are therefore suitable as transistors forming an analog memory of the product-sum circuit. Note that the product-sum circuit may be formed using both Si transistors and OS transistors.

[0083] One or more weight filters can be used for the convolution. When multiple weight filters are used, it is possible to extract multiple features contained in the image data. Figure 4 shows three weight filters (filter F a , F b , F c ) is used. The image data input to the convolution layer CL is filtered using the filter F a , F b , F c The image data D a , D b , D c The image data D a , D b , D c is also called a feature map.

[0084] Image data D generated by convolution a , D b , D c is converted by an activation function and then output to the pooling layer PL. As the activation function, ReLU (Rectified Linear Units) or the like can be used. ReLU is a function that outputs "0" when the input value is negative, and outputs the input value as is when the input value is "0" or greater. In addition, a sigmoid function, a tanh function, or the like can also be used as the activation function.

[0085] The pooling layer PL has a function of performing pooling on the image data input from the convolution layer CL. Pooling is a process of dividing the image data into a plurality of regions, extracting predetermined data for each region, and arranging the data in a matrix. Pooling reduces the spatial size of the image data while leaving the features extracted by the convolution layer CL. It is also possible to improve the position invariance or translation invariance of the features extracted by the convolution layer CL. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as the pooling method.

[0086] CNN extracts features by the above-mentioned convolution process and pooling process. Note that CNN can be configured with multiple convolution layers CL and multiple pooling layers PL. In FIG. 4, z layers (z is an integer equal to or greater than 1) of layers L each composed of a convolution layer CL and a pooling layer PL are provided (layers L1 to L z ), the convolution process and pooling process are performed z times. In this case, feature extraction can be performed in each layer L, enabling more advanced feature extraction.

[0087] The fully connected layer FCL has the function of judging an image using image data that has been subjected to convolution and pooling. All nodes of the fully connected layer FCL are connected to the previous layer of the fully connected layer FCL (here, the pooling layer PL, in Figure 4, the layer L z The image data output from the convolution layer CL or the pooling layer PL is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. The one-dimensional expanded data OPD is then output.

[0088] The configuration of the CNN is not limited to that shown in Fig. 4. For example, a pooling layer PL may be provided for each of a plurality of convolution layers CL. In addition, when it is desired to retain as much position information of extracted features as possible, the pooling layer PL may be omitted.

[0089] Furthermore, when classifying images from output data of the fully connected layer FCL, an output layer electrically connected to the fully connected layer FCL may be provided. The output layer can output the probability of classification into each class using a softmax function or the like as a likelihood function. The class to be classified may be, for example, the degree of fatigue. Specifically, the classes to be classified may be "very high fatigue", "high fatigue", "medium fatigue", "low fatigue", "very low fatigue", etc. This allows the fatigue level to be quantified from the image data.

[0090] In addition, when performing regression analysis such as numerical prediction from the output data of the fully connected layer FCL, an output layer electrically connected to the fully connected layer FCL may be provided. By using an identity function or the like in the output layer, a predicted value can be output. This makes it possible to calculate, for example, pupil diameter or pupil area from image data.

[0091] CNN can also perform supervised learning using image data as learning data with teacher data added. For example, backpropagation can be used for supervised learning. CNN learning can optimize the filter values ​​of the weight filter, the weight coefficients of the fully connected layer, and so on.

[0092] This concludes the explanation of convolutional neural networks (CNNs).

[0093] In the above-mentioned supervised learning, image data of the eye and its surroundings acquired from the front and image data of the eye and its surroundings acquired from the side or oblique direction are prepared as learning data, and the system is trained to output the pupil diameter or pupil area. For example, when the pupil diameter or pupil area is given as the teacher data, the pupil diameter or pupil area is output by regression using CNN. In this way, a trained model is generated that outputs the pupil diameter or pupil area from the image data of the eye and its surroundings acquired from the side or oblique direction.

[0094] The quantified fatigue level may be output by classifying the eyes using CNN. In this case, a trained model that outputs the quantified fatigue level is generated from image data of the eyes and their surroundings acquired from the side or oblique direction.

[0095] In step S003, information on the eyes and their surroundings is acquired for use in calculating the degree of fatigue. For example, an image of the eyes and their surroundings acquired from a side or oblique direction is acquired as the information on the eyes and their surroundings. The image of the eyes and their surroundings may be acquired by photographing them from the side using a camera or the like.

[0096] 5A and 5B show examples of capturing images from the side or oblique direction using cameras 112a and 112b. Fig. 5A is a diagram of the subject of the image capturing from above. Fig. 5B is a diagram of the subject of the image capturing from the front. The subject of the image capturing is a person (user) whose fatigue level is to be evaluated.

[0097] As shown in FIGS. 5A and 5B, the eye and its surroundings are photographed from the side or oblique direction using cameras 112a and 112b.

[0098] It is preferable that the distance to the subject is approximately equal between camera 111 (one or more of cameras 111a to 111d) shown in Fig. 3A and camera 112 (camera 112a and / or camera 112b) shown in Fig. 5A. This allows the fatigue level to be evaluated with high accuracy. In the method for evaluating the fatigue level according to one aspect of the present invention, since supervised learning is performed, the distance to the subject is not necessarily equal between camera 111 and camera 112.

[0099] In addition, it is preferable that the image captured by the camera 111 and the image captured by the camera 112 have the same resolution, aspect ratio, and the like. This allows the fatigue level to be evaluated with high accuracy. Note that in the method for evaluating the fatigue level according to one embodiment of the present invention, supervised learning is performed, so that the image captured by the camera 111 and the image captured by the camera 112 do not necessarily have to have the same resolution, aspect ratio, and the like.

[0100] 6A and 6B are schematic diagrams of a human visual field (binocular vision). Fig. 6A is a diagram of a person viewed from above. Fig. 6B is a diagram of a person viewed from the right side.

[0101] The human visual field is classified into the effective visual field, the induced visual field, the auxiliary visual field, etc. In FIG. 6A and FIG. 6B, the dashed line from the person to the fixation point C is the line of sight (visual axis), and the angle θ 1h and angle θ 1vis the range of the effective field of view, and the angle θ 2h and angle θ 2v is the range of the visual angle of the induced visual field, and the angle θ 3h and angle θ 3v is the range of the visual angle of the auxiliary visual field. Unless otherwise specified, the line of sight refers to the line from the person to the gaze point C when the gaze point is located at a position where the length of the line segment connecting the gaze point C and the right eye is equal to the length of the line segment connecting the gaze point C and the left eye. In addition, the horizontal direction refers to the direction horizontal to the plane including both eyes and the gaze. In addition, the vertical direction refers to the direction perpendicular to the plane including both eyes and the gaze.

[0102] The effective visual field is the area in which information can be received instantly. The horizontal visual angle of the effective visual field (angle θ 1h ) is said to be in the range of about 30° from the center of the line of sight, and the vertical viewing angle of the effective visual field (angle θ 1v ) is said to be in the range of about 20 degrees, centered slightly below the line of sight.

[0103] The induced visual field is an area that affects the spatial coordinate system. The horizontal visual angle of the induced visual field (angle θ 2h ) is said to be in the range of about 100° from the center of the line of sight, and the vertical visual angle of the induced visual field (angle θ 2v ) is said to be in the range of about 85°, centered slightly below the line of sight.

[0104] The auxiliary visual field is an area where the presence of the stimulus can be recognized. The horizontal visual angle of the auxiliary visual field (angle θ 3h ) is said to be in the range of about 200° from the center of the line of sight, and the vertical visual angle of the auxiliary field (angle θ 3v ) is said to be in the range of about 125°, centered slightly below the line of sight.

[0105] When working, most information is received from the effective visual field, and a small amount is received from the induced visual field. In addition, there is almost no information from the auxiliary visual field during work. In other words, it is difficult for workers to recognize information located in the auxiliary visual field.

[0106] Furthermore, when selecting a change in pupil over time as an index for evaluating fatigue level, the image acquired in step S003 needs to include the pupil. Visual information is recognized when an image projected onto the retina through the pupil, crystalline lens, etc. is transmitted to the brain via the optic nerve. In other words, since the auxiliary visual field also includes visual information, the pupil can be recognized within the auxiliary visual field.

[0107] From the above, the side or oblique direction in which the image of the eye and its surroundings is acquired is the direction in which the pupil is observed from within the auxiliary visual field or within the induced visual field near the auxiliary visual field in the horizontal direction, and is the angle θ a The side or oblique direction is specifically in the range of 45° to 100°, preferably 50° to 90°, and more preferably 60° to 85°, in the horizontal direction relative to the line of sight. This allows the image to be acquired from a position that is difficult for the user to visually recognize. Therefore, the image can be acquired without the user being aware of it.

[0108] In addition, when the side or oblique direction is within the above range in the horizontal direction, any angle in the vertical direction is outside the field of view of the induced field of view. Therefore, the side or oblique direction in the vertical direction may be any direction within the range in which the pupil can be photographed.

[0109] The front, where the image of the eye and its surroundings is acquired, is the direction in which the pupil is observed from within the induced visual field in the horizontal direction. Specifically, the front is in the range of 0° to 50°, preferably 0° to 30°, more preferably 0° to 15° in the horizontal direction relative to the line of sight. This allows the pupil to be photographed in a circular or nearly circular shape, and the pupil diameter or pupil area to be calculated with high accuracy.

[0110] If the change in pupil size over time is selected as an index for evaluating fatigue level, in step S004, the trained model is used to calculate pupil diameter or pupil area from image data of the eye and its surroundings acquired from the side or oblique direction.

[0111] As mentioned above, when the sympathetic nervous system is dominant, the pupils expand, and when the parasympathetic nervous system is dominant, the pupils contract. In other words, the pupil diameter changes with the disturbance of the autonomic nervous system. It is also said that the speed of change of the pupil slows down with the accumulation of fatigue. In this specification, the change over time of the pupil (pupil diameter or pupil area) refers to the change over time of the pupil (pupil diameter or pupil area), the speed of change of the pupil (pupil diameter or pupil area), the change over time of the expansion and contraction cycle of the pupil (pupil diameter or pupil area), etc.

[0112] Whether or not an abnormality has occurred in the change over time of the pupil (pupil diameter or pupil area) is determined based on the pupil (pupil diameter or pupil area) immediately after the start of step S003.

[0113] An example of a method for determining whether or not an abnormality has occurred in the change over time of the pupil (pupil diameter or pupil area) will be described with reference to FIGS. 7A and 7B.

[0114] 7A and 7B are schematic diagrams showing changes in pupil diameter over time. In FIG. 7A and FIG. 7B, the horizontal axis is time, and the vertical axis is pupil diameter. The solid lines in FIG. 7A and FIG. 7B show changes in pupil diameter over time. The dashed and dotted lines in FIG. 7A and FIG. 7B show the time average of pupil diameter.

[0115] FIG. 7A is a diagram showing a typical state in which the pupil diameter decreases over time. In order to determine whether the change in pupil diameter over time is abnormal, a threshold value of the pupil diameter is set in advance. For example, as shown by the dashed line in FIG. 7A, the upper limit of the pupil diameter is set to r max Let the lower limit of pupil diameter be r min In the example of FIG. 7A, the pupil diameter at time t is the lower limit r min At this time, it is determined that an abnormality has occurred in the change in the pupil over time.

[0116] For example, if the pupil (pupil diameter or pupil area) expands or contracts at a rate equal to or greater than a certain level based on the pupil (pupil diameter or pupil area) immediately after the start of step S003, it is determined that an abnormality has occurred.

[0117] FIG. 7B is a schematic diagram showing how the expansion and contraction cycle of the pupil diameter increases over time. u (u is a natural number). In order to determine whether the change in pupil diameter over time is abnormal, a threshold value for the expansion and contraction cycle of the pupil diameter is set in advance. For example, as shown in FIG. 7B, the upper limit of the expansion and contraction cycle of the pupil diameter is set to f max Let f be the lower limit of the pupil diameter expansion and contraction period. min In the example of FIG. 7B, the pupil diameter expansion and contraction period f t+7 is the upper limit of the pupil diameter expansion and contraction period f max In this case, it is determined that an abnormality has occurred in the change in the pupil over time.

[0118] The enlargement and contraction of the pupil diameter and the expansion and contraction cycle of the pupil diameter are observed together. For example, a fast Fourier transform may be performed on the change in pupil diameter over time. This makes it easier to determine whether there is an abnormality based on the expansion and contraction cycle of the pupil diameter.

[0119] Although the method of calculating the pupil diameter or pupil area from image data of the eye and its surroundings acquired from the side or oblique direction using the trained model has been exemplified, the method is not limited to this. For example, the fatigue level may be quantified from image data of the eye and its surroundings acquired from the side or oblique direction using the trained model. At this time, a threshold value of the fatigue level (upper limit of the fatigue level) is set in advance to determine whether the quantified fatigue level is abnormal.

[0120] The above is a description of the fatigue evaluation system. By using the fatigue evaluation system according to one aspect of the present invention, the system (particularly the acquisition unit) is not located in the line of sight of the user, so that the increase in the user's mental fatigue is suppressed. Therefore, the fatigue level during use can be evaluated with high accuracy.

[0121] The structures, methods, and the like described in this embodiment can be used in appropriate combination with structures, methods, and the like described in other embodiment modes.

[0122] (Embodiment 2) In this embodiment, a fatigue evaluation device will be described with reference to Fig. 8A to Fig. 9B. The fatigue evaluation device is an electronic device, or an instrument and an electronic device, that includes the fatigue evaluation system described in the previous embodiment.

[0123] Examples of devices that include part of the fatigue evaluation system include eyeglasses such as vision correction glasses and protective glasses, and safety equipment worn on the head such as helmets and gas masks.

[0124] The above-mentioned device includes at least the acquisition unit 103 of the fatigue level evaluation system described in the above embodiment. Also, the above-mentioned device includes a battery.

[0125] Examples of electronic devices that include a part of the fatigue evaluation system include information terminals, computers, etc. Here, the term "computer" refers to a tablet computer, a notebook computer, a desktop computer, as well as large computers such as a workstation and a server system.

[0126] The electronic device may be configured to use a GPS (Global Positioning System) receiver to acquire data on the position, travel distance, acceleration, etc. of the device. The acquired data may be combined with an index for evaluating the degree of fatigue to evaluate the degree of fatigue with higher accuracy.

[0127] An example of an instrument and electronic device equipped with the fatigue evaluation system is shown in Fig. 8A. Fig. 8A shows glasses 200 and a server 300 equipped with the fatigue evaluation system. Glasses 200 include a processing unit 201. Server 300 also includes a processing unit 301.

[0128] For example, processing unit 201 has acquisition unit 103 described in the previous embodiment, and processing unit 301 has accumulation unit 101, generation unit 102, storage unit 104, and measurement unit 105 described in the previous embodiment. Processing unit 201 and processing unit 301 each have a transmission / reception unit. Processing unit 201 has only acquisition unit 103, so that glasses 200 having processing unit 201 can be made lighter. Therefore, the burden on the user's body when wearing glasses 200 can be reduced.

[0129] Furthermore, when a camera is used as the acquisition unit 103, the acquisition unit 103 can be arranged in the frame of the glasses 200 near the eyes to take a close-up image of the eyes and their surroundings. This makes it easier to detect the eyes. Also, it is possible to reduce the reflection of the outside world on the eyes. This makes it possible to reduce the number of times that the images of the eyes and their surroundings are processed or corrected. Or, processing or correction becomes unnecessary.

[0130] 8A shows an example in which a camera is used as the acquisition unit 103, but the invention is not limited thereto. A pressure sensor, a distortion sensor, a temperature sensor, a gyro sensor, or the like may be used as the acquisition unit 103. In this case, the acquisition unit 103 may be installed in a direction other than the side of the eye or the diagonal direction. For example, the acquisition unit 103 may be installed at or near the point where the head and the frame of the eyeglasses 200 come into contact.

[0131] The processing unit 201 may include the output unit 106 described in the previous embodiment. By including the output unit 106 in the processing unit 201, the user can know the degree of fatigue while working. Components included in the output unit 106 include a display, a speaker, and the like.

[0132] It is preferable that the information provided by the output unit 106 is output as visual information such as color, or auditory information such as voice or music. Compared with visual information such as character strings, numerical values, and graphs, visual information such as color has less impact on the sense of sight and causes less stress to the user, and is therefore preferable. The same applies to auditory information such as voice and music. It is possible that the user's degree of fatigue can be reduced by registering favorite music or the like in advance as auditory information.

[0133] Note that the configurations of the processing unit 201 and the processing unit 301 are not limited to this. For example, the processing unit 201 may have the acquisition unit 103, the storage unit 104, the measurement unit 105, and the output unit 106, and the processing unit 301 may have the accumulation unit 101 and the generation unit 102. In this case, the processing unit 201 has a function of measuring the fatigue level, and the processing unit 301 has a function of generating a trained model.

[0134] With the above configuration, the fatigue level can be measured only by the processing unit 201, and therefore the frequency of communication between the processing unit 201 and the processing unit 301 can be minimized. Furthermore, with the above configuration, the processing unit 301 can transmit the trained model updated by the processing unit 301 to the processing unit 201, and the processing unit 201 can receive the trained model. Then, the trained model recorded in the processing unit 201 can be updated to the received trained model. This makes it possible to use trained data with improved accuracy, and to evaluate the fatigue level with higher accuracy.

[0135] The storage unit 104 may store information about the eye and its surroundings acquired by the acquisition unit 103. After a certain amount of the acquired information about the eye and its surroundings is accumulated in the storage unit 104, the accumulated information may be transmitted to an electronic device having the processing unit 301. This makes it possible to reduce the number of communications between the processing unit 201 and the processing unit 301.

[0136] Note that the electronic device having part of the fatigue evaluation system may be composed of multiple devices. Glasses, a server, and a mobile phone (smartphone), which is a type of information terminal, each equipped with the fatigue evaluation system are shown in FIG. 8B. Like the glasses 200 and server 300 shown in FIG. 8A, the glasses 200 have a processing unit 201, and the server 300 has a processing unit 301. Furthermore, the information terminal 310 has a processing unit 311.

[0137] For example, the processing unit 201 includes an acquisition unit 103. The processing unit 301 includes an accumulation unit 101 and a generation unit 102. The processing unit 311 includes a storage unit 104, a measurement unit 105, and an output unit 106. Furthermore, each of the processing unit 201, the processing unit 301, and the processing unit 311 includes a transmission / reception unit.

[0138] In the above configuration, when the user of the glasses 200 carries the information terminal 310, the user can check his / her own fatigue level via the information terminal 310.

[0139] Furthermore, if the information terminal 310 is carried by the boss of the user of the glasses 200, the boss can check the user's fatigue level via the information terminal 310. Therefore, even if the user and the boss are not close to each other, the boss can manage the user's health condition. Furthermore, if the information output from the output unit 106 is visual information such as a character string, a numerical value, or a graph related to the fatigue level, the boss can know the user's health condition in detail.

[0140] The glasses 200 shown in Figures 8A and 8B are not limited to vision correction glasses, but may also be sunglasses, color vision correction glasses, 3D glasses, Augmented Reality (AR) glasses, Mixed Reality (MR) glasses, fashion glasses, glasses for personal computers with blue light blocking function, etc.

[0141] In particular, in AR glasses, MR glasses, and the like, information on the degree of fatigue can be output as visual information such as character strings, numerical values, and graphs, allowing the degree of fatigue to be known in detail.

[0142] 9A is a diagram showing protective glasses including a part of the fatigue evaluation system. Protective glasses 210 shown in FIG. 9A have a processing unit 211. The processing unit 211 also includes an acquisition unit 103.

[0143] The processing unit 211 may have the same functions as the processing unit 201 of the glasses 200 shown in FIGS. 8A and 8B.

[0144] 9A shows a goggle type as an example of the protective glasses 210, but the present invention is not limited thereto and may be a spectacle type or a front type. Also, while FIG. 9A shows a single-lens type as an example, the present invention is not limited thereto and may be a twin-lens type.

[0145] 8A, 8B, and 9A, eyeglasses such as vision correction eyeglasses and protective eyeglasses are exemplified as an apparatus including a part of the fatigue evaluation system, but the present invention is not limited to these. For example, safety protection equipment worn on the head such as helmets and gas masks can be used.

[0146] So far, the fatigue evaluation device has been described as a combination of an instrument having a part of the fatigue evaluation system and an electronic device having a part of the fatigue evaluation system, but the present invention is not limited to this. The fatigue evaluation device may be, for example, a combination of a removable electronic device having a part of the fatigue evaluation system and the above-mentioned electronic device. FIG. 9B shows a head-worn safety protector 220 to which a removable electronic device 320 having a part of the fatigue evaluation system is attached. The removable electronic device 320 has an acquisition unit 103. By providing a part of the fatigue evaluation system in the removable electronic device 320, it is possible to utilize a safety protector that has been used in the past.

[0147] Furthermore, a part of the fatigue evaluation system may be provided in a display device that is worn on the head, such as a head-mounted display, smart glasses, etc. This allows the fatigue level to be evaluated even in a situation using virtual reality (VR), for example.

[0148] The fatigue evaluation device may be a single instrument or a single electronic device equipped with the fatigue evaluation system.

[0149] By using the fatigue evaluation device according to one embodiment of the present invention, the user's field of vision is ensured and the mental burden on the user is reduced. Therefore, the user's fatigue level can be evaluated with high accuracy. In addition, if the user is a worker, there is no need to interrupt work in order to evaluate the user's fatigue level, and this can prevent a decrease in labor productivity.

[0150] Furthermore, by using the fatigue evaluation device according to one aspect of the present invention, information on the eyes and their surroundings can be obtained from a position close to the eyes, thereby improving the accuracy of the fatigue evaluation.

[0151] The structures, methods, and the like described in this embodiment can be used in appropriate combination with structures, methods, and the like described in other embodiment modes. [Explanation of symbols]

[0152] : 100: fatigue evaluation system, 101: accumulation unit, 102: generation unit, 103: acquisition unit, 104: memory unit, 105: measurement unit, 106: output unit, 111: camera, 111a: camera, 111b: camera, 111c: camera, 111d: camera, 112: camera, 112a: camera, 112b: camera, 200: glasses, 201: processing unit, 210: protective glasses, 211: processing unit, 220: safety protector, 300: server, 301: processing unit, 310: information terminal, 311: processing unit, 320: electronic device

Claims

[Claim 1] The apparatus includes an accumulation unit, a generation unit, a storage unit, an acquisition unit, and a measurement unit, A first step of preparing learning data by acquiring a plurality of sets of first image information of the eye and its surroundings photographed from a side or oblique direction and second image information of the eye and its surroundings photographed from the front, and storing the learning data in the storage unit; a second step of generating a trained model that outputs a quantified pupil diameter when image information of the eye and its surroundings photographed from the side or oblique direction is input by performing supervised learning based on the training data in the generation unit, and storing the trained model in the storage unit; a third step of acquiring third image information by photographing the user's eyes and their surroundings from a side or oblique direction of the user in the acquisition unit, and storing the third image information in the storage unit; a fourth step of quantifying the pupil diameter included as information in the third image information by using the trained model in the measurement unit; a fifth step of repeatedly performing the third step and the fourth step to obtain time series data for determining whether or not an abnormality has occurred in the pupil diameter included as information in the third image information, and then determining whether or not an abnormality has occurred in the digitized pupil diameter obtained by performing the third step and the fourth step; By going through the above, the device has a function of evaluating the fatigue level of the user, In the fifth step, when the digitized pupil diameter falls below a threshold value, it is determined that an abnormality has occurred. Fatigue rating system.

Images