Fatigue Assessment System

Abstract

To provide a fatigue degree evaluation system.SOLUTION: A fatigue degree evaluation system comprises an accumulation unit, a generation unit, a storage unit, an acquisition unit and a measurement unit. The accumulation unit has a function for accumulating a plurality of first images and a plurality of second images, the first images being the images of eyes and around the eyes acquired from a side face or an oblique direction, and the second images being the images of the eyes and around the eyes acquired from a front face. The generation unit has a function for performing learning with a teacher to generate a learned model. The storage unit has a function for storing the learned model. The acquisition unit has a function for acquiring third images, the third images being the images of the eyes and around the eyes acquired from the side face or the oblique direction. The measurement unit has a function for measuring a fatigue degree from the third images on the basis of the learned model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE INVENTION An aspect of the present invention relates to a method for assessing fatigue level, a system for assessing fatigue level, and an apparatus for assessing fatigue level. [Background technology]

[0002] In modern society, proper management of workers' health is an important issue, as it not only contributes to the health of workers but also to improving labor productivity and preventing accidents. Of course, proper management of health is not limited to workers, but is also an important issue for students, housewives, and others.

[0003] Deterioration of health is caused by the accumulation of fatigue. Fatigue can be divided into physical fatigue, mental fatigue, and nervous fatigue. Symptoms that appear due to the accumulation of physical fatigue are relatively easy to notice. On the other hand, symptoms that appear due to the accumulation of mental fatigue and nervous fatigue are often difficult to notice. Recently, there has been an increase in work using visual display terminals (VDTs), which places a heavy burden on the eyes, creating an environment in which nervous fatigue is easily accumulated.

[0004] One of the causes of fatigue is psychological stress (also simply referred to as stress). Chronic fatigue is also said to lead to autonomic nervous system disorders. Therefore, in recent years, methods for measuring fatigue levels and stress states using machine learning and the like have been attracting attention. Patent Document 1 discloses a method for detecting mental fatigue using flashing light. Patent Document 2 discloses an autonomic nervous function and stress level evaluation device equipped with a machine learning device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-301841 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-259609 Summary of the Invention [Problem to be solved by the invention]

[0006] When assessing fatigue and stress levels using the detection and evaluation devices disclosed in Patent Documents 1 and 2, if the user is a worker, they must interrupt their work, which may result in a decrease in labor productivity. Furthermore, by visually recognizing the detection device, the user may accumulate additional mental fatigue in addition to the mental fatigue that had accumulated before using the detection device. Therefore, it is difficult to correctly detect mental fatigue.

[0007] In view of the above, an object of one embodiment of the present invention is to evaluate the degree of fatigue. Another object of one embodiment of the present invention is to evaluate the degree of fatigue while suppressing a decrease in labor productivity.

[0008] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract other problems from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0009] In view of the above-mentioned problems, one aspect of the present invention provides a system (fatigue level assessment system) that generates a pre-trained model by performing machine learning using information about the eyes and their surroundings as training data, and assesses fatigue level based on information about the eyes and their surroundings acquired from a position that is difficult for a user to visually recognize. Another aspect of the present invention provides an instrument and an electronic device equipped with the fatigue level assessment system.

[0010] One aspect of the present invention is a fatigue assessment system having an accumulation unit, a generation unit, a memory unit, an acquisition unit, and a measurement unit. The accumulation unit has a function of accumulating a plurality of first images and a plurality of second images. The plurality of first images are images of the eye and its surroundings taken from the side or an oblique direction. The plurality of second images are images of the eye and its surroundings taken from the front. The generation unit has a function of performing supervised learning and generating a trained model. The memory unit has a function of storing the trained model. The acquisition unit has a function of acquiring a third image. The third image is an image of the eye and its surroundings taken from the side or an oblique direction. The measurement unit has a function of measuring the fatigue level from the third image based on the trained model.

[0011] In the fatigue level assessment system, it is preferable that at least one of pupils and blinks is provided as training data for the supervised learning.

[0012] In the fatigue evaluation system, it is preferable that one of the plurality of first images and one of the plurality of second images are acquired at the same time.

[0013] In the fatigue evaluation system, the side or oblique direction is preferably an angle of 60° to 85° with respect to the horizontal direction relative to the line of sight.

[0014] The fatigue evaluation system preferably includes an output unit, and the output unit preferably has a function of providing information.

[0015] Another aspect of the present invention is a fatigue assessment device that includes, among the above-mentioned fatigue assessment systems, glasses that include a memory unit, an acquisition unit, and a measurement unit, and a server that includes an accumulation unit and a generation unit. [Effects of the Invention]

[0016] According to one embodiment of the present invention, the degree of fatigue can be evaluated. Furthermore, according to one embodiment of the present invention, the degree of fatigue can be evaluated while suppressing a decrease in labor productivity.

[0017] Note that the effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. Note that the other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from the description in the specification, drawings, etc., and can be extracted as appropriate from these descriptions. Note that one embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a fatigue level evaluation system. [Figure 2] FIG. 2 is a flow chart showing an example of a method for evaluating the fatigue level. [Figure 3] 3A to 3C are diagrams illustrating a method for photographing the eye and its surroundings. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a CNN. [Figure 5] 5A and 5B are diagrams illustrating a method for photographing the eye and its surroundings. [Figure 6] 6A and 6B are schematic diagrams of a human visual field. [Figure 7] 7A and 7B are schematic diagrams showing changes in pupil diameter over time. [Figure 8] 8A and 8B are diagrams illustrating an instrument and an electronic device in which the fatigue evaluation system is incorporated. [Figure 9] Fig. 9A is a diagram illustrating an appliance in which a part of the fatigue evaluation system is incorporated, and Fig. 9B is a diagram illustrating an electronic device in which a part of the fatigue evaluation system is incorporated. DETAILED DESCRIPTION OF THE INVENTION

[0019] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be readily understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0020] In the configuration of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted. Furthermore, when referring to similar functions, the same hatch pattern may be used and no particular reference numeral may be assigned.

[0021] Furthermore, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent the actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0022] It should also be noted that the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and are not intended to limit the number.

[0023] (Embodiment 1) In this embodiment, a fatigue evaluation system and a method for evaluating a fatigue level, which are one embodiment of the present invention, will be described with reference to FIGS. 1 to 7B. FIG.

[0024] <Configuration example of fatigue evaluation system> First, an example of the configuration of a fatigue level evaluation system will be described with reference to FIG.

[0025] 1 is a diagram showing an example of the configuration of a fatigue level assessment system 100. The fatigue level assessment system 100 includes an accumulation unit 101, a generation unit 102, an acquisition unit 103, a storage unit 104, a measurement unit 105, and an output unit 106.

[0026] The accumulation unit 101, the generation unit 102, the acquisition unit 103, the storage unit 104, the measurement unit 105, and the output unit 106 are connected to each other via a transmission path. The transmission path includes a network such as a local area network (LAN) or the Internet. The network can use either or both of wired and wireless communication.

[0027] Furthermore, when wireless communication is used in the above network, in addition to short-range communication means such as Wi-Fi (registered trademark) and Bluetooth (registered trademark), various communication means can be used, such as communication means compliant with the third generation mobile communication system (3G), communication means compliant with LTE (sometimes called 3.9G), communication means compliant with the fourth generation mobile communication system (4G), or communication means compliant with the fifth generation mobile communication system (5G).

[0028] The storage unit 101 stores learning data.

[0029] The generation unit 102 has a function of performing machine learning.

[0030] The acquisition unit 103 has a function of acquiring information. Here, the information acquired by the acquisition unit 103 is information about the eyes and their surroundings. The acquisition unit 103 is, for example, one or more selected from a camera, a pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, etc.

[0031] The storage unit 104 stores the information acquired by the acquisition unit 103. It also stores trained models.

[0032] Note that there are cases where the storage unit 104 does not need to be provided, such as when the trained model and the information acquired by the acquisition unit 103 are stored in the accumulation unit 101.

[0033] The measurement unit 105 has a function of measuring the level of fatigue. The function of measuring the level of fatigue includes a function of calculating the level of fatigue and a function of determining whether the level of fatigue is abnormal.

[0034] The output unit 106 has a function of providing information. The information includes the fatigue level calculated by the measurement unit 105, the determination result of whether the fatigue level is abnormal, etc. Components included in the output unit 106 include a display, a speaker, etc.

[0035] The above is a description of an example of the configuration of the fatigue level assessment system 100.

[0036] <Method for assessing fatigue level> Next, an example of a method for evaluating the fatigue level will be described with reference to FIGS. 2 to 7B.

[0037] As mentioned above, chronic fatigue is said to lead to disturbances in the autonomic nervous system. The autonomic nervous system consists of the sympathetic nervous system, which is active during physical activity, during the day, and when under stress, and the parasympathetic nervous system, which is active at rest, at night, and when relaxed. When the sympathetic nervous system is dominant, pupils dilate, heart rate increases, and blood pressure rises. On the other hand, when the parasympathetic nervous system is dominant, pupils constrict, heart rate decreases, blood pressure drops, and drowsiness occurs.

[0038] When the balance of the autonomic nervous system is disrupted, it can cause hypothermia, decreased blinking and tear production, etc. In addition, maintaining a hunched or arched posture for a long period of time can also lead to disruption of the autonomic nervous system.

[0039] From the above, if the disturbance or balance of the autonomic nervous system can be evaluated, the degree of fatigue can be objectively evaluated. In other words, the degree of fatigue can be objectively evaluated by evaluating changes over time in pupils (pupil diameter or pupil area), heart rate or pulse, blood pressure, body temperature, blinking, posture, etc.

[0040] Fig. 2 is a flow diagram showing an example of a method for evaluating a fatigue level. The method for evaluating a fatigue level includes steps S001 to S006 shown in Fig. 2. Steps S001 and S002 are steps for generating a trained model, and steps S003 to S006 are steps for measuring a fatigue level. In other words, the method for evaluating a fatigue level includes a method for generating a trained model and a method for measuring a fatigue level.

[0041] [How to generate a trained model] First, an example of a method for generating a trained model will be described. The method for generating a trained model includes steps S001 and S002 shown in FIG. 2.

[0042] In step S001, training data to be used for generating a trained model is prepared. For example, information about the eyes and their surroundings is acquired as the training data. In other words, step S001 can be rephrased as a process of acquiring information about the eyes and their surroundings. As will be described later, it is preferable that the information about the eyes and their surroundings is acquired from, for example, the side and front views.

[0043] The information on the eyes and their surroundings is acquired using one or more selected from a camera, a pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, etc. The information on the eyes and their surroundings may be obtained using a publicly available data set.

[0044] The training data may include teacher data (also called teacher signals or correct labels) such as pupils (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, and eye congestion. In particular, pupils (pupil diameter or pupil area) and blinking tend to change over time due to mental fatigue, so they are preferable as teacher data.

[0045] The information about the eyes and their surroundings prepared as learning data is stored in storage unit 101. After the learning data is stored in storage unit 101, the process proceeds to step S002.

[0046] In step S002, machine learning is performed based on the learning data stored in the storage unit 101. The machine learning is performed by the generation unit .

[0047] For the machine learning, it is preferable to use, for example, supervised learning, and it is more preferable to use supervised learning that uses a neural network (particularly, deep learning).

[0048] As deep learning, it is preferable to use, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder (AE), a variational autoencoder (VAE), or the like.

[0049] A trained model is generated by the machine learning, and the trained model is stored in the storage unit 104.

[0050] Note that the pupils (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, bloodshot eyes, etc. provided as training data vary depending on the individual, such as age, body type, and gender. Therefore, the trained model may be updated to suit the user.

[0051] The above is an example of a method for generating a trained model.

[0052] [Method for measuring fatigue level] Next, an example of a method for measuring fatigue level will be described. The method for measuring fatigue level includes steps S003 to S006 shown in Fig. 2. Note that the method for measuring fatigue level includes a method for calculating fatigue level and a method for determining whether fatigue level is abnormal.

[0053] In step S003, information on the eyes and their surroundings is acquired for use in calculating the degree of fatigue.

[0054] It is preferable that information about the eyes and their surroundings used to calculate the fatigue level be obtained, for example, from a side or oblique direction. By obtaining information about the eyes and their surroundings from a side or oblique direction, the information can be obtained from a position that is difficult for the user to visually recognize. Therefore, the information can be obtained without the user being aware of it.

[0055] The information about the eyes and their surroundings used to calculate the degree of fatigue is obtained using one or more sensors selected from a camera, a pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, and the like.

[0056] Furthermore, information on the eyes and their surroundings, which is used to calculate the degree of fatigue, is acquired in chronological order.

[0057] Information about the eyes and their surroundings to be used in calculating the degree of fatigue is stored in the storage unit 104. After the information is stored in the storage unit 104, the process proceeds to step S004.

[0058] In step S004, the fatigue level is calculated. The fatigue level is calculated using the trained model generated in step S002 and the information about the eyes and their surroundings acquired in step S003. The fatigue level is calculated by the measurement unit 105.

[0059] Calculating the level of fatigue refers to quantifying an index for assessing the level of fatigue, such as pupil (pupil diameter or pupil area), pulse rate, blood pressure, body temperature, blinking, posture, or eye congestion.

[0060] Note that the calculation of the fatigue level is not limited to quantifying the index for evaluating the fatigue level. For example, the fatigue level may be quantified from the information on the eyes and their surroundings acquired in step S003 using a trained model.

[0061] Before proceeding to step S005, steps S003 and S004 are repeatedly performed for a certain period of time, thereby obtaining time-series data for determining whether an abnormality has occurred in the index for evaluating the fatigue level.

[0062] In step S005, it is determined whether an abnormality has occurred in the index for evaluating the fatigue level.

[0063] If it is determined that an abnormality has occurred in the index for assessing the fatigue level, the fatigue level is determined to be high. If it is determined that the fatigue level is high, proceed to step S006. On the other hand, if it is determined that no abnormality has occurred in the index for assessing the fatigue level, the fatigue level is determined to be not high. If it is determined that the fatigue level is not high, proceed to step S003.

[0064] When the fatigue level is quantified in step S004, it is determined whether or not there is an abnormality in the fatigue level numerical value. If it is determined that there is an abnormality in the fatigue level numerical value, the fatigue level is determined to be high. If it is determined that the fatigue level is high, the process proceeds to step S006. On the other hand, if it is determined that there is no abnormality in the fatigue level numerical value, the fatigue level is determined to be not high. If it is determined that the fatigue level is not high, the process proceeds to step S003.

[0065] In step S006, information is output. The information includes an index for evaluating the fatigue level calculated by the measurement unit 105, a quantified fatigue level, a determination result as to whether the fatigue level is abnormal, etc. The information is output as visual information such as character strings, numerical values, graphs, and colors, or auditory information such as voice and music, for example.

[0066] After outputting the above information, the process ends.

[0067] The above is an explanation of an example of a method for calculating the fatigue level.

[0068] This concludes the description of an example of a method for assessing fatigue level.

[0069] <<Examples of how to assess fatigue>> In this section, a specific example of a method for evaluating fatigue level will be described with reference to Figures 3A to 7B. Here, a change in pupil (pupil diameter or pupil area) over time is selected as an index for evaluating fatigue level.

[0070] The training data prepared in step S001 is, for example, image data of the eyes and their surroundings. In this case, the image data of the eyes and their surroundings preferably includes image data of the eyes and their surroundings acquired from the front and image data of the eyes and their surroundings acquired from the side or an oblique direction. Compared to image data acquired from the side or an oblique direction, image data acquired from the front allows for detection of the pupil (pupil diameter or pupil area) with higher accuracy. Therefore, by using image data acquired from the front and image data acquired from the side or an oblique direction as training data, a trained model with higher accuracy can be generated than when only image data acquired from the side or an oblique direction is used as training data.

[0071] It is preferable to obtain images of the eyes and their surroundings for use as learning data by photographing them from the front, side, or oblique direction using a camera, for example.

[0072] 3A to 3C show examples of capturing images from the front, side, or oblique direction using cameras 111a to 111d. FIG. 3A is a view of the subject of the image capture from above. FIG. 3B is a view of the subject of the image capture from the right side. FIG. 3C is a view of the subject of the image capture from the front. For clarity of the figures, cameras 111a, 111b, and 111d are omitted from FIG. 3B, and cameras 111c and 111d are omitted from FIG. 3C. The subject of the image capture does not have to be limited to the person (user) whose fatigue level is to be evaluated.

[0073] 3A to 3C, the eyes and their surroundings are photographed from the front using cameras 111c and 111d, while the eyes and their surroundings are photographed from the side or obliquely using cameras 111a and 111b.

[0074] Before machine learning, image data for use as training data may be processed or corrected. Examples of image data processing or correction include cutting out areas unnecessary for machine learning, converting to grayscale, using a median filter, or using a Gaussian filter. Processing or correcting image data can reduce noise generated during machine learning.

[0075] It is preferable to prepare multiple combinations of image data of the eye and its surroundings acquired from the front and image data of the eye and its surroundings acquired from the side or oblique direction, all acquired at the same time, as the training data. Capturing images of the eye and its surroundings from multiple angles at the same time makes the above processing or correction easier. This allows for the generation of a highly accurate trained model. For example, processing or correction is performed on the image data of the eye and its surroundings acquired from the side or oblique direction, taking into account the image data of the eye and its surroundings acquired from the front. This enhances the pupil outline in the image data of the eye and its surroundings acquired from the side or oblique direction, allowing the pupil (pupil diameter or pupil area) to be detected with high accuracy.

[0076] If many images can be acquired from the side or oblique direction, only the image data acquired from the side or oblique direction may be used as training data. Also, if image data of the eye and its surroundings acquired from the side or oblique direction is processed or corrected in consideration of image data of the eye and its surroundings acquired from the front, only the image data acquired from the side or oblique direction may be used as training data.

[0077] As described above, since the training data is image data, it is preferable to use a convolutional neural network for the machine learning performed in step S002.

[0078] [Convolutional Neural Networks] Here, we will explain convolutional neural networks (CNNs).

[0079] FIG. 4 shows an example of the configuration of a CNN. The CNN is composed of a convolutional layer CL, a pooling layer PL, and a fully connected layer FCL. Image data IPD is input to the CNN, and feature extraction is performed. In this embodiment, the image data IPD is image data of the eye and its surroundings.

[0080] The convolution layer CL has the function of performing convolution on image data. Convolution is performed by repeatedly performing product-sum operations between a portion of the image data and the filter value of a weight filter (also called a kernel). Image features are extracted by the convolution in the convolution layer CL.

[0081] The above-mentioned product-sum operation may be performed on software using a program, or may be performed by hardware. When the product-sum operation is performed by hardware, a product-sum operation circuit may be used. This product-sum operation circuit may be a digital circuit or an analog circuit.

[0082] The product-sum circuit may be configured using transistors having Si in a channel formation region (also referred to as Si transistors) or transistors using metal oxide in a channel formation region (also referred to as OS transistors). In particular, OS transistors have extremely low off-state current and are therefore suitable as transistors constituting the analog memory of the product-sum circuit. Note that the product-sum circuit may be configured using both Si transistors and OS transistors.

[0083] One or more weight filters can be used for convolution. When multiple weight filters are used, it becomes possible to extract multiple features contained in the image data. Figure 4 shows three weight filters (filter F a , F b , F c ) is used. The image data input to the convolution layer CL is filtered using the filter F a , F b , F c The image data D a , D b , D c The image data D a , D b , D c is also called a feature map.

[0084] Image data D generated by convolution a , D b , D c is converted by an activation function and then output to the pooling layer PL. As the activation function, ReLU (Rectified Linear Units) or the like can be used. ReLU is a function that outputs "0" when the input value is negative and outputs the input value as is when the input value is "0" or greater. In addition, as the activation function, a sigmoid function, a tanh function, or the like can also be used.

[0085] The pooling layer PL has the function of performing pooling on the image data input from the convolutional layer CL. Pooling is a process of dividing the image data into multiple regions, extracting predetermined data for each region, and arranging it in a matrix. Pooling reduces the spatial size of the image data while retaining the features extracted by the convolutional layer CL. It also improves the position invariance or translation invariance of the features extracted by the convolutional layer CL. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as pooling.

[0086] CNN extracts features by the above-mentioned convolutional processing and pooling processing. CNN can be configured with multiple convolutional layers CL and multiple pooling layers PL. In FIG. 4, z layers (z is an integer equal to or greater than 1) of layers L, each consisting of a convolutional layer CL and a pooling layer PL, are provided (layers L1 to L2). z ), the convolution process and pooling process are performed z times. In this case, feature extraction can be performed in each layer L, enabling more advanced feature extraction.

[0087] The fully connected layer FCL has the function of determining the image using the image data that has been convolved and pooled. All nodes of the fully connected layer FCL are connected to the previous layer of the fully connected layer FCL (here, the pooling layer PL, in Figure 4, the layer L z The image data output from the convolution layer CL or the pooling layer PL is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. The one-dimensional expanded data OPD is then output.

[0088] The configuration of the CNN is not limited to that shown in Figure 4. For example, a pooling layer PL may be provided for each of multiple convolutional layers CL. Furthermore, if it is desired to retain as much position information of extracted features as possible, the pooling layer PL may be omitted.

[0089] Furthermore, when classifying images from the output data of the fully connected layer FCL, an output layer electrically connected to the fully connected layer FCL may be provided. The output layer can use a softmax function or the like as a likelihood function to output the probability of classification into each class. The classes to be classified may be, for example, the degree of fatigue. Specifically, the classes to be classified may be "very high fatigue," "high fatigue," "moderate fatigue," "low fatigue," "very low fatigue," etc. This makes it possible to quantify the fatigue level from the image data.

[0090] Furthermore, when performing regression analysis such as numerical prediction from the output data of the fully connected layer FCL, an output layer electrically connected to the fully connected layer FCL may be provided. By using an identity function or the like in the output layer, a predicted value can be output. This makes it possible to calculate, for example, pupil diameter or pupil area from image data.

[0091] CNN can also perform supervised learning, using image data as training data accompanied by training data. For example, backpropagation can be used for supervised learning. CNN learning can optimize the filter values ​​of weight filters, weight coefficients of fully connected layers, etc.

[0092] This concludes the explanation of convolutional neural networks (CNNs).

[0093] In the above-described supervised learning, image data of the eye and its surroundings acquired from the front and image data of the eye and its surroundings acquired from the side or oblique direction are prepared as training data, and the system is trained to output pupil diameter or pupil area. For example, when pupil diameter or pupil area is provided as training data, the pupil diameter or pupil area is output by regression using CNN. In this way, a trained model is generated that outputs pupil diameter or pupil area from image data of the eye and its surroundings acquired from the side or oblique direction.

[0094] Alternatively, a quantified fatigue level may be output by classifying the eyes using CNN. In this case, a trained model that outputs a quantified fatigue level is generated from image data of the eyes and their surroundings acquired from the side or oblique direction.

[0095] In step S003, information about the eyes and their surroundings is acquired for use in calculating the fatigue level. For example, an image of the eyes and their surroundings acquired from the side or an oblique direction is acquired as the information about the eyes and their surroundings. The image of the eyes and their surroundings may be acquired by photographing them from the side using a camera or the like.

[0096] 5A and 5B show examples of capturing images from the side or oblique direction using cameras 112a and 112b. FIG. 5A is a view of the subject of the image capturing from above. FIG. 5B is a view of the subject of the image capturing from the front. The subject of the image capturing is a person (user) whose fatigue level is to be evaluated.

[0097] As shown in FIGS. 5A and 5B, the eye and its surroundings are photographed from the side or oblique direction using camera 112a and camera 112b.

[0098] It is preferable that the distance to the subject of camera 111 (one or more of cameras 111a to 111d) shown in FIG. 3A and the distance to the subject of camera 112 (camera 112a and / or camera 112b) shown in FIG. 5A are approximately equal. This allows the fatigue level to be evaluated with high accuracy. In the method for evaluating the fatigue level according to one aspect of the present invention, supervised learning is performed, so the distance to the subject of camera 111 and camera 112 do not necessarily have to be equal.

[0099] Furthermore, it is preferable that the image captured by the camera 111 and the image captured by the camera 112 have the same resolution, aspect ratio, and the like. This allows the fatigue level to be evaluated with high accuracy. Note that in the method for evaluating the fatigue level according to one embodiment of the present invention, supervised learning is performed, and therefore the image captured by the camera 111 and the image captured by the camera 112 do not necessarily have to have the same resolution, aspect ratio, and the like.

[0100] 6A and 6B are schematic diagrams of a person's visual field (binocular vision). Fig. 6A is a diagram of a person viewed from above, and Fig. 6B is a diagram of a person viewed from the right side.

[0101] The human visual field is classified into the effective visual field, the induced visual field, the auxiliary visual field, etc. In Fig. 6A and Fig. 6B, the dashed line from the person to the fixation point C is the line of sight (visual axis), and the angle θ 1h and angle θ 1vis the range of the field of view angle of the effective field of view, and the angle θ 2h and angle θ 2v is the range of the visual angle of the induced field of view, and the angle θ 3h and angle θ 3v is the range of the visual angle of the auxiliary visual field. Unless otherwise specified, the line of sight refers to the line from the person to the point of gaze C when the point of gaze is located at a position where the length of the line segment connecting the point of gaze C and the right eye is equal to the length of the line segment connecting the point of gaze C and the left eye. Furthermore, the horizontal direction refers to the direction horizontal to the plane including both eyes and the line of sight. Furthermore, the vertical direction refers to the direction perpendicular to the plane including both eyes and the line of sight.

[0102] The effective visual field is the area in which information can be received instantly. The horizontal visual angle of the effective visual field (angle θ 1h ) is said to be within a range of approximately 30° from the center of the line of sight, and the vertical visual field angle of the effective visual field (angle θ 1v ) is said to be in the range of about 20 degrees, centered slightly below the line of sight.

[0103] The induced visual field is a region that affects the spatial coordinate system. The horizontal visual angle of the induced visual field (angle θ 2h ) is said to be within a range of approximately 100° from the center of the line of sight, and the vertical visual angle of the induced visual field (angle θ 2v ) is said to be in the range of about 85°, centered slightly below the line of sight.

[0104] The auxiliary visual field is an area where the presence of the stimulus can be recognized. The horizontal visual angle of the auxiliary visual field (angle θ 3h ) is said to be in the range of about 200° around the line of sight, and the vertical visual angle of the auxiliary visual field (angle θ 3v ) is said to be in the range of about 125°, centered slightly below the line of sight.

[0105] When working, most information is received from the effective visual field, with a small amount also received from the induced visual field. In addition, there is almost no information from the auxiliary visual field when working. In other words, it is difficult for workers to recognize information located in the auxiliary visual field.

[0106] Furthermore, if the change in pupil size over time is selected as an index for assessing fatigue level, the image acquired in step S003 must include the pupil. Visual information is recognized when an image is projected onto the retina through the pupil, lens, etc., and transmitted to the brain via the optic nerve. In other words, because the auxiliary visual field also includes visual information, the pupil can be recognized within the auxiliary visual field.

[0107] From the above, the side or oblique direction in which the image of the eye and its surroundings is acquired is the direction in which the pupil is observed from within the auxiliary visual field or the induced visual field near the auxiliary visual field in the horizontal direction, and is the angle θ a The range is 45° to 100°, preferably 50° to 90°, and more preferably 60° to 85°, in the horizontal direction relative to the line of sight. This allows the image to be acquired from a position that is difficult for the user to visually recognize. Therefore, the image can be acquired without the user being aware of it.

[0108] Note that if the side or oblique direction is within the above range in the horizontal direction, any angle in the vertical direction will be outside the field of view of the induced field of view. Therefore, the side or oblique direction in the vertical direction may be any direction within the range in which the pupil can be photographed.

[0109] The "front" from which images of the eye and its surroundings are acquired refers to the horizontal direction from which the pupil is observed from within the induced field of view. Specifically, the "front" refers to an angle from 0° to 50°, preferably 0° to 30°, and more preferably 0° to 15°, horizontally relative to the line of sight. This allows for the capture of a pupil with a circular or nearly circular shape, enabling the pupil diameter or pupil area to be calculated with high accuracy.

[0110] If changes in pupil size over time are selected as an index for assessing fatigue level, in step S004, the trained model is used to calculate pupil diameter or pupil area from image data of the eye and its surrounding area acquired from the side or oblique direction.

[0111] As mentioned above, when the sympathetic nervous system is dominant, the pupil dilates, and when the parasympathetic nervous system is dominant, the pupil constricts. In other words, pupil diameter changes as the autonomic nervous system becomes unstable. It is also said that the rate at which pupil size changes slows as fatigue accumulates. In this specification, the change in the pupil (pupil diameter or pupil area) over time refers to the change in the pupil (pupil diameter or pupil area) over time, the rate at which the pupil (pupil diameter or pupil area) changes, the change in the expansion and contraction cycle of the pupil (pupil diameter or pupil area) over time, etc.

[0112] Whether or not an abnormality has occurred in the change over time of the pupil (pupil diameter or pupil area) is determined based on the pupil (pupil diameter or pupil area) immediately after the start of step S003.

[0113] An example of a method for determining whether or not an abnormality has occurred in the change over time of the pupil (pupil diameter or pupil area) will be described with reference to FIGS. 7A and 7B.

[0114] 7A and 7B are schematic diagrams showing changes in pupil diameter over time. In FIGS. 7A and 7B, the horizontal axis represents time, and the vertical axis represents pupil diameter. The solid lines in FIGS. 7A and 7B show changes in pupil diameter over time. The dashed-dotted lines in FIGS. 7A and 7B show the average pupil diameter over time.

[0115] FIG. 7A is a diagram showing a typical decrease in pupil diameter over time. In order to determine whether the change in pupil diameter over time is abnormal, a threshold value for the pupil diameter is set in advance. For example, as shown by the dashed line in FIG. 7A, the upper limit of the pupil diameter is set to r max Let the lower limit of pupil diameter be r min In the example of FIG. 7A, the pupil diameter at time t is set to the lower limit r min At this time, it is determined that an abnormality has occurred in the change in the pupil over time.

[0116] For example, if the pupil (pupil diameter or pupil area) expands or contracts at a rate equal to or greater than a certain level, based on the pupil (pupil diameter or pupil area) immediately after the start of step S003, it is determined that an abnormality has occurred.

[0117] FIG. 7B is a diagram showing the pupil diameter expansion / contraction cycle as time passes. u (u is a natural number). In order to determine whether the change in pupil diameter over time is abnormal, a threshold value for the expansion / contraction cycle of the pupil diameter is set in advance. For example, as shown in FIG. 7B, the upper limit of the expansion / contraction cycle of the pupil diameter is set to f max The lower limit of the pupil diameter expansion and contraction cycle is f min In the example of FIG. 7B, the pupil diameter expansion / contraction cycle f t+7 is the upper limit of the pupil diameter expansion and contraction cycle f max In this case, it is determined that an abnormality has occurred in the change in the pupil over time.

[0118] The expansion and contraction of the pupil diameter and the expansion and contraction period of the pupil diameter are observed together. For example, a fast Fourier transform may be performed on the change in pupil diameter over time. This makes it easier to determine whether there is an abnormality based on the expansion and contraction period of the pupil diameter.

[0119] Although the above example illustrates a method for calculating pupil diameter or pupil area from image data of the eye and its surroundings acquired from the side or oblique direction using a trained model, the present invention is not limited to this. For example, the trained model may be used to digitize the fatigue level from image data of the eye and its surroundings acquired from the side or oblique direction. In this case, a fatigue level threshold (upper limit of fatigue level) is set in advance to determine whether the digitized fatigue level is abnormal.

[0120] The above is a description of the fatigue assessment system. By using the fatigue assessment system according to one aspect of the present invention, the system (particularly the acquisition unit) is not positioned in the user's line of sight, which prevents the user from experiencing increased mental fatigue. Therefore, the fatigue level during use can be assessed with high accuracy.

[0121] The structures, methods, and the like described in this embodiment can be used in appropriate combination with structures, methods, and the like described in other embodiments.

[0122] (Embodiment 2) In this embodiment, a fatigue evaluation device will be described with reference to Figures 8A to 9B. The fatigue evaluation device is an electronic device, or a tool and an electronic device, equipped with the fatigue evaluation system described in the previous embodiment.

[0123] Examples of devices that include part of the fatigue evaluation system include eyeglasses such as vision correction glasses and safety glasses, and safety equipment worn on the head such as helmets and gas masks.

[0124] The device includes at least the acquisition unit 103 of the fatigue level evaluation system described in the previous embodiment. The device also includes a battery.

[0125] Examples of electronic devices that include a part of the fatigue evaluation system include information terminals, computers, etc. Here, the term "computer" refers to a tablet computer, a notebook computer, a desktop computer, as well as large computers such as a workstation and a server system.

[0126] The device may be equipped with a GPS (Global Positioning System) receiver, so that the electronic device can use the GPS to acquire data on the device's position, travel distance, acceleration, etc. By combining the acquired data with an index for assessing fatigue level, the fatigue level can be assessed with higher accuracy.

[0127] An example of an instrument and electronic device equipped with the fatigue level evaluation system is shown in Fig. 8A. Fig. 8A shows glasses 200 and a server 300 equipped with the fatigue level evaluation system. Glasses 200 have a processing unit 201. Server 300 also has a processing unit 301.

[0128] For example, processing unit 201 includes acquisition unit 103, as described in the previous embodiment, and processing unit 301 includes accumulation unit 101, generation unit 102, memory unit 104, and measurement unit 105, as described in the previous embodiment. Processing unit 201 and processing unit 301 each include a transmission / reception unit. Since processing unit 201 includes only acquisition unit 103, the weight of eyeglasses 200 including processing unit 201 can be reduced. Therefore, the physical burden on the user when wearing eyeglasses 200 can be reduced.

[0129] Furthermore, when a camera is used as the acquisition unit 103, by placing the acquisition unit 103 near the eyes in the frame of the eyeglasses 200, it is possible to take close-up images of the eyes and their surroundings. This makes it easier to detect the eyes. It is also possible to reduce the amount of external scenery reflected in the eyes. This reduces the number of times that processing or correction is required for images of the eyes and their surroundings. Alternatively, processing or correction may become unnecessary.

[0130] 8A shows an example in which a camera is used as the acquisition unit 103, but the invention is not limited to this. A pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, or the like may also be used as the acquisition unit 103. In this case, the acquisition unit 103 may be installed in a direction other than the side of the eye or diagonally. For example, the acquisition unit 103 may be installed at or near the point where the head and the frame of the eyeglasses 200 come into contact.

[0131] The processing unit 201 may include the output unit 106 described in the previous embodiment. The processing unit 201 includes the output unit 106, which allows the user to know the degree of fatigue while working. Components included in the output unit 106 include a display, a speaker, and the like.

[0132] It is preferable that the information provided by the output unit 106 be output as visual information such as color, or auditory information such as voice or music. Compared to visual information such as character strings, numerical values, and graphs, visual information such as color has less impact on the sense of sight and causes less stress to the user, making it preferable. The same applies to auditory information such as voice or music. It may be possible to reduce the user's level of fatigue by registering their favorite music or other auditory information in advance.

[0133] Note that the configurations of the processing unit 201 and the processing unit 301 are not limited to this. For example, the processing unit 201 may include the acquisition unit 103, the storage unit 104, the measurement unit 105, and the output unit 106, and the processing unit 301 may include the accumulation unit 101 and the generation unit 102. In this case, the processing unit 201 has a function of measuring the fatigue level, and the processing unit 301 has a function of generating a trained model.

[0134] With the above configuration, the fatigue level can be measured only by the processing unit 201, thereby minimizing the frequency of communication between the processing unit 201 and the processing unit 301. Furthermore, with the above configuration, the processing unit 301 can transmit the trained model updated by the processing unit 301 to the processing unit 201, and the processing unit 201 can receive the trained model. Then, the trained model recorded in the processing unit 201 can be updated to the received trained model. This makes it possible to use trained data with improved accuracy, and to evaluate the fatigue level with higher accuracy.

[0135] The storage unit 104 may store information about the eye and its surroundings acquired by the acquisition unit 103. After a certain amount of the acquired information about the eye and its surroundings has been accumulated in the storage unit 104, the accumulated information may be transmitted to an electronic device having the processing unit 301. This makes it possible to reduce the number of communications between the processing unit 201 and the processing unit 301.

[0136] Note that the electronic device having part of the fatigue evaluation system may be configured with multiple devices. Glasses, a server, and a mobile phone (smartphone), which is a type of information terminal, each equipped with the fatigue evaluation system are shown in FIG. 8B. Similar to glasses 200 and server 300 shown in FIG. 8A, glasses 200 have processing unit 201, and server 300 has processing unit 301. Information terminal 310 also has processing unit 311.

[0137] For example, the processing unit 201 includes an acquisition unit 103. The processing unit 301 includes an accumulation unit 101 and a generation unit 102. The processing unit 311 includes a storage unit 104, a measurement unit 105, and an output unit 106. Furthermore, each of the processing unit 201, the processing unit 301, and the processing unit 311 includes a transmission / reception unit.

[0138] In the above configuration, when the user of the glasses 200 carries the information terminal 310, the user can check his / her own fatigue level via the information terminal 310.

[0139] Furthermore, if the information terminal 310 is carried by the boss of the user of the glasses 200, the boss can check the user's fatigue level via the information terminal 310. Therefore, even if the user and the boss are not close to each other, the boss can manage the user's health condition. Furthermore, if the information output from the output unit 106 is visual information such as character strings, numerical values, and graphs related to the fatigue level, the boss can know the user's health condition in detail.

[0140] The glasses 200 shown in Figures 8A and 8B are not limited to vision correction glasses, but may also be sunglasses, color vision correction glasses, 3D glasses, augmented reality (AR) glasses, mixed reality (MR) glasses, fashion glasses, personal computer glasses with blue light blocking function, etc.

[0141] In particular, in AR glasses, MR glasses, and the like, information about the degree of fatigue can be output as visual information such as character strings, numerical values, and graphs, allowing the degree of fatigue to be known in detail.

[0142] 9A is a diagram showing protective glasses equipped with part of the fatigue level evaluation system. Protective glasses 210 shown in FIG. 9A have a processing unit 211. The processing unit 211 also includes an acquisition unit 103.

[0143] The processing unit 211 may have the same functions as the processing unit 201 of the glasses 200 shown in FIGS. 8A and 8B.

[0144] 9A illustrates a goggle type as protective glasses 210, but is not limited to this and may be a spectacle type or a front type. Also, while FIG. 9A illustrates a single-lens type as protective glasses 210, is not limited to this and may be a twin-lens type.

[0145] 8A, 8B, and 9A show eyeglasses such as corrective glasses and safety glasses as examples of devices including a part of the fatigue evaluation system, but the present invention is not limited to these. For example, other types of safety protection equipment worn on the head, such as helmets and gas masks, can also be used.

[0146] Up to this point, the fatigue assessment device has been described as a configuration that combines an instrument having part of a fatigue assessment system with an electronic device having part of the fatigue assessment system, but the present invention is not limited to this. For example, the fatigue assessment device may be a configuration that combines a detachable electronic device having part of the fatigue assessment system with the electronic device described above. FIG. 9B shows a head-worn safety protector 220 equipped with a detachable electronic device 320 having part of the fatigue assessment system. The detachable electronic device 320 has an acquisition unit 103. By incorporating part of the fatigue assessment system in the detachable electronic device 320, it is possible to utilize safety protectors that have been used in the past.

[0147] Furthermore, a part of the fatigue assessment system may be provided in a display device worn on the head, such as a head-mounted display, smart glasses, etc. This allows fatigue assessment to be performed in situations using virtual reality (VR), for example.

[0148] The fatigue assessment device may be a single instrument or a single electronic device equipped with a fatigue assessment system.

[0149] By using the fatigue assessment device according to one aspect of the present invention, the user's field of vision is ensured and the mental burden on the user is reduced. Therefore, the user's fatigue level can be assessed with high accuracy. Furthermore, if the user is a worker, there is no need to interrupt work in order to assess the user's fatigue level, which can prevent a decline in labor productivity.

[0150] Furthermore, by using the fatigue evaluation device according to one embodiment of the present invention, information on the eyes and their surroundings can be obtained from a position close to the eyes, thereby improving the accuracy of fatigue evaluation.

[0151] The structures, methods, and the like described in this embodiment can be used in appropriate combination with structures, methods, and the like described in other embodiments. [Explanation of symbols]

[0152] : 100: fatigue evaluation system, 101: accumulation unit, 102: generation unit, 103: acquisition unit, 104: memory unit, 105: measurement unit, 106: output unit, 111: camera, 111a: camera, 111b: camera, 111c: camera, 111d: camera, 112: camera, 112a: camera, 112b: camera, 200: glasses, 201: processing unit, 210: protective glasses, 211: processing unit, 220: safety protector, 300: server, 301: processing unit, 310: information terminal, 311: processing unit, 320: electronic device

Claims

[Claim 1] The apparatus includes an accumulation unit, a generation unit, a storage unit, an acquisition unit, and a measurement unit, a first step of preparing learning data by acquiring a plurality of sets of first image information of the eye and its surroundings photographed from the side or oblique direction and second image information of the eye and its surroundings photographed from the front, and storing the learning data in the storage unit; a second step of generating a trained model that outputs a quantified pupil diameter when image information of an eye and its surroundings photographed from the side or oblique direction is input by performing supervised learning based on the training data in the generation unit, and storing the trained model in the storage unit; a third step of acquiring third image information by capturing an image of the user's eyes and their surroundings from a side or oblique direction of the user in the acquisition unit, and storing the third image information in the storage unit; a fourth step of quantifying the pupil diameter included as information in the third image information by using the trained model in the measurement unit; a fifth step of repeatedly performing the third step and the fourth step to obtain time series data for determining whether or not an abnormality has occurred in the pupil diameter included as information in the third image information, and then determining whether or not an abnormality has occurred in the digitized pupil diameter obtained by performing the third step and the fourth step; By going through this, it has a function of evaluating the fatigue level of the user, In the fifth step, when the digitized pupil diameter falls below a threshold value, it is determined that an abnormality has occurred. Fatigue rating system.

Images