Fatigue evaluation system

Abstract

To provide a fatigue degree evaluation system.SOLUTION: A fatigue level evaluation system includes an accumulation unit, a generation unit, a memory unit, an acquisition unit, and a measurement unit, the accumulation unit is configured to accumulate a plurality of first images and a plurality of second images, the plurality of first images are images of an eye and the periphery of the eye captured from a side or an oblique direction, and the plurality of second images are images of the eye and the periphery of the eye captured from the front, the generation portion is configured to perform supervised learning and generate a trained model, the memory portion is configured to store the trained model, the acquisition portion is configured to acquire a third image, the third image is an image of an eye and the periphery of the eye acquired from a side or an oblique direction, and the measurement portion is configured to measure the degree of fatigue from the third image on the basis of the trained model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] One aspect of the present invention relates to a method for assessing fatigue level. The present invention also relates to a fatigue evaluation system. [Background technology]

[0002] In modern society, proper management of workers' health is important not only for workers' health but also for their productivity. This is an important issue as it will lead to improved productivity and accident prevention. Proper management is an important issue not only for workers, but also for students, housewives, and others.

[0003] Deterioration of health is caused by the accumulation of fatigue. Fatigue can be physical and mental. It can be divided into fatigue and nervous fatigue. Symptoms that appear due to the accumulation of physical fatigue are as follows: On the other hand, symptoms caused by the accumulation of mental and nervous fatigue are relatively easy to notice. Recently, visual display (VDT) has become a common sight. Terminal work is increasing, creating an environment where nervous fatigue can easily accumulate.

[0004] One of the causes of fatigue is psychological stress (also simply called stress). In addition, chronic fatigue is said to lead to autonomic nervous system disorders. Methods for measuring fatigue levels and stress levels using machine learning and other methods are attracting attention. Patent Document 1 discloses a method for detecting mental fatigue using flashing light. discloses an autonomic nervous function and stress level evaluation device equipped with a machine learning device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-301841 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-259609 Summary of the Invention [Problem to be solved by the invention]

[0006] The fatigue level was measured using the detection device and evaluation device disclosed in Patent Document 1 and Patent Document 2. When assessing the level of stress and stress, if the employer is a worker, he / she may need to suspend work-related activities. In addition, the user must be able to visually identify the detection device, which may result in a decrease in labor productivity. By recognizing this, in addition to the mental fatigue that has accumulated before using the detection device, Therefore, it is difficult to detect mental fatigue correctly. difficult.

[0007] In view of the above, an object of one embodiment of the present invention is to evaluate the degree of fatigue. An object of one embodiment is to evaluate a degree of fatigue while suppressing a decrease in labor productivity.

[0008] The description of these problems does not preclude the existence of other problems. It is not necessary for one embodiment to solve all of these problems. The subject matter will be self-evident from the description, drawings, claims, etc. It is possible to extract other issues from the drawings, claims, etc. [Means for solving the problem]

[0009] In view of the above-mentioned problems, one aspect of the present invention is to provide a method for detecting an eye condition by using information about the eyes and their surroundings as learning data. By performing learning, a pre-trained model is generated, making it difficult for users to visually recognize. A system that evaluates the degree of fatigue based on information about the eyes and their surroundings obtained from the position (Fatigue degree Furthermore, one aspect of the present invention provides a device equipped with the fatigue level evaluation system. Providing instruments and electronic equipment.

[0010] One aspect of the present invention is a fatigue measurement device having an accumulation unit, a generation unit, a storage unit, an acquisition unit, and a measurement unit. The storage unit stores a plurality of first images and a plurality of second images. The first images are images of the eye and other parts taken from the side or oblique direction. The second images are images of the eye and its surroundings taken from the front. The generation unit has a function of performing supervised learning and generating a trained model. The acquisition unit has a function of acquiring a third image. The third image is an image of the eye and its surroundings taken from the side or oblique direction. The measurement unit has the function of measuring the fatigue level from the third image based on the trained model. .

[0011] In the fatigue evaluation system, the supervised learning uses the pupil and Preferably, at least one of blinking and blinking is given.

[0012] In the fatigue evaluation system, one of the plurality of first images and one of the plurality of second images It is preferable that the first and second signals are acquired at the same time.

[0013] In the fatigue evaluation system, the side or oblique direction means a direction parallel to the line of sight. It is preferable that the angle is 60° or more and 85° or less in the horizontal direction.

[0014] In addition, it is preferable that the fatigue evaluation system has an output unit. Preferably, the power unit has a function of providing information.

[0015] Furthermore, one aspect of the present invention is a fatigue evaluation system including a memory unit, an acquisition unit, and a measurement unit. and a server including a storage unit and a generation unit. It is a value device. [Effects of the Invention]

[0016] According to one embodiment of the present invention, the degree of fatigue can be evaluated. This makes it possible to evaluate the degree of fatigue while suppressing a decline in labor productivity.

[0017] The effects of one embodiment of the present invention are not limited to the effects listed above. This does not preclude the existence of other effects. The effects not mentioned in this section are obvious to those skilled in the art. It can be derived from the descriptions in the specifications, drawings, etc., and can be extracted appropriately from these descriptions. It should be noted that one aspect of the present invention has at least one of the effects listed above and / or other effects. Therefore, one aspect of the present invention is, in some cases, There are cases where the effects listed above are not achieved. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a fatigue level evaluation system. [Figure 2] FIG. 2 is a flow chart showing an example of a method for evaluating the fatigue level. [Figure 3]3A to 3C are diagrams illustrating a method for photographing the eye and its surroundings. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a CNN. [Figure 5] 5A and 5B are diagrams illustrating a method for photographing the eye and its surroundings. [Figure 6] 6A and 6B are schematic diagrams of a human visual field. [Figure 7] 7A and 7B are schematic diagrams showing changes in pupil diameter over time. [Figure 8] 8A and 8B are diagrams illustrating an instrument and an electronic device in which the fatigue evaluation system is incorporated. [Figure 9] Fig. 9A is a diagram illustrating an appliance in which a part of the fatigue evaluation system is incorporated, and Fig. 9B is a diagram illustrating an electronic device in which a part of the fatigue evaluation system is incorporated. DETAILED DESCRIPTION OF THE INVENTION

[0019] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description. The present invention is not limited to the above, and various changes and modifications may be made in form and detail without departing from the spirit and scope of the present invention. It will be readily apparent to those skilled in the art that the present invention can be modified in various ways. The present invention is not to be construed as being limited to the description in the form of

[0020] In the configuration of the invention described below, the same parts or parts having similar functions The same reference numerals are used in common between different drawings, and repeated explanations will be omitted. When referring to similar functions, the hatch pattern may be the same and no particular reference numeral may be given.

[0021] In addition, the position, size, range, etc. of each component shown in the drawings are not necessarily the same as those in the actual embodiment for ease of understanding. Therefore, the disclosed invention may not necessarily represent the actual position, size, range, etc. The position, size, range, etc. are not necessarily limited to those disclosed in the drawings.

[0022] In addition, the ordinal numbers "first," "second," and "third" used in this specification refer to the number of components. It should be noted that this is added to avoid confusion and is not intended to limit the number.

[0023] (Embodiment 1) In this embodiment, a fatigue level evaluation system and a fatigue level evaluation device according to one aspect of the present invention are described. This method will be described with reference to FIGS. 1 to 7B.

[0024] <Configuration example of fatigue evaluation system> First, an example of the configuration of a fatigue level evaluation system will be described with reference to FIG.

[0025] FIG. 1 is a diagram showing an example of the configuration of a fatigue level assessment system 100. 0 includes an accumulation unit 101, a generation unit 102, an acquisition unit 103, a storage unit 104, and a measurement unit 10 5 and an output unit 106.

[0026] The accumulation unit 101, the generation unit 102, the acquisition unit 103, the storage unit 104, and the measurement unit 1 The output unit 105 and the output unit 106 are connected via a transmission line. The transmission route may include a local area network (LAN) or a network such as the Internet. The network may be either wired or wireless, or both. Communication by either method can be used.

[0027] In addition, when wireless communication is used in the above network, Wi-Fi (registered trademark), B In addition to short-range communication methods such as Bluetooth (registered trademark), third-generation mobile communication systems (3G) compliant communication means, LTE (sometimes called 3.9G) compliant communication means , communication means compliant with the fourth generation mobile communication system (4G) or the fifth generation mobile communication system Various communication methods can be used, including those compliant with 5G systems.

[0028] The storage unit 101 stores learning data.

[0029] The generation unit 102 has a function of performing machine learning.

[0030] The acquisition unit 103 has a function of acquiring information. The information acquired is information about the eyes and their surroundings. The acquisition unit 103 is, for example, a camera, a pressure sensor, and the like. The sensor may be one or more selected from a sensor, a distortion sensor, a temperature sensor, a gyro sensor, and the like.

[0031] The storage unit 104 stores the information acquired by the acquisition unit 103. The model is stored.

[0032] Note that the storage unit 104 may not be provided in some cases. For example, , and the information acquired by the acquisition unit 103 is stored in the accumulation unit 101.

[0033] The measurement unit 105 has a function of measuring the degree of fatigue. It includes a function to calculate the fatigue level and a function to determine whether the fatigue level is abnormal.

[0034] The output unit 106 has a function of providing information. The output unit 106 outputs the determined fatigue level and the result of determining whether the fatigue level is abnormal. Components include a display and speakers.

[0035] The above is a description of an example of the configuration of the fatigue level assessment system 100.

[0036] <Method for assessing fatigue level> Next, an example of a method for evaluating the fatigue level will be described with reference to FIGS. 2 to 7B.

[0037] As mentioned above, chronic fatigue is said to lead to autonomic nervous system disorders. The meridian contains the sympathetic nervous system, which is active during physical activity, during the day, and when you are tense, and the sympathetic nervous system, which is active during rest, at night, and when you are relaxed. When the sympathetic nervous system is dominant, the pupils dilate, the heart beats faster, and the parasympathetic nervous system becomes active. On the other hand, when the parasympathetic nervous system is dominant, pupils constrict, the heart rate increases, and Symptoms include a decrease in heart rate, a drop in blood pressure, and drowsiness.

[0038] When the balance of the autonomic nervous system is impaired, it can cause hypothermia, decreased blinking and tear production, etc. In addition, if you maintain a hunched or arched posture for a long period of time, it may lead to a disturbance in the autonomic nervous system. .

[0039] From the above, if we can evaluate the disorder or balance of the autonomic nervous system, we can objectively measure the degree of fatigue. That is, pupil (pupil diameter or pupil area), heartbeat or pulse By evaluating changes over time in blood pressure, body temperature, blinking, posture, etc., fatigue levels can be objectively assessed. It is possible.

[0040] FIG. 2 is a flow chart showing an example of a method for evaluating the degree of fatigue. , includes steps S001 to S006 shown in FIG. Step S002 is a process for generating a trained model, and step S003 Steps S001 to S006 are steps for measuring the degree of fatigue. The method includes a method for generating a trained model and a method for measuring fatigue level.

[0041] [How to generate a trained model] First, an example of a method for generating a trained model will be described. The generating method includes steps S001 and S002 shown in FIG.

[0042] In step S001, the training data used to generate the trained model is prepared. For example, information about the eyes and their surroundings is acquired as the learning data. Step S001 can be rephrased as a process of acquiring information about the eye and its surroundings. As will be described later, information about the eyes and their surroundings can be obtained from, for example, the side and the front. preferable.

[0043] Information on the eyes and their surroundings is collected from cameras, pressure sensors, strain sensors, temperature sensors, and The information is acquired using one or more sensors selected from the eye and surrounding area. The information may come from publicly available data sets.

[0044] The training data includes the pupil as training data (also called training signal or correct label). (pupil diameter or pupil area), pulse, blood pressure, body temperature, blinking, posture, eye redness, etc. In particular, pupil size (pupil diameter or pupil area) or blinking may be affected by mental fatigue. This is preferable as training data because it tends to change over time.

[0045] The information on the eyes and their surroundings prepared as learning data is stored in the storage unit 101. After the learning data is stored in storage unit 101, the process proceeds to step S002.

[0046] In step S002, machine learning is performed based on the learning data stored in the storage unit 101. The machine learning is performed by the generation unit 102.

[0047] For the machine learning, it is preferable to use, for example, supervised learning, and a neural network It is preferable to use supervised learning using a network (especially deep learning). stomach.

[0048] Deep learning, for example, convolutional neural networks (CNNs) Recurrent Neural Network Recurrent Neural Network (RNN), auto-encoding Autoencoder (AE), Variational Autoencoder (VAE) It is preferable to use a standard autoencoder or similar.

[0049] A trained model is generated by the above machine learning. The trained model is stored in the memory unit 1 It is stored in 04.

[0050] In addition, the pupil (pupil diameter or pupil area), pulse, blood pressure, and Temperature, blinking, posture, and bloodshot eyes vary depending on individual factors such as age, body type, and gender. The trained model may be updated to suit the user.

[0051] The above is an example of a method for generating a trained model.

[0052] [Method for measuring fatigue level] Next, an example of a method for measuring the fatigue level will be described. The method for measuring the fatigue level includes steps S003 to S006 shown in the following. The present invention includes a method for calculating the fatigue level and a method for determining whether the fatigue level is abnormal.

[0053] In step S003, information on the eyes and their surroundings is acquired for use in calculating the degree of fatigue. do.

[0054] The information on the eyes and their surroundings used to calculate the degree of fatigue may be, for example, information on the side or oblique position. It is preferable to obtain information about the eyes and their surroundings from a side or oblique direction. By acquiring information from a location that is difficult for the user to visually recognize, Therefore, the information can be obtained without the user being aware of it.

[0055] The information on the eyes and their surroundings used to calculate the fatigue level is collected by a camera, a pressure sensor, and , a distortion sensor, a temperature sensor, a gyro sensor, or the like. do.

[0056] Furthermore, information on the eyes and their surroundings, which is used to calculate the degree of fatigue, is acquired in chronological order.

[0057] Information about the eyes and their surroundings used to calculate the degree of fatigue is stored in the storage unit 104. After the information is stored in the storage unit 104, the process proceeds to step S004.

[0058] In step S004, the fatigue level is calculated. The generated trained model and the information on the eyes and their surroundings acquired in step S003 are The fatigue level is calculated by the measuring unit 105.

[0059] Calculating fatigue level refers to quantifying the index used to evaluate fatigue level. For example, pupil size (pupil diameter or pupil area), pulse rate, blood pressure, body temperature, and dizziness are used as indicators for this purpose. Use at least one of the following: water, posture, bloodshot eyes.

[0060] The calculation of the fatigue level is not limited to the quantification of an index for evaluating the fatigue level. For example, Using the trained model, fatigue is detected from the information on the eyes and their surroundings acquired in step S003. The degree of effort may be quantified.

[0061] Before proceeding to step S005, steps S003 and S004 are performed. This is repeated for a certain period of time. This allows us to check whether there are any abnormalities in the indicators used to evaluate fatigue. It is possible to obtain time series data to determine whether

[0062] In step S005, it is determined whether an abnormality has occurred in the index for evaluating the fatigue level. conduct.

[0063] If an abnormality is detected in the indicators used to evaluate fatigue, the fatigue level is considered high. If the fatigue level is determined to be high, the process proceeds to step S006. If it is determined that no abnormalities have occurred in the evaluation indicators, it is determined that the fatigue level is not high. If it is determined that the fatigue level is not high, the process proceeds to step S003.

[0064] In addition, when the fatigue level is quantified in step S004, if an abnormality occurs in the fatigue level numerical value, If it is determined that the fatigue level is abnormal, the fatigue level If it is determined that the fatigue level is high, the process proceeds to step S006. If it is determined that there is no abnormality in the fatigue level value, it is determined that the fatigue level is not high. If it is determined that the fatigue level is not high, the process proceeds to step S003.

[0065] In step S006, information is output. Indicators for assessing fatigue level, numerically calculated fatigue level, and judgment results on whether fatigue level is abnormal The information may be, for example, visual information such as character strings, numerical values, graphs, and colors, or audio and sound information. It is output as auditory information such as music.

[0066] After outputting the above information, the process ends.

[0067] The above is an explanation of an example of a method for calculating the fatigue level.

[0068] This concludes the description of an example of a method for assessing fatigue level.

[0069] <<Examples of how to assess fatigue>> In this section, a specific example of a method for evaluating fatigue level will be described with reference to FIGS. 3A to 7B. Here, the pupil (pupil diameter or pupil area) is used as an index to evaluate the degree of fatigue. Select Time Variation.

[0070] As the learning data prepared in step S001, for example, image data of the eyes and their surroundings are used. In this case, the image data of the eyes and their surroundings is obtained from the front. Image data of the eye and its surroundings taken from the side or oblique direction. It is preferable that the image data is obtained from the side or oblique direction. In image data acquired from the front, pupils (pupil diameter or pupil area) can be detected with high accuracy. Therefore, image data acquired from the front and image data acquired from the side or oblique direction can be By using the image data acquired from the side or oblique direction as training data, This allows for the generation of a trained model with higher accuracy than when only the data is used as training data. Cut.

[0071] The images of the eyes and their surroundings used for learning data are obtained by taking correct images using a camera, for example. It is advisable to take photographs from the front, side or oblique direction.

[0072] Cameras 111a to 111d are used to take pictures from the front, side, or oblique direction. Examples of such a photograph are shown in Figures 3A to 3C. Figure 3A shows the subject of the photograph as seen from above. FIG. 3B is a view of the subject of photography from the right side. FIG. 3C is a view of the subject of photography from the front. For clarity of illustration, in FIG. 3B, the camera 111a and the camera 111b are shown. In FIG. 3C, cameras 111c and 111d are omitted. The subjects of the photograph are not limited to those whose fatigue level is to be evaluated (users). It's okay.

[0073] As shown in FIGS. 3A to 3C, the eyes and their surroundings from the front are viewed by the camera 111c and The eyes and the like are photographed from the side or oblique direction using the camera 111d. The surroundings are photographed using camera 111a and camera 111b.

[0074] Before machine learning is performed, image data for learning may be processed or corrected. Image data can be processed or corrected, for example, by cutting out parts not required for machine learning, removing grayscale, etc. There are various filters such as scale conversion, median filter, and Gaussian filter. Processing or correction can reduce noise that occurs in machine learning.

[0075] As training data, images of the eye and its surroundings taken from the front at the same time were used. data, and a set of image data of the eye and its surroundings acquired from a side or oblique angle. It is preferable to prepare multiple combinations of the eye and its surroundings at the same time. By taking a photograph from a different angle, the above processing or correction can be easily performed. It is possible to generate highly trained models, e.g., for frontal capture of eyes and their Taking into account the surrounding image data, images of the eye and its surroundings acquired from the side or oblique direction are This allows for the processing or correction of image data acquired from the side or oblique angles. The pupil outline in the image data of the area around the pupil is enhanced, and the pupil (pupil diameter or pupil area) is ) can be detected with high accuracy.

[0076] In addition, when many images can be acquired from the side or oblique direction, Alternatively, only image data acquired from an oblique direction may be used as learning data. Considering the image data of the eye and its surroundings acquired from the side or oblique direction, When processing or correcting image data of the eye and its surroundings, Alternatively, only image data acquired from the image processing device may be used as training data.

[0077] As described above, the learning data is image data, so the machine learning performed in step S002 Preferably, a convolutional neural network is used for training.

[0078] [Convolutional Neural Networks] Here, we will explain convolutional neural networks (CNNs).

[0079] Figure 4 shows an example of the CNN configuration. The CNN consists of a convolutional layer CL, a pooling layer PL, and a fully connected The CNN is composed of a composite FCL. Image data IPD is input to the CNN, In this embodiment, the image data IPD is an image of the eye and its surroundings. It's data.

[0080] The convolution layer CL has the function of performing convolution on image data. Repeated multiplication and accumulation operations are performed between a portion of the image data and the filter value of the weighting filter (also called the kernel). The convolution in the convolution layer CL extracts image features. can be.

[0081] The above multiplication and accumulation operation may be performed on software using a program, or on hardware. When the multiply-accumulate operation is performed by hardware, the multiply-accumulate operation circuit may be This product-sum calculation circuit may be a digital circuit or an analog circuit. A logging circuit may also be used.

[0082] The product-sum operation circuit is a transistor having Si in the channel formation region (Si transistor). It may be formed by a metal oxide in the channel forming region. In particular, the OS transistor Since the off-state current is extremely small, In addition, it is suitable for multiply-and-accumulate operations using both Si transistors and OS transistors. A circuit may be configured.

[0083] The convolution can use one or more weight filters. When using this data, it is possible to extract multiple features contained in the image data. uses three filters (filter F a , F b , F c ) is used in the following example: The image data input to the convolution layer CL is filtered by the filter F a , F b , F c The image data D a , D b , D c is generated. Image data D a , D b , D c is also called a feature map.

[0084] Image data D generated by convolution a , D b , D c is transformed by the activation function After that, it is output to the pooling layer PL. The activation function is ReLU (Rectif ReLU is a linear unit (RLU) that is used when the input value is If the value is negative, it outputs "0", and if the input value is "0" or greater, it outputs the input value as is. In addition, sigmoid function, tanh function, etc. are used as activation functions. It is also possible.

[0085] The pooling layer PL performs pooling on the image data input from the convolution layer CL. Pooling is a process of dividing image data into multiple regions and assigning a specific value to each region. This is a process of extracting certain data and arranging them in a matrix. The spatial size of the image data is reduced while retaining the features extracted by the layer CL. ,Enhancing the position or translation invariance of features extracted by the convolutional layer CL As for pooling, there are max pooling, average pooling, Lp pooling, etc. Gu etc. can be used.

[0086] CNN extracts features by the above convolution and pooling processes. A neural network can be composed of multiple convolutional layers CL and multiple pooling layers PL. In Figure 4, a layer L consisting of a convolutional layer CL and a pooling layer PL is (z is an integer of 1 or more) z ), convolution and pooling This shows a configuration in which ring processing is performed z times. In this case, feature extraction is performed in each layer L. This allows for more advanced feature extraction.

[0087] The fully connected layer FCL uses the convolutional and pooled image data to generate the All nodes in the fully connected layer FCL are connected to the previous layer (this Here, we have the pooling layer PL, and in Figure 4, we have the layer L z All nodes in the pooling layer (PL) The image data output from the convolution layer CL or the pooling layer PL is is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. Then, the one-dimensionally expanded data OPD is output.

[0088] The configuration of the CNN is not limited to that shown in Figure 4. For example, if the pooling layer PL has multiple It may be provided for each convolution layer CL. In addition, it is necessary to retain the position information of the extracted features as much as possible. If desired, the pooling layer PL may be omitted.

[0089] In addition, when classifying images using the output data of the fully connected layer FCL, the fully connected layer FCL and the The output layer may be provided with an electrically connected output layer. Using functions such as scalar functions, the probability of being classified into each class can be output. For example, the class may be the degree of fatigue. "Very high fatigue," "High fatigue," "Moderate fatigue," "Low fatigue," "The level of fatigue is very low." This allows the fatigue level to be quantified from the image data. It is possible.

[0090] In addition, when performing regression analysis such as numerical prediction from the output data of the fully connected layer FCL, An output layer electrically connected to the composite layer FCL may be provided. By using the above, it is possible to output a predicted value. , pupil diameter or pupil area can be calculated.

[0091] In addition, CNN uses image data as training data and trains it with training data. For example, backpropagation can be used for supervised learning. Through CNN learning, the filter values ​​of the weight filters and the weight coefficients of the fully connected layers can be optimized. It can be made into

[0092] This concludes the explanation of convolutional neural networks (CNNs).

[0093] In the supervised learning described above, images of the eyes and their surroundings taken from the front are used as training data. Image data of the eye and its surroundings taken from the side or oblique direction are prepared. For example, the training data is a set of pupil diameters or pupil areas. Given a given diameter or pupil area, we can use CNN regression to find the pupil diameter or pupil area. The above steps output an image of the eye and its surroundings taken from the side or oblique direction. From the data, a trained model is generated that outputs pupil diameter or pupil area.

[0094] It should be noted that the fatigue level may be quantified by classifying using CNN. When this happens, the image data of the eye and its surroundings obtained from the side or oblique direction is digitized. A trained model is generated that outputs the fatigue level.

[0095] In step S003, information on the eyes and their surroundings is acquired for use in calculating the degree of fatigue. For example, information on the eyes and their surroundings obtained from the side or oblique direction is used. The images of the eyes and their surroundings are acquired using a camera or other device. It is best to take a photo from the side.

[0096] Using the camera 112a and the camera 112b, photographs are taken from the side or oblique direction. Examples are shown in Figures 5A and 5B. Figure 5A shows the subject of the image being photographed from above. FIG. 5B is a front view of the subject of the photograph. The subject of the photograph has a fatigue level of They are people who are evaluated (users).

[0097] As shown in Figures 5A and 5B, the eye and its surroundings from the side or oblique direction are The image is taken using camera 112a and camera 112b.

[0098] Note that the camera 111 shown in FIG. 3A (one of the cameras 111a to 111d) 5A (camera 112a and / or camera 112) In b), it is preferable that the distance to the subject is approximately the same. This allows the degree of fatigue to be measured with high accuracy. In the method for evaluating the fatigue level according to one embodiment of the present invention, Since the camera 111 and the camera 112 perform learning, the distance to the subject is not necessarily the same. They do not have to be equal.

[0099] In addition, the image captured by the camera 111 and the image captured by the camera 112 are It is preferable that the display size, resolution, aspect ratio, etc. are the same. This will reduce fatigue and improve accuracy. In the method for evaluating the degree of fatigue according to one embodiment of the present invention, To perform supervised learning, an image taken by camera 111 and an image taken by camera 112 are taken. The image does not necessarily have to have the same resolution, aspect ratio, etc. as the projected image.

[0100] 6A and 6B show schematic diagrams of a person's visual field (binocular vision). 6B is a view of the person as seen from the right side.

[0101] The human visual field is classified into the effective visual field, the guided visual field, and the auxiliary visual field. In the figure, the dashed line from the person to the gaze point C is the line of sight (visual axis), and the angle θ 1h and Angle θ 1v is the range of the field of view angle of the effective field of view, and the angle θ 2h and angle θ 2v is the induced visual field is the range of the viewing angle, and the angle θ 3h and angle θ 3v is the range of the auxiliary field of view. Unless otherwise specified, the line of sight is the length of the line connecting the gaze point C and the right eye. The length of the line connecting the gaze point C and the left eye is equal to the gaze point C. The horizontal direction refers to the line to point C. The horizontal direction refers to the direction horizontal to the plane that includes both eyes and the line of sight. The vertical direction refers to the direction perpendicular to the plane that contains both eyes and the line of sight.

[0102] The effective visual field is the area in which information can be received instantly. The angle θ shown in FIG. 1h ) is said to be a range of about 30 degrees from the center of the line of sight, The vertical viewing angle (angle θ shown in Figure 6B) 1v ) is located slightly below the line of sight and at an angle of about 20° It's called the range.

[0103] The induced visual field is the area that affects the spatial coordinate system. angle (angle θ shown in Figure 6A) 2h ) is said to be in the range of about 100° around the line of sight, and guidance The vertical viewing angle of the field of view (angle θ shown in Figure 6B) 2v ) is located slightly below the line of sight, about 8 It is said to be in the range of 5°.

[0104] The auxiliary visual field is an area where the presence of a stimulus can be recognized. angle (angle θ shown in Figure 6A) 3h ) is said to be in the range of about 200° around the line of sight, The vertical viewing angle of the field of view (angle θ shown in Figure 6B) 3v ) is located slightly below the line of sight and is approximately 1 It is said to be in the range of 25°.

[0105] The information received during work is mostly from the effective visual field, and a small amount from the induced visual field. In addition, there is almost no information from the auxiliary field of vision during work. It is difficult to recognize the information placed on the screen.

[0106] In addition, when selecting the change in pupil size over time as an index for assessing fatigue, The image acquired by S003 must include the pupil. Visual information is obtained from the pupil, lens, etc. The image projected onto the retina through the optic nerve is transmitted to the brain and recognized. In other words, because the auxiliary visual field also contains visual information, it is possible to recognize the pupil within the auxiliary visual field. can.

[0107] From the above, the side or oblique direction in which images of the eye and its surroundings are acquired is the same as the horizontal direction. In the direction of the auxiliary visual field, the pupil is observed from within the auxiliary visual field or within the induced visual field near the auxiliary visual field. Therefore, the angle θ shown in FIG. a The side or diagonal direction specifically refers to the range of the line of sight. horizontally to the surface of the substrate, the angle is 45° or more and 100° or less, preferably 50° or more and 90° or less, The angle is preferably between 60° and 85°. This makes it difficult for the user to visually recognize the angle. Therefore, the image can be acquired without the user being aware of it. An image can be acquired.

[0108] In addition, if the side or oblique direction is within the above range in the horizontal direction, In the vertical direction, any angle falls outside the induced visual field. The "sideways" or "oblique directions" may be any directions within a range in which the pupil can be photographed.

[0109] In addition, the front where the image of the eye and its surroundings is acquired is the induced field of view in the horizontal direction. The direction in which the pupil is observed from the inside. Specifically, the front is the horizontal direction to the line of sight, 0 0° or more and 50° or less, preferably 0° or more and 30° or less, and more preferably 0° or more and 15° or less. This allows you to photograph a pupil that is circular or close to circular, and the pupil diameter or pupil size is The hole area can be calculated with high accuracy.

[0110] When the change in pupil size over time is selected as an index for evaluating the degree of fatigue, step S00 In Section 4, we use a trained model to analyze the eye and its surroundings captured from the side or oblique direction. The pupil diameter or pupil area is calculated from the image data of the side.

[0111] As mentioned above, when the sympathetic nervous system is dominant, the pupils dilate, and when the parasympathetic nervous system is dominant, In other words, the pupil diameter changes with the disturbance of the autonomic nervous system. It is said that the rate of pupil size slows down as fatigue accumulates. The change in pupil (pupil diameter or pupil area) over time is the change in pupil (pupil diameter or pupil area) over time. or pupil area), and the rate of change in the pupil (pupil diameter or pupil area) expansion and contraction cycle over time. Refers to...

[0112] Whether or not there is an abnormality in the change in pupil (pupil diameter or pupil area) over time can be determined by step The judgment is based on the pupil (pupil diameter or pupil area) immediately after the start of S003.

[0113] A method for determining whether abnormalities have occurred in the change of pupil (pupil diameter or pupil area) over time An example will be described with reference to FIGS. 7A and 7B.

[0114] 7A and 7B are schematic diagrams showing the change in pupil diameter over time. In the graphs, the horizontal axis represents time and the vertical axis represents pupil diameter. The dashed-dotted lines in Figures 7A and 7B show the time-averaged pupil diameter.

[0115] FIG. 7A is a diagram that schematically shows how the pupil diameter decreases over time. In order to determine whether the change in pupil diameter over time is abnormal, a threshold value for pupil diameter is set in advance. For example, as shown by the dashed line in Figure 7A, the upper limit of the pupil diameter is set to r max and pupil diameter The lower limit of r min In the example of FIG. 7A, the pupil diameter at time t is set to the lower limit r m in At this time, it is determined that an abnormality has occurred in the change in the pupil over time.

[0116] For example, using the pupil (pupil diameter or pupil area) immediately after the start of step S003 as a reference, When the pupil (pupil diameter or pupil area) expands or contracts at a certain rate or more, It is determined that an abnormality has occurred.

[0117] Figure 7B shows a schematic diagram of the pupil diameter expansion and contraction cycle as time passes. The expansion and contraction period of the pupil diameter is f u (u is a natural number.) The diameter of the pupil In order to determine whether the time change is abnormal, a threshold value for the expansion and contraction cycle of the pupil diameter is set in advance. For example, as shown in FIG. 7B, the upper limit of the pupil diameter expansion / contraction cycle is set as f max year, The lower limit of the pupil diameter expansion and contraction cycle is f min In the example of FIG. 7B, the pupil diameter expansion / contraction cycle f t+7 is the upper limit of the pupil diameter expansion and contraction cycle f max At this time, abnormalities occur in the changes in the pupil over time. is determined to be alive.

[0118] The expansion and contraction of the pupil diameter and the expansion and contraction cycle of the pupil diameter are observed together. A fast Fourier transform may be performed on the change in diameter over time. It becomes easier to determine whether or not there is an abnormality based on the period.

[0119] Using the trained model, we can visualize the eyes and their surroundings captured from the side or oblique direction. We have given an example of a method for calculating pupil diameter or pupil area from image data of the side. For example, a trained model can be used to analyze eye images acquired from the side or oblique direction. The fatigue level may be calculated numerically from the image data of the area and its surroundings. In order to determine whether the fatigue level is abnormal, a fatigue threshold (the upper limit of fatigue level) is set in advance. Set the limit.

[0120] The above is a description of the fatigue level evaluation system. By using this system, the system (especially the acquisition unit) is not positioned in the user's line of sight. Therefore, the increase in mental fatigue of the user can be suppressed. Therefore, the degree of fatigue during use can be evaluated with high accuracy. It is possible.

[0121] The configurations, methods, etc. described in this embodiment may be used in combination with the configurations, methods, etc. described in other embodiments. They can be used in appropriate combination.

[0122] (Embodiment 2) In this embodiment, a fatigue evaluation device will be described with reference to FIGS. 8A to 9B. The fatigue evaluation device is an electronic device equipped with the fatigue evaluation system described in the previous embodiment. or instruments and electronic equipment.

[0123] Examples of devices that are part of the fatigue evaluation system include corrective glasses and safety glasses. Examples of such equipment include safety equipment worn on the head, such as eyeglasses, helmets, and gas masks.

[0124] The above-mentioned device is a device that includes at least an acquisition unit in the fatigue level evaluation system described in the previous embodiment. 103. The device also has a battery.

[0125] Examples of electronic devices that include a part of the fatigue evaluation system include information terminals and computers. In this case, the computer includes a tablet computer, In addition to laptop and desktop computers, workstations and servers This includes large computers such as server systems.

[0126] The above equipment is equipped with a GPS (Global Positioning System) By installing a receiver, the electronic device can use GPS to determine the location and distance traveled by the device. The acquired data may be used to evaluate the degree of fatigue. By combining it with an index, fatigue levels can be assessed with greater accuracy.

[0127] An example of an instrument and an electronic device equipped with a fatigue evaluation system is shown in FIG. 8A. The glasses 200 and the server 300 are equipped with a fatigue evaluation system. The server 300 includes a processing unit 201. The server 300 also includes a processing unit 301.

[0128] For example, the processing unit 201 includes the acquisition unit 103 described in the previous embodiment, and the processing unit 3 01 includes the accumulation unit 101, the generation unit 102, the storage unit 104, and the like, which have been described in the previous embodiment. , and a measurement unit 105. Furthermore, each of the processing unit 201 and the processing unit 301 includes: The processing unit 201 has only the acquisition unit 103. Therefore, the weight of the eyeglasses 200 can be reduced. This reduces the burden on the user's body.

[0129] In addition, when a camera is used as the acquisition unit 103, the acquisition unit 103 is By placing the camera near the eyes, it is possible to take close-up shots of the eyes and their surroundings. This makes it easier to detect the eyes. It also reduces the amount of external light reflected in the eyes. This reduces the number of times that images of the eye and its surroundings need to be processed or corrected. Alternatively, processing or correction becomes unnecessary.

[0130] Although FIG. 8A shows an example in which a camera is used as the acquisition unit 103, the acquisition unit 103 is not limited to this. The acquisition unit 103 may include a pressure sensor, a strain sensor, a temperature sensor, a gyro sensor, etc. In this case, the acquisition unit 103 is not installed on the side of the eye or in a direction other than an oblique direction. For example, the acquisition unit 103 may be located at a position where the head contacts the frame of the eyeglasses 200 or at a position where the head contacts the frame of the eyeglasses 200. It may be installed nearby.

[0131] The processing unit 201 may include the output unit 106 described in the previous embodiment. The unit 201 has the output unit 106, so that the user can know the degree of fatigue while working. Components included in the output unit 106 include a display, a speaker, and the like.

[0132] The information provided by the output unit 106 may be visual information such as color, or audio information such as sound or music. It is preferable to output the information as text, numbers, graphs, etc. Therefore, visual information such as color has little impact on the visual sense and causes little stress to the user. The same applies to auditory information such as voice and music. By registering your favorite music in advance, you may be able to reduce the level of fatigue of your users.

[0133] The configuration of the processing unit 201 and the processing unit 301 is not limited to this. 201 includes an acquisition unit 103, a storage unit 104, a measurement unit 105, and an output unit 106. The processing unit 301 may include a storage unit 101 and a generation unit 102. The unit 201 has a function of measuring the fatigue level, and the processing unit 301 generates a trained model. It has a function.

[0134] With the above configuration, the fatigue level can be measured only by the processing unit 201. The frequency of communication between the unit 201 and the processing unit 301 can be minimized. By doing so, the processing unit 301 transmits the trained model updated by the processing unit 301 to the processing unit 201. The processing unit 201 can receive the trained model. Update the trained model recorded in 201 to the received trained model. This allows for the use of training data with improved accuracy, and the fatigue level can be calculated with higher accuracy. It can be evaluated.

[0135] The memory unit 104 may store information about the eyes and their surroundings acquired by the acquisition unit 103. After a certain amount of acquired information about the eye and its surroundings is accumulated in the memory unit 104, the accumulated information is The information may be transmitted to an electronic device having the processing unit 301. The number of communications with the processing unit 301 can be reduced.

[0136] The fatigue evaluation system may be configured with a plurality of electronic devices each including a part of the fatigue evaluation system. The system is composed of glasses, a server, and a mobile phone (smartphone), which is a type of information terminal, equipped with a vision evaluation system. 8B. Similar to the glasses 200 and server 300 shown in FIG. 8A, The glasses 200 have a processing unit 201, and the server 300 has a processing unit 301. The terminal 310 has a processing unit 311 .

[0137] For example, the processing unit 201 includes an acquisition unit 103. The processing unit 301 includes a storage unit 101 and The processing unit 311 includes a memory unit 104, a measurement unit 105, and an output unit 106. In addition, each of the processing units 201, 301, and 311 It has a transceiver unit.

[0138] In the above configuration, when the user of the glasses 200 carries the information terminal 310, You can check your own fatigue level via 310.

[0139] In addition, if the information terminal 310 is owned by the boss of the user of the glasses 200, the information terminal 3 10, the user's fatigue level can be confirmed. Even if the user's supervisor is not nearby, the user's supervisor can monitor the user's health status. In addition, the information output from the output unit 106 may be a character string, a numerical value, a graph, or the like relating to the fatigue level. Any visual information can provide detailed information about the user's health condition.

[0140] The glasses 200 shown in FIGS. 8A and 8B are not limited to vision correction glasses, but may also be used for sunglasses, colored glasses, etc. Vision correction glasses, 3D glasses, Augmented Reality (AR) glasses , Mixed Reality (MR) glasses, fashion glasses, blue light filters The glasses may be used for personal computers having a viewing function.

[0141] In particular, in AR glasses and MR glasses, information about fatigue level is stored as a string or a numerical value. By outputting the information as visual information such as graphs, the degree of fatigue can be known in detail.

[0142] FIG. 9A is a diagram showing protective glasses equipped with a part of the fatigue evaluation system. The safety glasses 210 have a processing unit 211. The processing unit 211 also includes an acquisition unit 103. It can be enjoyed.

[0143] The processing unit 211 is similar to the processing unit 201 of the glasses 200 shown in FIGS. 8A and 8B. It is desirable to have the following functions.

[0144] Although FIG. 9A shows a goggle type as an example of the protective glasses 210, the protective glasses are not limited to this. The camera may be of a spectacle type or a front type. However, the present invention is not limited to this and may be of a twin-lens type.

[0145] 8A, 8B, and 9A show the following devices that are part of the fatigue evaluation system: Although the above examples include eyeglasses such as corrective eyeglasses and safety eyeglasses, the present invention is not limited to these. Examples of safety equipment include head-mounted protective equipment such as a head protector, a gas mask, etc.

[0146] So far, we have discussed fatigue evaluation devices, including a device that includes part of a fatigue evaluation system, and a fatigue evaluation system. The configuration in which an electronic device equipped with a part of the evaluation system is combined has been described. The fatigue evaluation device is, for example, a detachable device that includes a part of the fatigue evaluation system. The electronic device may be a combination of the electronic device described above. A head-mounted device with a removable electronic device 320 that includes part of the hearing assessment system. The detachable electronic device 320 includes an acquisition unit 103. By providing a part of the evaluation system in a detachable electronic device 320, Safety equipment is available.

[0147] In addition, part of the fatigue assessment system is implemented on, for example, a head-mounted display, a smart The display device may be mounted on a head such as glasses. For example, even in situations where virtual reality (VR) is used, fatigue can be reduced. The degree of effort can be evaluated.

[0148] The fatigue evaluation device may be a single device or a single electric device equipped with a fatigue evaluation system. It may also be a child device.

[0149] By using the fatigue evaluation device according to one aspect of the present invention, the user's field of vision is ensured and the user can Therefore, the user's level of fatigue can be evaluated with high accuracy. In addition, if the user is a worker, there is no need to stop work to assess fatigue. This will help prevent a decline in labor productivity.

[0150] Furthermore, by using the fatigue evaluation device according to one aspect of the present invention, information on the eyes and their surroundings can be obtained as follows: It can be acquired from a position close to the eyes, which increases the accuracy of fatigue evaluation. can.

[0151] The configurations, methods, etc. described in this embodiment may be used in combination with the configurations, methods, etc. described in other embodiments. They can be used in appropriate combination. [Explanation of symbols]

[0152] : 100: fatigue evaluation system, 101: accumulation unit, 102: generation unit, 103: acquisition unit, 1 04: memory unit, 105: measurement unit, 106: output unit, 111: camera, 111a: camera, 111b: camera, 111c: camera, 111d: camera, 112: camera, 112a: Camera, 112b: camera, 200: glasses, 201: processing unit, 210: safety glasses, 211 : Processing unit, 220: Safety protector, 300: Server, 301: Processing unit, 310: Information terminal, 311: Processing unit, 320: Electronic equipment

Claims

[Claim 1] The apparatus includes an accumulation unit, a generation unit, a storage unit, an acquisition unit, and a measurement unit, the storage unit has a function of storing a plurality of first images and a plurality of second images; the plurality of first images are images of the eye and its surroundings taken from a side or oblique direction, the plurality of second images are images of the eye and its surroundings taken from the front, The generation unit has a function of performing supervised learning and generating a trained model, The storage unit has a function of storing the trained model, the acquisition unit has a function of acquiring a third image, the third image is an image of the eye and its surroundings taken from a side or oblique direction, The measurement unit has a function of measuring a fatigue level from the third image based on the trained model. Fatigue rating system.

Images