Information processing device

The information processing device uses machine learning to analyze pupil size changes for real-time and accurate detection of user fatigue and drowsiness in information terminals, addressing the inefficiencies of existing methods.

JP2024161623AActive Publication Date: 2024-11-19SEMICON ENERGY LAB CO LTD

Patent Information

Application Number
JP2024150460
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-09-06
Filing Date
2024-09-02
Publication Date
2024-11-19
Estimated Expiration
2040-08-25

AI Technical Summary

Technical Problem

Existing methods for detecting user fatigue and drowsiness in information terminals like smartphones require dedicated devices and are not efficient in real-time detection, accuracy, or simplicity.

Method used

An information processing device equipped with an imaging unit and calculation unit performing machine learning, specifically using neural networks, to analyze pupil size changes over time for fatigue and drowsiness detection.

Benefits of technology

Enables real-time, accurate, and simple estimation of user fatigue and drowsiness without the need for dedicated devices, allowing for timely adjustments to reduce user discomfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161623000001_ABST
    Figure 2024161623000001_ABST
Patent Text Reader

Abstract

To provide an information processing system having a function to detect fatigue and drowsiness, etc. of a user.SOLUTION: An information processing system includes a first information processing device and a second information processing device and has a function to perform calculation by machine learning. The first information processing device has a function to acquire a first video including a face and to detect eyes from two or more first images included in the first video. The first information processing device also has a function to detect pupils from the eyes detected from the first image, calculate the sizes of the pupils, and learn using a temporal change of the sizes of the pupils. The second information processing device has a function to acquire a second video including a face and to detect eyes from two or more second images included in a second video. The second information processing device also has a function to detect pupils from the eyes detected from the second image, to calculate the sizes of the pupils, and to make an inference on the basis of the results of the learning made by the first information processing device with respect to a temporal change of the sizes of the pupils.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] TECHNICAL FIELD One aspect of the present invention relates to an information processing device, an information processing system, an information processing method, and an information terminal. [Background technology]

[0002] When using an information terminal such as a smartphone or a tablet for a long period of time, the user may feel fatigued, sleepy, etc. In particular, staring at the screen of the information terminal for a long period of time may cause eye fatigue. Patent Document 1 discloses an eye fatigue detection device and a detection method.

[0003] Pupil diameter changes depending on the presence or absence of fatigue, drowsiness, etc. For example, when fatigued or drowsy, the pupil diameter is smaller than when not fatigued or drowsy. In addition, while pupil diameter generally fluctuates periodically, when fatigued or drowsy, the period of pupil diameter fluctuation is longer than when not fatigued or drowsy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-169601 A Summary of the Invention [Problem to be solved by the invention]

[0005] It is preferable to be able to detect the fatigue, drowsiness, etc. of a user in real time while using an information terminal such as a smartphone or tablet, because, for example, the operation of the information terminal can be changed depending on the presence or absence of the user's fatigue, drowsiness, etc. When detecting fatigue, drowsiness, etc. in real time, it is preferable that the information terminal being used itself has a function of estimating the user's fatigue, drowsiness, etc. However, a dedicated device is required to detect eye fatigue using the method shown in Patent Document 1.

[0006] An object of one embodiment of the present invention is to provide an information processing device having a function of detecting fatigue, drowsiness, etc. of a user in real time. Another object of one embodiment of the present invention is to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user with high accuracy. Another object of one embodiment of the present invention is to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user in a simple manner. Another object of one embodiment of the present invention is to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user in a short time.

[0007] An object of one embodiment of the present invention is to provide an information processing system having a function of detecting fatigue, drowsiness, etc. of a user in real time. Another object of one embodiment of the present invention is to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user with high accuracy. Another object of one embodiment of the present invention is to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user in a simple manner. Another object of one embodiment of the present invention is to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user in a short time.

[0008] It should be noted that the description of multiple problems does not preclude the existence of each other's problems. One embodiment of the present invention does not need to solve all of the problems exemplified. In addition, problems other than those listed will become apparent from the description of this specification, and such problems may also be problems of one embodiment of the present invention. [Means for solving the problem]

[0009] One embodiment of the present invention is an information processing device that has an imaging unit and a calculation unit having a function of performing calculations through machine learning, where the imaging unit has a function of acquiring a video that is a collection of two or more frames of images, the calculation unit has a function of detecting a first object from each of two or more images included in the video, the calculation unit has a function of detecting a second object from each of the detected first objects, the calculation unit has a function of calculating a size of the second object for each of the detected second objects, and the calculation unit has a function of performing machine learning using changes in the size of the second objects over time.

[0010] In the above aspect, the machine learning may be performed by a neural network.

[0011] In the above aspect, the video may include a face, the first object may be an eye, and the second object may be a pupil.

[0012] Another embodiment of the present invention is an information processing device having a function of making inference based on learning results obtained by learning using changes over time in the size of a first object shown in two or more first images included in a first video, wherein the information processing device has a function of acquiring a second video, the information processing device has a function of detecting a second object from each of two or more second images included in the second video, the information processing device has a function of detecting a third object from each of the detected second objects, the information processing device has a function of calculating the size of the third object for each of the detected third objects, and the information processing device has a function of making inference based on the learning results regarding the changes over time in the size of the third object.

[0013] In the above aspect, the learning and inference may be performed by a neural network, and the learning result may include a weighting coefficient.

[0014] Also, in the above aspect, the first video may include a first face, the second video may include a second face, the first and third objects may be pupils, and the second object may be an eye.

[0015] In the above aspect, the information processing device may have a function of estimating fatigue of a person having the second face. Effect of the Invention

[0016] According to one aspect of the present invention, it is possible to provide an information processing device having a function of detecting fatigue, drowsiness, etc. of a user in real time. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user with high accuracy. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user in a simple manner. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing device having a function of estimating fatigue, drowsiness, etc. of a user in a short time.

[0017] According to one aspect of the present invention, it is possible to provide an information processing system having a function of detecting fatigue, drowsiness, etc. of a user in real time. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user with high accuracy. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user in a simple manner. Alternatively, according to one aspect of the present invention, it is possible to provide an information processing system having a function of estimating fatigue, drowsiness, etc. of a user in a short time.

[0018] Note that the description of multiple effects does not preclude the existence of other effects. Moreover, one embodiment of the present invention does not necessarily have all of the exemplified effects. Furthermore, problems, effects, and novel features of one embodiment of the present invention other than those described above will become apparent from the description and drawings in this specification. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a flowchart showing an example of a method of operating the information processing device. [Diagram 3] FIG. 3 is a flowchart showing an example of a method of operating the information processing device. [Figure 4] FIG. 4 is a flowchart showing an example of a method of operating the information processing device. [Diagram 5] 5A, 5B, and 5C are schematic diagrams showing an example of a method of operating the information processing device. [Figure 6] 6A and 6B are schematic diagrams showing an example of a method of operating the information processing device. [Figure 7] 7A1 and 7A2, and also FIGS. 7B1 and 7B2 are schematic diagrams showing an example of an operation method of the information processing device. [Figure 8] 8A and 8B are schematic diagrams showing an example of a method of operating the information processing device. [Figure 9] 9A and 9B are schematic diagrams showing an example of a method of operating the information processing device. [Figure 10] FIG. 10A, and FIGS. 10B1 and 10B2 are schematic diagrams showing an example of a method of operating the information processing device. [Figure 11] FIG. 11 is a diagram illustrating AnoGAN that can be applied to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described. However, one aspect of the present invention is not limited to the following description, and it will be easily understood by those skilled in the art that the form and details of the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, one aspect of the present invention should not be interpreted as being limited to the description of the embodiment shown below.

[0021] In addition, in the drawings attached to this specification, the components are classified by function and shown in block diagrams as independent blocks, but in actuality it is difficult to completely separate the components by function, and one component may be involved in multiple functions or one function may be realized by multiple components.

[0022] (Embodiment 1) In this embodiment, an information processing system according to one embodiment of the present invention and an information processing method using the information processing system will be described. The information processing system and the information processing method according to one embodiment of the present invention can estimate fatigue, drowsiness, and the like of a user of an information terminal such as a smartphone or a tablet. In particular, eye fatigue of the user of the information terminal can be detected.

[0023] <Example of information processing system configuration> 1 is a block diagram showing an example of a configuration of an information processing system 10 according to an embodiment of the present invention. The information processing system 10 includes an information processing device 20 and an information processing device 30.

[0024] The information processing device 20 has an imaging unit 21, a display unit 22, a calculation unit 23, a main memory unit 24, an auxiliary memory unit 25, and a communication unit 26. Data and the like can be transmitted between the components of the information processing device 20 via a transmission path 27. Moreover, the information processing device 30 has an imaging unit 31, a display unit 32, a calculation unit 33, a main memory unit 34, an auxiliary memory unit 35, and a communication unit 36. Data and the like can be transmitted between the components of the information processing device 30 via the transmission path 37.

[0025] The imaging units 21 and 31 have a function of capturing images and acquiring captured image data. The display units 22 and 32 have a function of displaying images.

[0026] The calculation unit 23 and the calculation unit 33 have a function of performing calculation processing. The calculation unit 23 has a function of performing a predetermined calculation processing on data transmitted to the calculation unit 23 from, for example, the imaging unit 21, the main memory unit 24, the auxiliary memory unit 25, or the communication unit 26 via the transmission path 27. The calculation unit 33 has a function of performing a predetermined calculation processing on data transmitted to the calculation unit 33 from, for example, the imaging unit 31, the main memory unit 34, the auxiliary memory unit 35, or the communication unit 36 ​​via the transmission path 37. In addition, the calculation unit 23 and the calculation unit 33 have a function of performing calculation by machine learning. For example, they have a function of performing calculation using a neural network. The calculation unit 23 and the calculation unit 33 can have, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), etc.

[0027] The main memory unit 24 and the main memory unit 34 have a function of storing data, programs, etc. The calculation unit 23 can read the data, programs, etc. stored in the main memory unit 24 and execute calculation processing. For example, the calculation unit 23 can execute a predetermined calculation processing on the data read from the main memory unit 24 by executing a program read from the main memory unit 24. Furthermore, the calculation unit 33 can read the data, programs, etc. stored in the main memory unit 34 and execute calculation processing. For example, the calculation unit 33 can execute a predetermined calculation processing on the data read from the main memory unit 34 by executing a program read from the main memory unit 34.

[0028] It is preferable that the main memory unit 24 and the main memory unit 34 operate faster than the auxiliary memory unit 25 and the auxiliary memory unit 35. The main memory unit 24 and the main memory unit 34 may include, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0029] The auxiliary storage unit 25 and the auxiliary storage unit 35 have a function of storing data, programs, etc. for a longer period of time than the main storage unit 24 and the main storage unit 34. The auxiliary storage unit 25 and the auxiliary storage unit 35 may have, for example, a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary storage unit 25 and the auxiliary storage unit 35 may also have a non-volatile memory such as a resistive random access memory (ReRAM, also called a resistive memory), a phase change random access memory (PRAM), a ferroelectric random access memory (FeRAM), a magnetoresistive random access memory (MRAM, also called a magnetoresistive memory), or a flash memory.

[0030] The communication unit 26 has a function of transmitting and receiving data, etc., to devices, etc., provided outside the information processing device 20. The communication unit 36 ​​has a function of transmitting and receiving data, etc., to devices, etc., provided outside the information processing device 30. For example, by supplying data, etc., from the communication unit 26 to the communication unit 36, it is possible to supply data, etc., from the information processing device 20 to the information processing device 30. Furthermore, the communication unit 26 and the communication unit 36 ​​can have a function of supplying data, etc., to a network and a function of acquiring data, etc., from the network.

[0031] Here, when the calculation unit 23 and the calculation unit 33 have a function of performing calculations by machine learning, for example, the calculation unit 23 can learn and the learning result can be supplied from the information processing device 20 to the information processing device 30. For example, when the calculation unit 23 and the calculation unit 33 have a function of performing calculations using a neural network, the calculation unit 23 can learn and obtain weighting coefficients, etc., and the weighting coefficients, etc. can be supplied from the information processing device 20 to the information processing device 30. As described above, even if the calculation unit 33 provided in the information processing device 30 does not learn, inference based on the learning result by the calculation unit 23 provided in the information processing device 20 can be performed on the data input to the calculation unit 33. Therefore, the calculation processing capacity of the calculation unit 33 can be made lower than that of the calculation unit 23.

[0032] When the calculation unit 23 performs learning and the learning result is supplied from the information processing device 20 to the information processing device 30, the information processing device 20 can be provided in, for example, a server. When the information processing device 20 is provided in the server, the information processing device 20 does not need to be provided with the imaging unit 21 and the display unit 22. In other words, the imaging unit 21 and the display unit 22 may be provided outside the information processing device 20.

[0033] Moreover, the information processing device 30 can be provided in an information terminal such as a smartphone, a tablet, or a personal computer. Moreover, at least some of the components of the information processing device 20 and at least some of the components of the information processing device 30 may both be provided in a server. For example, the calculation unit 23 and the calculation unit 33 may be provided in the server. In this case, for example, data acquired by the information terminal is supplied to the calculation unit 33 via a network, and the calculation unit 33 provided in the server performs inference or the like on the data. Then, the inference result is supplied to the information terminal via the network, so that the information terminal can acquire the inference result.

[0034] <An example of information processing method> The following describes an example of an information processing method using the information processing system 10. Specifically, the following describes an example of a method for estimating fatigue, drowsiness, and the like of a user of an information terminal in which an information processing device 30 of the information processing system 10 is provided, by calculation using machine learning.

[0035] 2 and 3 are flowcharts illustrating an example of a method for estimating fatigue, drowsiness, etc., by calculation using machine learning. Learning is shown in FIG. 2, and inference is shown in FIG.

[0036] An example of the learning method will be described with reference to FIG. 2 etc. First, the imaging unit 21 captures a moving image. For example, a moving image including a human face is captured (step S01). Here, a moving image refers to a collection of images of two or more frames. Although details will be described later, learning data is created based on the moving image captured by the imaging unit 21, and the calculation unit 23 performs learning. Therefore, for example, when the imaging unit 21 captures a moving image including a human face, it is preferable that the imaging unit 21 captures the moving image of a large number of people of different genders, races, physiques, etc.

[0037] Note that image processing may be performed on the moving image captured by the imaging unit 21. For example, noise removal, grayscaling, normalization, contrast adjustment, etc. may be performed. Furthermore, binarization, etc. may be performed on the images included in the moving image. By performing such processing, the subsequent steps can be performed with high accuracy. For example, the detection of the first object performed in step S02 described later can be performed with high accuracy.

[0038] Next, the calculation unit 23 detects a first object from each captured image. For example, if a moving image of a face is captured in step S01, the first object may be an eye (step S02). The first object may be detected by, for example, a cascade classifier. For example, the first object may be detected by Haar Cascades. Note that if the first object is an eye and both eyes are included in one image, only one eye may be detected.

[0039] Then, the calculation unit 23 detects a second object from each of the detected first objects. For example, if the first object is an eye, the second object can be a pupil (step S03). For example, the pupil can be detected from the eye by circular extraction. The method of detecting the pupil from the eye will be described in detail later.

[0040] Here, the pupil is a hole surrounded by the iris and can be called the "black part of the eye." The pupil has the function of adjusting the amount of light projected onto the retina. The iris is, for example, a thin membrane between the cornea and the crystalline lens, and can be considered the colored part of the eye.

[0041] Next, the calculation unit 23 calculates the size of each of the detected second objects (step S04). For example, if the second object is detected by circular extraction, the radius or diameter of the second object can be regarded as the size of the second object. Furthermore, if the shape of the second object is extracted as an ellipse, the length of the major axis and the length of the minor axis can be regarded as the size of the second object. Furthermore, the area of ​​the second object can be regarded as the size of the second object.

[0042] Thereafter, the calculation unit 23 performs learning using the size of the second object, and obtains the learning result (step S05). Specifically, the learning result is obtained based on the change over time in the size of the second object. The learning can be performed using, for example, a neural network. In this case, the learning result can be a weighting coefficient or the like as described above. The learning method will be described in detail later.

[0043] Next, the information processing device 20 supplies the learning result to the information processing device 30 (step S06). Specifically, the learning result acquired by the calculation unit 23 is transmitted to the communication unit 26 via the transmission path 27, and then supplied from the communication unit 26 to the communication unit 36. The learning result supplied to the communication unit 36 ​​can be stored in the auxiliary storage unit 35. The learning result may also be stored in the auxiliary storage unit 25.

[0044] Next, an example of an inference method based on a learning result acquired by the method shown in Fig. 2 or the like will be described with reference to Fig. 3 etc. First, the imaging unit 31 captures a video. For example, a video including the face of a user of an information terminal in which the information processing device 30 is provided is captured (step S11). Note that, in step S01 shown in Fig. 2, if image processing is performed on the video captured by the imaging unit 21, it is preferable to perform similar image processing on the video captured by the imaging unit 31, since this allows for accurate inference.

[0045] Next, the calculation unit 33 detects a first object from each image included in the captured video. For example, if a video of a face is captured in step S11, the first object may be an eye (step S12). The first object may be detected by a method similar to the detection method used in step S02 shown in FIG. 2.

[0046] Then, the calculation unit 33 detects a second object from each of the detected first objects. For example, if the first object is an eye, the second object can be a pupil (step S13). The second object can be detected by a method similar to the detection method used in step S03 shown in FIG. 2.

[0047] Next, the calculation unit 33 calculates the size of each of the detected second objects (step S14). The method of calculating the size can be the same as the method used in step S04 shown in FIG.

[0048] Thereafter, the calculation unit 33 to which the learning result acquired by the calculation unit 23 in step S05 shown in Fig. 2 is input performs inference based on the change over time in the size of the second object. For example, if the video captured by the imaging unit 31 includes the face of a user of the information terminal in which the information processing device 30 is provided, and the second object is the pupil of the user's eye, the calculation unit 33 can estimate the user's fatigue, drowsiness, etc. (step S15). The inference method will be described in detail later.

[0049] The size of the pupil changes not only depending on the presence or absence of fatigue and drowsiness, but also depending on, for example, the brightness of the environment. Therefore, in step S01 shown in Fig. 2, it is preferable to capture multiple videos of the face of the same person with different environmental brightness. This makes it possible to accurately estimate the fatigue, drowsiness, etc. of the user of the information terminal in which the information processing device 30 is installed, regardless of, for example, the brightness of the environment.

[0050] In one aspect of the present invention, the information processing device 30 having the function of estimating fatigue, drowsiness, etc. as described above is provided in an information terminal such as a smartphone, tablet, personal computer, etc. This makes it possible to detect the fatigue, drowsiness, etc. of a user of the information terminal in real time without using a dedicated device.

[0051] [An example of pupil detection method] Next, a description will be given of an example of a method for detecting a pupil, which is performed in steps S03 and S13. Fig. 4 is a flowchart showing an example of the method for detecting a pupil.

[0052] First, the calculation unit acquires an image 41 including the detected eye (step S31). FIG. 5A is a schematic diagram for explaining step S31. As shown in FIG. 5A, the calculation unit acquires an image 41 including the eye detected from the image captured by the imaging unit. Specifically, in step S03, the calculation unit acquires an image including the eye detected by the calculation unit 23 in step S02 as image 41. In addition, in step S13, the calculation unit acquires an image including the eye detected by the calculation unit 33 in step S12 as image 41. Note that if the image captured by the imaging unit is a color image, the calculation unit may convert image 41 to grayscale after acquiring image 41.

[0053] Next, the calculation unit performs an expansion process on the image 41 to obtain an image 42, and then performs an erosion process to obtain an image 43 (step S32). That is, the image 41 is subjected to a closing process to obtain an image 43. Fig. 5B is a schematic diagram for explaining the expansion process and the erosion process.

[0054] After that, the calculation unit subtracts image 43 from image 41 to obtain image 44 (step S33). That is, image 44 is an image represented by the difference between image 41 and image 43. In step S33, the calculation unit performs a Black-hat transform using image 41 and image 43 to obtain image 44.

[0055] Next, the calculation unit adds together the image 41 acquired in step S31 and the image 44 acquired in step S33 to acquire an image 45 (step S34). Note that if the image 41 is converted to grayscale in step S31, the image 41 after conversion to grayscale can be added together with the image 44 in step S34.

[0056] It is to be noted that all or part of the processes shown in steps S32 to S34 may not be performed. Also, processes other than the processes shown in steps S32 to S34 may be performed.

[0057] Thereafter, the calculation unit performs image processing on the image 45 to obtain an image 46 (step S35). For example, the calculation unit performs processes such as noise removal and smoothing on the image 45. In addition, it performs processes such as edge detection and binarization. Specifically, for example, after performing noise removal on the image 45 using a median filter and smoothing using a Gaussian filter, it performs edge detection using the Canny method and binarization. Note that the noise removal may be performed using, for example, a moving average filter. Also, the smoothing may be performed using, for example, a moving average filter or a median filter. Furthermore, the edge detection may be performed using, for example, a Laplacian filter.

[0058] Next, the calculation unit detects the iris 47 from the image 46. For example, the iris 47 can be detected by using a Hough transform. When the Hough transform is used, the iris 47 can be detected as, for example, a circle. Alternatively, the iris 47 can be detected as, for example, an ellipse. Note that the iris 47 may be detected by using a generalized Hough transform.

[0059] Then, the calculation unit acquires an image 49 including the detected iris 47 (step S36). For example, the image 49 is extracted from the image 46 based on the coordinates of the detected iris 47 in the image 46.

[0060] 5C is a schematic diagram for explaining step S36. For example, as shown in FIG. 5C, image 49 can be a rectangle whose four sides are in contact with iris 47. For example, when iris 47 is detected in a circular shape, image 49 can be a square whose four sides are in contact with iris 47. Note that each side of image 49 does not have to be in contact with iris 47. For example, an image of a predetermined number of pixels centered on iris 47 may be image 49.

[0061] Next, the calculation unit detects the pupil 48 from the image 49 (step S37). For example, the pupil 48 is detected from the image 49 by calculation using a neural network.

[0062] Step S37 is performed using a generator that has been trained in advance. Here, the generator is a program that performs calculations by machine learning and has a function of outputting data corresponding to input data. Specifically, the generator can perform inference on the data input to the generator by learning.

[0063] FIG. 6A is a schematic diagram for explaining the learning. Here, the generator that performs learning is the generator 50. When a neural network is used as the generator 50, the generator 50 can use a Convolutional Neural Network (CNN). In particular, it is preferable to use a U-net as a type of CNN. In the U-net, an input image is downsampled by convolution, and then upsampled by deconvolution using the feature amount obtained by downsampling. It can be said that the calculation unit 23 and the calculation unit 33 have the function of the generator 50.

[0064] The generator 50 can learn by supervised learning using data 51 and data 52. The data 51 can be a collection of images 59. The images 59 include an iris 57 and a pupil 58. The images 59 can be acquired by the information processing device 20 in a manner similar to steps S01 and S02 shown in FIG. 2 and steps S31 to S36 shown in FIG. 4. Note that in step S01, the imaging unit 21 captures a moving image of a face, but when acquiring the image 59, it is not necessary to capture a moving image. For example, the imaging unit 21 may capture one image (one frame) per person.

[0065] The data 52 is data indicating the coordinates of the pupil 58 included in the image 59. Specifically, the image may be a binary image in which the color of the pupil 58 is made different from the color of the other parts. The data 52 may be acquired, for example, by painting over the pupil 58 included in the image 59. Alternatively, the data 52 may be acquired by acquiring an image including an eye in a manner similar to step S31 shown in Fig. 4, and then painting over the pupil 58 included in the image including the eye.

[0066] The generator 50 learns so that when data 51 is input to the generator 50, the output data approaches data 52. That is, the generator 50 learns using the data 52 as correct answer data. As the generator 50 learns, it generates a learning result 53. When a neural network is used as the generator 50, the learning result 53 can be a weighting coefficient or the like.

[0067] Learning of the generator 50, that is, generation of the learning result 53, can be performed by, for example, the calculation unit 23 of the information processing device 20. By supplying the learning result 53 from the information processing device 20 to the information processing device 30, the calculation unit 33 can also perform inference similar to that of the calculation unit 23. The learning result 53 generated by the calculation unit 23 can be stored in, for example, the auxiliary storage unit 25. In addition, the learning result 53 generated by the calculation unit 23 and supplied to the information processing device 30 can be stored in, for example, the auxiliary storage unit 35.

[0068] This completes the learning of the generator 50.

[0069] Fig. 6B is a schematic diagram illustrating step S37. That is, Fig. 6B is a schematic diagram illustrating detection of pupil 48 from image 49.

[0070] 6B, in step S37, the image 49 acquired by the calculation unit in step S36 is input to the generator 50 into which the learning result 53 has been read. This enables the generator 50 to perform inference on the image 49 and output data indicating the coordinates of the pupil 48. For example, the generator 50 can output a binary image in which the color of the pupil 48 is made different from the color of other parts.

[0071] By the above method, the pupils can be detected in step S03 or step S13 from among the eyes detected in step S02 or step S12.

[0072] By detecting the pupils using calculations based on machine learning, it is possible to detect the pupils in a shorter time than by visual inspection, for example. Furthermore, even if the surrounding scenery is reflected in the pupils, the pupils can be detected with high accuracy.

[0073] 6A and 6B. For example, image 49 may be a color image, and after converting color image 49 to grayscale, edge detection of pupil 48 may be performed. Then, after edge detection, pupil 48 may be detected.

[0074] The grayscale conversion of the image 49 can be performed using, for example, partial least squares (PLS) regression. By grayscaling the image 49, the difference between the luminance of the pupil 48 and the luminance of the iris 47 can be increased. This makes it possible to emphasize the boundary between the pupil 48 and the iris 47, thereby enabling accurate edge detection of the pupil 48. Therefore, the pupil 48 can be detected with high accuracy.

[0075] The edge detection of the pupil 48 can be performed, for example, by the Canny method or a Laplacian filter. Furthermore, the detection of the pupil 48 after the edge detection can be performed, for example, by using a Hough transform. When the Hough transform is used, for example, the pupil 48 can be detected as a circle. Alternatively, for example, the pupil 48 can be detected as an ellipse. The pupil 48 may be detected using a generalized Hough transform.

[0076] When detecting not only the iris but also the pupil in step S03 or step S13, imaging can be performed using infrared light in step S01 or step S11. The iris reflects infrared light. On the other hand, the pupil does not reflect infrared light. Therefore, by performing imaging using infrared light in step S01 or step S11, it is possible to clearly distinguish the iris from the pupil. Therefore, it is possible to detect the pupil with high accuracy.

[0077] [Example of a method for estimating fatigue, drowsiness, etc._1] Next, an example of a method for estimating fatigue, drowsiness, etc. of a user of an information terminal provided with the information processing device 30 by calculation using machine learning will be described. Specifically, an example of a learning method using the size of the pupil, which is performed in step S05, will be described. Also, an example of a method for estimating fatigue, drowsiness, etc. by inference based on the learning result, which is performed in step S15, will be described. Note that in the following description, the second object is assumed to be the pupil.

[0078] FIG. 7A1 is a schematic diagram for explaining step S05. In step S05, learning is performed by the generator 60, which is a program that performs calculations by machine learning. The generator 60 can use a neural network. Although details will be described later, time-series data, such as changes in pupil size over time, is input to the generator 60. Therefore, when a neural network is used as the generator 60, it is preferable to use a recurrent neural network (RNN) as the generator 60. Alternatively, it is preferable to use a long short-term memory (LSTM) as the generator 60. Alternatively, it is preferable to use a gated recurrent unit (GRU).

[0079] The generator 60 can learn using data 61 and data 62. Data 61 is data acquired in step S04, and can be the change in pupil size over time. As described above, for example, if the pupil is detected by circular extraction, the radius or diameter of the pupil can be the size of the pupil. Also, if the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be the size of the pupil. Also, the area of ​​the pupil can be the size of the pupil. In FIG. 7A1, the change in pupil size over time from time 1 to n-1 (n is an integer of 3 or more) is used as data 61.

[0080] The data 61 may be a time-dependent change in the ratio between the size of the pupil and the size of the iris. In this case, it is preferable that the iris and the pupil are extracted in the same shape. For example, when the iris is extracted in a circular shape, it is preferable that the pupil is also extracted in a circular shape. Also, when the iris is extracted in an elliptical shape, it is preferable that the pupil is also extracted in an elliptical shape. By using the time-dependent change in the ratio between the size of the pupil and the size of the iris as the data 61, for example, when detecting the pupil by the method shown in step S37, the resolutions of the images 49 including the iris 47 and the pupil 48 can be made different from each other. For example, the resolution of the image 49 including the iris 47 and the pupil 48 of the first person can be made different from the resolution of the image 49 including the iris 47 and the pupil 48 of the second person.

[0081] Data 62 is the pupil size at time n, that is, the pupil size at a time after the time of measurement of the pupil size included in data 61. If data 61 is the change over time in the ratio of pupil size to iris size, data 62 is also the ratio of pupil size to iris size.

[0082] FIG. 7A2 is a diagram showing an example of the relationship between pupil diameter and time. In FIG. 7A2, black circles indicate actual measurement values ​​of pupil diameter. In other diagrams, actual measurement points may also be indicated by black circles. As shown in FIG. 7A2, data 62 may be the pupil size at a time after the time at which the pupil size included in data 61 was measured. For example, data 62 may be the pupil size at a time next to the last time at which the pupil size included in data 61 was measured.

[0083] Here, if generator 60 is provided with a function for estimating the presence or absence of fatigue, the change over time in pupil size of a person who is tired is not included in data 61 and data 62. That is, data 61 is the change over time in pupil size of a person who is not tired, and data 62 is the pupil size of a person who is not tired. Also, if generator 60 is provided with a function for estimating the presence or absence of drowsiness, the change over time in pupil size of a person who is sleepy is not included in data 61 and data 62. That is, data 61 is the change over time in pupil size of a person who is not sleepy, and data 62 is the pupil size of a person who is not sleepy.

[0084] The generator 60 learns so that when data 61 is input to the generator 60, the output data approaches data 62. That is, the generator 60 learns using the data 62 as correct answer data. As the generator 60 learns, it generates a learning result 63. When a neural network is used as the generator 60, the learning result 63 can be a weighting coefficient or the like.

[0085] 7B1 and 7B2 are schematic diagrams for explaining step S15, and are diagrams showing an example of a method for estimating fatigue, drowsiness, and the like of a user of an information terminal in which an information processing device 30 is provided, using a generator 60. In step S15, first, as shown in FIG. 7B1, data 64 indicating a change in pupil size over time acquired in step S14 is input to the generator 60 into which a learning result 63 is read. For example, when the change in pupil size over time from time 1 to n-1 is used as input data during learning of the generator 60 in step S05, the change in pupil size over time from time 1 to n-1 is also used as input data during inference in step S15. That is, the data 64 is set as the change in pupil size over time from time 1 to n-1 of the user of the information terminal in which the information processing device 30 is provided. As a result, the generator 60 performs inference on the data 64 and outputs data 65. Furthermore, data 65, which is inferred data at time n, may be used to infer data at time n+1 using data at times 2 to n as input data.

[0086] In addition, when data 61 is the change over time in the ratio between the size of the pupil and the size of the iris, data 64 is also the change over time in the ratio between the size of the pupil and the size of the iris. By using data 64 as the change over time in the ratio between the size of the pupil and the size of the iris, for example, when detecting the pupil by the method shown in step S37, the resolutions of images 49 including the iris 47 and the pupil 48 can be made different from each other. For example, the resolution of image 49 acquired by the calculation unit to calculate the ratio between the size of the pupil 48 and the size of the iris 47 at time 1 can be made different from the resolution of image 49 acquired by the calculation unit to calculate the ratio between the size of the pupil 48 and the size of the iris 47 at time n-1.

[0087] Data 65 is an estimate of the pupil size at a time after the time at which the pupil size included in data 64 was measured, calculated by performing inference on data 64 based on learning result 63. For example, if data 64 represents the change over time in pupil size from time 1 to time n-1, data 65 can represent the pupil size at time n. In FIG. 7B1, the actual measured values ​​of pupil size at times 1 to n-1 are represented as x1 to x n-1 The estimated pupil size at time n is expressed as x n (E). If data 64 is the change over time in the ratio of the pupil size to the iris size, data 65 is the ratio of the pupil size to the iris size.

[0088] Next, as shown in FIG. 7B2, data 66 representing the actual measurement value of the pupil size at time n is compared with data 65 output from the generator 60. That is, the actual measurement value of the pupil size at time n is compared with the estimated value. The presence or absence of fatigue, drowsiness, etc. is estimated based on the comparison result. For example, when the generator 60 has a function of estimating the presence or absence of fatigue, the learning of the generator 60 is performed using, for example, the change over time in the pupil size of a person without fatigue as input data. Therefore, when the user of the information terminal in which the information processing device 30 is provided is in a state of no fatigue, the data 65 is close to the data 66. That is, the difference between the data 65 and the data 66 is small. On the other hand, when the user of the information terminal in which the information processing device 30 is provided is in a state of fatigue, the difference between the data 66 and the data 65 is larger than when the user of the information terminal in which the information processing device 30 is provided is in a state of no fatigue. From the above, by comparing the data 66 and the data 65, the presence or absence of fatigue of the user of the information terminal in which the information processing device 30 is provided can be estimated. The same applies to the case of estimating the presence or absence of drowsiness. If data 65 is an estimated value of the ratio between the size of the pupil and the size of the iris, data 66 is an actual measured value of the ratio between the size of the pupil and the size of the iris.

[0089] The function of the generator 60 can be provided to both the calculation unit 23 and the calculation unit 33. In this case, the calculation unit 23 of the information processing device 20 performs learning of the generator 60 to generate the learning result 63, and the learning result 63 can be supplied from the information processing device 20 to the information processing device 30. As described above, even if the calculation unit 33 provided in the information processing device 30 does not perform learning, inference based on the learning result by the calculation unit 23 provided in the information processing device 20 can be performed on the data input to the calculation unit 33. Therefore, the calculation processing capacity of the calculation unit 33 can be made lower than that of the calculation unit 23. The learning result 63 can be stored in the auxiliary storage unit 25 and the auxiliary storage unit 35.

[0090] [Example of a method for estimating fatigue, drowsiness, etc._2] 8A and 8B are schematic diagrams for explaining step S05, and are an example of a learning method of the generator different from the above-mentioned method. Specifically, FIG. 8A is a diagram showing an example of a method of creating data to be input to the generator as learning data, and FIG. 8B is a diagram showing an example of a learning method of the generator 80. The generator 80 is a program that performs calculations by machine learning. For example, a neural network can be used as the generator 80.

[0091] Data 81 shown in FIG. 8A is data acquired in step S04, and can be the change in pupil size over time. As described above, for example, if the pupil is detected by circular extraction, the radius or diameter of the pupil can be the pupil size. Also, if the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be the pupil size. Also, the pupil size can be the pupil area. Note that, similar to the learning method shown in FIG. 7A1, data 81 can be the change in the ratio of the pupil size to the iris size over time.

[0092] Here, data 81 is subjected to a Fourier transform to generate data 82. As shown in Fig. 8A, the change in pupil diameter over time can be converted into the frequency characteristics of the pupil diameter by the Fourier transform. If data 81 is the change in the ratio of the pupil size to the iris size over time, data 82 can be the frequency characteristics of the ratio of the pupil size to the iris size.

[0093] The learning of the generator 80 can be performed using data 82 and data 83 as shown in FIG. 8B. The data 82 represents the frequency characteristic of the pupil diameter as described above. The data 83 can be a label indicating the presence or absence of fatigue. For example, the data 83 includes both the frequency characteristic of the pupil size of a person with fatigue and the frequency characteristic of the pupil size of a person without fatigue. Then, the frequency characteristic of the pupil size of a person with fatigue is associated with a label of "with fatigue", and the frequency characteristic of the pupil size of a person without fatigue is associated with a label of "without fatigue". The data 83 may also be a label indicating the presence or absence of drowsiness.

[0094] The generator 80 learns so that when data 82 is input to the generator 80, the output data approaches data 83. That is, the generator 80 learns using data 83 as correct answer data. As the generator 80 learns, it generates a learning result 84. When a neural network is used as the generator 80, the learning result 84 can be a weighting coefficient or the like.

[0095] By performing a Fourier transform on the change in pupil size over time, the data input to the generator 80 can be made to be data that is not time series data. This allows the generator 80 to perform learning and inference without using an RNN as the generator 80.

[0096] 9A and 9B are schematic diagrams for explaining step S15, and are diagrams showing an example of a method for estimating fatigue, drowsiness, and the like of a user of an information terminal in which an information processing device 30 is provided, using a generator 80.

[0097] Data 85 shown in FIG. 9A is data acquired in step S04, and can be the change in pupil size over time. As described above, for example, if the pupil is detected by circular extraction, the radius or diameter of the pupil can be the pupil size. If the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be the pupil size. The pupil size can be the pupil area. Note that, if data 81 shown in FIG. 8A is the change in the ratio of the pupil size to the iris size over time, data 85 is also the change in the ratio of the pupil size to the iris size over time.

[0098] Here, data 85 is subjected to a Fourier transform to generate data 86. As shown in Fig. 9A, the change in pupil diameter over time can be converted into the frequency characteristics of the pupil diameter by the Fourier transform. If data 85 is the change in the ratio of the pupil size to the iris size over time, data 86 can be the frequency characteristics of the ratio of the pupil size to the iris size.

[0099] Then, as shown in Fig. 9B, data 86 after the Fourier transform is input to the generator 80. This enables the generator 80 to perform inference on the data 86 and output data 87 indicating the presence or absence of fatigue. Note that if the data 87 shown in Fig. 9B is a label indicating the presence or absence of drowsiness, the data 87 output by the generator 80 can be data indicating the presence or absence of drowsiness.

[0100] The function of the generator 80 can be provided to both the calculation unit 23 and the calculation unit 33, similarly to the function of the generator 60 and the function of the generator 70. This allows the calculation unit 33 to have a lower calculation processing capability than the calculation unit 23.

[0101] [Example of a method for estimating fatigue, drowsiness, etc._3] FIG. 10A is a schematic diagram for explaining step S05, and is an example of a generator learning method different from the above-mentioned method. In FIG. 10A, learning of the generator 70 is performed. The generator 70 is a program that performs calculations by machine learning. For example, a neural network, for example an autoencoder, can be used as the generator 70.

[0102] When generator 70 performs learning, data 71 is input to generator 70. Data 71 is data acquired in step S04, and can be the change in pupil size over time. Here, if generator 70 is provided with a function to estimate the presence or absence of fatigue, data 71 does not include the change in pupil size over time of a person who is tired. In other words, data 71 is the change in pupil size over time of a person who is not tired. Also, if generator 70 is provided with a function to estimate the presence or absence of drowsiness, data 71 does not include the change in pupil size over time of a person who is sleepy. In other words, data 71 is the change in pupil size over time of a person who is not sleepy.

[0103] As described above, for example, if the pupil is detected by circular extraction, the radius or diameter of the pupil can be regarded as the size of the pupil. Also, if the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be regarded as the size of the pupil. Also, the area of ​​the pupil can be regarded as the size of the pupil.

[0104] As in the learning method shown in Fig. 7A1, data 71 can be the change over time in the ratio of pupil size to iris size. As in the case shown in Fig. 8A, data 71 can be obtained by performing a Fourier transform on the change over time in pupil size.

[0105] The generator 70 learns so that when data 71 is input to the generator 70, the output data 72 approaches the input data 71. In other words, the generator 70 learns so that the data 71 and the data 72 become equal. As the generator 70 learns, it generates a learning result 73. When a neural network is used as the generator 70, the learning result 73 can be a weighting coefficient or the like.

[0106] 10B1 and 10B2 are schematic diagrams for explaining step S15, and are diagrams showing an example of a method for estimating fatigue, drowsiness, and the like of a user of an information terminal in which an information processing device 30 is provided, using a generator 70. In step S15, first, as shown in FIG. 10B1, data 74 indicating changes in pupil size over time acquired in step S14 is input to the generator 70 into which the learning result 73 has been read. As a result, the generator 70 performs inference on the data 74 and outputs data 75.

[0107] If data 71 represents the change over time in the ratio between the size of the pupil and the size of the iris, then data 74 also represents the change over time in the ratio between the size of the pupil and the size of the iris. If data after Fourier transform is used as data 71, then data after Fourier transform is also used for data 74. For example, if data 71 is the Fourier transform of the change over time in the size of the pupil, then data 74 is also the Fourier transform of the change over time in the size of the pupil.

[0108] Next, as shown in FIG. 10B2, data 74, which is data input to the generator 70, is compared with data 75, which is data output from the generator 70. The presence or absence of fatigue, drowsiness, etc. is estimated based on the comparison result. For example, when the generator 70 has a function of estimating the presence or absence of fatigue, the learning of the generator 70 is performed using, for example, the change over time in the pupil size of a person without fatigue. Therefore, when the user of the information terminal in which the information processing device 30 is provided is in a state without fatigue, data 75, which is output data from the generator 70, is close to data 74, which is input data to the generator 70. That is, the difference between the data 74 and the data 75 is small. On the other hand, when the user of the information terminal in which the information processing device 30 is provided is in a state with fatigue, the difference between the data 74 and the data 75 is larger than when the user of the information terminal in which the information processing device 30 is provided is in a state without fatigue. As described above, by comparing the data 74 and the data 75, the presence or absence of fatigue of the user of the information terminal in which the information processing device 30 is provided can be estimated. The same applies to the case of estimating the presence or absence of drowsiness.

[0109] The function of the generator 70 can be provided to both the calculation unit 23 and the calculation unit 33, similarly to the function of the generator 60. This allows the calculation unit 33 to have a lower calculation processing capability than the calculation unit 23.

[0110] [Example of a method for estimating fatigue, drowsiness, etc._4] The learning performed in step S05 and the inference based on the learning result performed in step S15 may be performed using a Generative Adversarial Network (GAN). For example, AnoGAN (Anormaly GAN) may be used. FIG. 11 is a diagram for explaining AnoGAN that can perform the above learning and inference.

[0111] 11 includes a generator 91 and a discriminator 92. The generator 91 and the discriminator 92 can be configured by a neural network.

[0112] Data 93, which is time-series data representing a change over time in the size of the pupil of a person who is not fatigued, sleepy, or the like, obtained by imaging, is input to the discriminator 92. Alternatively, data 95, which is time-series data generated by a generator 91 to which data 94 is input, is input to the discriminator 92. The discriminator 92 has a function of determining whether the input data is data 93 obtained by imaging or data 95 generated by the generator 91 (also referred to as authenticity determination). Note that the data 93 may be data obtained by Fourier transforming the time-series data representing a change over time in the size of the pupil of a person who is not fatigued, sleepy, or the like, obtained by imaging.

[0113] The determination result is output as data 96. The data 96 can be, for example, a continuous value between 0 and 1. In this case, for example, after the end of learning, the discriminator 92 may output a value close to 1 as the data 96 if the input data is data 93 acquired by imaging, and may output a value close to 0 as the data 96 if the input data is data 95 generated by the generator 91.

[0114] The data 94 is a multidimensional random number (also called a latent variable). Here, the latent variable represented by the data 94 is assumed to be a latent variable z. The generator 91 has a function of generating data based on such data 94 that is as similar as possible to data representing the change over time in the pupil size of a person who is not fatigued, sleepy, or the like.

[0115] The learning is performed by alternating between the learning of the discriminator 92 and the learning of the generator 91. That is, when the discriminator 92 is being learned, the weight coefficients of the neural network constituting the generator 91 are fixed. Also, when the generator 91 is being learned, the weight coefficients of the neural network constituting the discriminator 92 are fixed.

[0116] When the discriminator 92 is learning, data 93 acquired by imaging or data 95 generated by the generator 91 is input to the discriminator 92. A correct label is assigned to the data input to the discriminator 92. A correct label can be determined for data 96 output by the discriminator 92 as follows. For example, when data 93 is input to the discriminator 92, the correct label is set to "1", and when data 95 is input to the discriminator 92, the correct label is set to "0". By learning in the above manner, the discriminator 92 becomes able to determine authenticity.

[0117] When the generator 91 is trained, data 94 representing a latent variable z is input to the generator 91. Then, the generator 91 generates data 95 based on the input data 94. The correct label of data 96 is set to "1". Then, the generator 91 trains so that the value of data 96 output from the discriminator 92 becomes "1". As the training of the generator 91 progresses, the generator 91 becomes able to generate data similar to data 93 acquired by imaging as data 95.

[0118] When the learning of the generator 91 is completed, the generator 91 will be able to generate data 95 similar to the data 93 acquired by imaging, regardless of the latent variable z input as data 94.

[0119] Next, the operation during inference will be described.

[0120] First, assume that data representing the change over time in pupil size of a person who is not fatigued, drowsy, etc. is obtained by photography. At this time, the space of latent variables is searched, and a latent variable z1 that generates data most similar to the data representing the change over time in pupil size of a person who is not fatigued, drowsy, etc. is found by gradient descent or the like. The generator 91 has a function of generating data that is extremely similar to the data representing the change over time in pupil size of a person who is not fatigued, drowsy, etc. through learning. Therefore, the data generated by the generator from the latent variable z1 and the data representing the change over time in pupil size of a person who is not fatigued, drowsy, etc. obtained by photography are extremely similar.

[0121] Next, assume that data representing the change over time in the size of the pupil of a person who is fatigued, sleepy, etc. is acquired by imaging. At this time, the space of latent variables is searched, and a latent variable z2 that generates data that is closest to the data representing the change over time in the size of the pupil of a person who is fatigued, sleepy, etc. is found by gradient descent or the like. The generator 91 has the ability to generate data that is very close to the data representing the change over time in the size of the pupil of a person who is not fatigued, sleepy, etc., by learning, but does not have the ability to generate data similar to the data representing the change over time in the size of the pupil of a person who is fatigued, sleepy, etc. Therefore, the data generated by the generator from the latent variable z2 and the data representing the change over time in the size of the pupil of a person who is fatigued, sleepy, etc. obtained by imaging are not very similar data. As described above, the presence or absence of fatigue, sleepiness, etc. can be estimated by the generator 91.

[0122] As described above, as shown in FIG. 7 to FIG. 11, the information processing device 30 estimates the fatigue, drowsiness, etc. of the user of the information terminal in which the information processing device 30 is provided by calculation using machine learning. By using machine learning, it is possible to accurately estimate fatigue, drowsiness, etc., without manually setting, for example, the change over time of the feature amount when estimating that there is fatigue and the change over time of the feature amount when estimating that there is no fatigue. Specifically, it is possible to accurately estimate fatigue, drowsiness, etc., without manually setting, for example, the change over time of the pupil size when estimating that there is fatigue and the change over time of the pupil size when estimating that there is no fatigue. In addition, since it is possible to estimate fatigue, drowsiness, etc., without manually setting, for example, the change over time of the feature amount when estimating that there is fatigue and the change over time of the feature amount when estimating that there is no fatigue, it is possible to estimate fatigue, drowsiness, etc., in a simple manner.

[0123] When the information processing device 30 estimates that the user is fatigued, sleepy, or the like, an alarm indicating that fatigue, sleepiness, or the like is occurring can be displayed, for example, on the display unit of the information terminal in which the information processing device 30 is installed. This makes it possible to prompt the user of the information terminal to stop using the information terminal as soon as possible. Alternatively, the power supply of the information terminal in which the information processing device 30 is installed can be turned off. This makes it possible to prevent the occurrence of health damage caused by the user of the information terminal continuing to use the information terminal despite the occurrence of fatigue, sleepiness, or the like. [Explanation of symbols]

[0124] 10: Information processing system, 20: Information processing device, 21: Imaging unit, 22: Display unit, 23: Calculation unit, 24: Main memory unit, 25: Auxiliary memory unit, 26: Communication unit, 27: Transmission path, 30: Information processing device, 31: Imaging unit, 32: Display unit, 33: Calculation unit, 34: Main memory unit, 35: Auxiliary memory unit, 36: Communication unit, 37: Transmission path, 41: Image, 42: Image, 43: Image, 44: Image, 45: Image, 46: Image, 47: Iris, 48: Pupil, 49: Image, 50: Generator, 51: Data, 52: Data, 53: Learning result, 57 :Iris,58:Pupil,59:Image,60:Generator,61:Data,62:Data,63:Learning result,64:Data,65:Data,66:Data,70:Generator,71:Data,72:Data,73:Learning result,74:Data,75:Data,80:Generator,81:Data,82:Data,83:Data,84:Learning result,85:Data,86:Data,87:Data,91:Generator,92:Discriminator,93:Data,94:Data,95:Data,96:Data

Claims

1. The imaging unit and the calculation unit having a function of performing calculations by machine learning are included. The imaging unit has a function of acquiring a moving image which is a collection of images of two or more frames, the calculation unit has a function of detecting a first object from each of two or more of the images included in the moving image, the calculation unit has a function of detecting a second object from each of the detected first objects; the calculation unit has a function of calculating a size of each of the detected second objects, The calculation unit is an information processing device having a function of performing learning using a change in size of the second object over time.

2. In claim 1, The information processing device performs the machine learning using a neural network.

3. In claim 1 or 2, An information processing device, wherein the video includes a face, the first object is an eye, and the second object is a pupil.

4. An information processing device having a function of performing inference based on a learning result obtained by performing learning using a change over time in size of a first object shown in two or more first images included in a first moving image, the information processing device has a function of acquiring a second moving image; the information processing device has a function of detecting a second object from each of two or more second images included in the second moving image; the information processing device has a function of detecting a third object from each of the detected second objects; the information processing device has a function of calculating a size of each of the detected third objects; The information processing device has a function of making an inference based on the learning result regarding a change over time in the size of the third object.

5. In claim 4, The learning and the inference are performed by a neural network; The information processing device, wherein the learning result includes a weighting coefficient.

6. In claim 4 or 5, An information processing device, wherein the first video includes a first face, the second video includes a second face, the first and third objects are pupils, and the second object is an eye.

7. 7. The information processing device according to claim 6, further comprising a function for estimating fatigue of a person having the second face.

Citation Information

Patent Citations

  • Driver fatigue detecting system based on smart mobile phone

    CN104013414A

  • Fatigue driving monitoring device and method

    CN105354985A

  • Automobile fatigue-driving warning device

    CN105513278A

  • System, method and device for obtaining fatigue characteristic of user

    CN106137181A

  • Fatigue deciding method and observing device using same

    JP1997218375A

Cited By

  • Shape recognition method and shape recognition device

    CN121708080A