Information processing device

Abstract

The present invention provides an information processing device, an information processing system, and an information terminal that have a function to detect user fatigue, drowsiness, etc. [Solution] In an information processing system, the information processing device has a calculation unit that has the function of performing calculations by machine learning, and the calculation unit performs the following steps: step S01 to acquire a video which is a collection of two or more frames of images; step S02 to detect an eye, which is a first object, from each of two or more images included in the video; step S03 to detect a pupil, which is a second object, from each of the detected eyes; step S04 to calculate the size of each of the detected pupils; step S05 to perform learning using the change in pupil size over time; and step S06 to supply the learning results to another information processing device.

Description

Technical Field

[0001] One aspect of the present invention relates to an information processing apparatus. Another aspect of the present invention relates to an information processing system Another aspect of the present invention relates to an information processing method. Another aspect of the present invention relates to an information terminal.

Background Art

[0002] When an information terminal such as a smartphone or a tablet is used for a long time, the user may feel fatigue, drowsiness, etc. In particular, when the user stares at the screen of the information terminal for a long time, the user may feel eye fatigue. Patent Document 1 discloses an eye fatigue detection device and a detection method.

[0003] The pupil diameter changes depending on the presence or absence of fatigue, drowsiness, etc. For example, when there is fatigue or drowsiness, the pupil diameter becomes smaller than when there is no fatigue or drowsiness. Also, generally, the pupil diameter fluctuates periodically, but when there is fatigue or drowsiness, the fluctuation period of the pupil diameter becomes longer than when there is no fatigue or drowsiness.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] During the use of an information terminal such as a smartphone or a tablet, if the user's fatigue, drowsiness, etc. can be detected in real time, for example, the operation of the information terminal can be adjusted according to the presence or absence of the user's fatigue, drowsiness, etc. ​​​It is preferable because it can be changed. When detecting fatigue, drowsiness, etc. in real time, it can be used. It is preferable that the information terminal being used has a function to estimate the user's fatigue, drowsiness, etc. However, detecting eye fatigue using the method shown in Patent Document 1 requires a dedicated device. That is the case.

[0006] One aspect of the present invention is an information processing system that has a function to detect user fatigue, drowsiness, etc., in real time. One objective of this invention is to provide a device for processing. Alternatively, one aspect of this invention provides a device that accurately reduces user fatigue. One of the objectives is to provide an information processing device that has a function to estimate fatigue, drowsiness, etc. One aspect of the present invention relates to an information processing system that has the function of estimating the user's fatigue, drowsiness, etc., in a simple manner. One objective of this invention is to provide a device that reduces user fatigue in a short amount of time. One of the objectives is to provide an information processing device that has a function to estimate fatigue, drowsiness, etc.

[0007] One aspect of the present invention is an information processing system that has a function to detect user fatigue, drowsiness, etc., in real time. One of the objectives is to provide a system for precise analysis. Alternatively, one aspect of the present invention provides a system for precise analysis to the user. One of the objectives is to provide an information processing system that has the function of estimating fatigue, drowsiness, etc. Alternatively, one aspect of the present invention has a function for estimating the user's fatigue, drowsiness, etc., in a simple manner. One objective is to provide an information processing system that can be used for a short time. Alternatively, one aspect of the present invention is to provide an information processing system that can be used for a short time. The objective is to provide an information processing system that has a function to estimate the user's fatigue, drowsiness, etc. It shall be one of them.

[0008] Furthermore, the description of multiple problems does not preclude the existence of each other. One embodiment of the present invention is It is not necessary to solve all the problems exemplified. Furthermore, if there are problems other than those listed, this specification... It will become clear from the description of , and such problems can also be the problems of one aspect of the present invention. .

Means for Solving the Problems

[0009] One aspect of the present invention includes an imaging unit and an arithmetic unit having a function of performing arithmetic operations by machine learning. The imaging unit has a function of acquiring a moving image which is a set of two or more frames of images, and the arithmetic unit has a function of detecting a first object from each of two or more images included in the moving image. The arithmetic unit has a function of detecting a second object from each of the detected first objects. The arithmetic unit has a function of calculating the size of the second object for each of the detected second objects. The arithmetic unit has a function of performing machine learning using the change over time of the size of the second object. It is an information processing device.

[0010] Also, in the above aspect, the machine learning may be performed by a neural network.

[0011] Also, in the above aspect, the moving image may include a face, the first object may be an eye, and the second object may be a pupil.

[0012] Also, one aspect of the present invention is an information processing device having a function of making an inference based on a learning result obtained by performing learning using the change over time of the size of a first object shown in two or more first images included in a first moving image. The information processing device has a function of acquiring a second moving image. The information processing device has a function of detecting a second object from each of two or more second images included in the second moving image. The information processing device has a function of detecting a third object from each of the detected second objects. The information processing device performs for each of the detected third objects. has a function of detecting a second object from each of two or more second images included in the second moving image. The information processing device has a function of detecting a third object from each of the detected second objects. The information processing device performs and has a function of calculating the size of a third object, and the information processing apparatus is an information processing apparatus having a function of making an inference based on a learning result with respect to the temporal change of the size of the third object.

[0013] Also, in the above aspect, the learning and the inference are performed by a neural network, and the learning result may include weight coefficients.

[0014] Also, in the above aspect, the first video may include a first face, the second video may include a second face, the first and third objects may be pupils, and the second object may be an eye.

[0015] Also, in the above aspect, the information processing apparatus may have a function of estimating the fatigue of a person having the second face.

Advantages of the Invention

[0016] According to one aspect of the present invention, it is possible to provide an information processing apparatus having a function of detecting in real time the fatigue, drowsiness, etc. of a user. Or, according to one aspect of the present invention, it is possible to provide an information processing apparatus having a function of accurately estimating the fatigue, drowsiness, etc. of a user. Or, according to one aspect of the present invention, it is possible to provide an information processing apparatus having a function of estimating the fatigue, drowsiness, etc. of a user by a simple method. Or, according to one aspect of the present invention, it is possible to provide an information processing apparatus having a function of estimating the fatigue, drowsiness, etc. of a user in a short time. According to one aspect of the present invention, it is possible to provide an information processing system having a function of detecting in real time the fatigue, drowsiness, etc. of a user. Or, according to one aspect of the present invention, it is possible to provide an information processing system having a function of accurately estimating the fatigue, drowsiness, etc. of a user. Or

[0017] According to one aspect of the present invention, it is possible to provide an information processing system having a function of detecting in real time the fatigue, drowsiness, etc. of a user. Or, according to one aspect of the present invention, it is possible to provide an information processing system having a function of accurately estimating the fatigue, drowsiness, etc. of a user. Or ​ According to one aspect of the present invention, an information system has a function for estimating the user's fatigue, drowsiness, etc., in a simple manner. A report processing system can be provided. Alternatively, according to one aspect of the present invention, a user can be provided with information in a short time. This system can provide an information processing system that has the function of estimating fatigue, drowsiness, etc.

[0018] Furthermore, the description of multiple effects does not preclude the existence of other effects. The invention does not necessarily have to possess all of the effects exemplified. Furthermore, in one aspect of the present invention... For issues, effects, and novel features other than those mentioned above, please refer to the description and drawings in this specification. It will become clear naturally. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of an information processing system. [Figure 2] Figure 2 is a flowchart showing an example of how an information processing device operates. [Figure 3] Figure 3 is a flowchart showing an example of how an information processing device operates. [Figure 4] Figure 4 is a flowchart showing an example of how an information processing device operates. [Figure 5] Figures 5A, 5B, and 5C are schematic diagrams illustrating an example of how an information processing device operates. [Figure 6] Figures 6A and 6B are schematic diagrams illustrating an example of how an information processing device operates. [Figure 7] Figures 7A1 and 7A2, and 7B1 and 7B2 are schematic diagrams illustrating an example of how an information processing device operates. [Figure 8] Figures 8A and 8B are schematic diagrams illustrating an example of how an information processing device operates. [Figure 9] Figures 9A and 9B are schematic diagrams illustrating an example of how an information processing device operates. [Figure 10] Figures 10A, 10B1, and 10B2 are schematic diagrams illustrating an example of how an information processing device operates. [Figure 11] Figure 11 illustrates an AnoGAN that can be applied to one aspect of the present invention. [Modes for carrying out the invention]

[0020] Embodiments of the present invention will be described below. However, one aspect of the present invention is not limited to the following description. Without departing from the spirit and scope of the present invention, its form and details may be modified in various ways. Those skilled in the art will readily understand what is possible. Therefore, one aspect of the present invention is as follows: The description of the embodiment shown is not to be limited to the content described therein.

[0021] Furthermore, in the drawings attached to this specification, the components are classified according to their function and are shown as independent blocks. Although a block diagram is shown as an example, the actual components are not completely separated by function. This is difficult because one component can be involved in multiple functions, or one function can be involved in multiple components. It could potentially be achieved.

[0022] (Embodiment 1) In this embodiment, an information processing system according to one aspect of the present invention, and an information processing system using the present invention This document will explain the information processing method. An information processing system and information processing according to one aspect of the present invention. The method involves detecting fatigue, drowsiness, etc., in users of information terminals such as smartphones or tablets. It can determine this. In particular, it can detect eye strain in users of information terminals.

[0023] <Example of an information processing system configuration> Figure 1 shows an example configuration of an information processing system 10, which is an information processing system according to one aspect of the present invention. This is a block diagram. The information processing system 10 consists of an information processing device 20 and an information processing device 30 It has, and

[0024] The information processing device 20 includes an imaging unit 21, a display unit 22, a calculation unit 23, a main memory unit 24, and an auxiliary It has an auxiliary storage unit 25 and a communication unit 26. Data between components of the information processing device 20 Data and other information can be transmitted via the transmission line 27. Furthermore, the information processing device 30 can capture images. Image unit 31, display unit 32, calculation unit 33, main memory unit 34, auxiliary memory unit 35, communication unit 36 and, The transmission of data etc. between the components of the information processing device 30 is via a transmission line This can be done via 37.

[0025] The imaging unit 21 and imaging unit 31 have the function of taking images and acquiring imaging data. The unit 22 and the display unit 32 have the function of displaying images.

[0026] The calculation unit 23 and the calculation unit 33 have the function of performing calculations. The calculation unit 23, for example, Calculations are performed via the transmission line 27 from the image unit 21, main memory unit 24, auxiliary memory unit 25, or communication unit 26. The calculation unit 33 has the function of performing predetermined calculations on the data transmitted to unit 23. For example, the transmission line 37 from the imaging unit 31, the main memory unit 34, the auxiliary memory unit 35, or the communication unit 36 It has the function of performing predetermined calculations on the data transmitted to the calculation unit 33 via the arithmetic unit. Furthermore, the arithmetic unit 23 and the arithmetic unit 33 have the function of performing calculations using machine learning. For example, It has the function of performing calculations using a neural network. Calculation unit 23, and calculation unit 33 For example, CPU (Central Processing Unit) and GPU ( It may have a Graphics Processing Unit, etc.

[0027] The main memory units 24 and 34 have the function of storing data, programs, and the like. The arithmetic unit 23 reads the data and programs stored in the main memory unit 24 and performs It can perform calculations. For example, the calculation unit 23 reads from the main memory unit 24. By executing the program, predetermined operations are performed on the data read from the main memory unit 24. Processing can be executed. In addition, the arithmetic unit 33 can process the data stored in the main memory unit 34. It can also read programs and execute calculations. For example, the calculation unit 33 This is done by executing a program read from the main memory 34, It is possible to perform predetermined calculations on the entered data.

[0028] The main memory unit 24 and the main memory unit 34 move faster than the auxiliary memory unit 25 and the auxiliary memory unit 35. It is preferable to do so. The main memory unit 24 and the main memory unit 34 are, for example, DRAM (Dyna). mic Random Access Memory), SRAM (Static Ra It may have (ndom Access Memory), etc.

[0029] Auxiliary storage units 25 and 35 store data and programs, etc., in the main storage unit 24. And it has the function of storing data for a longer period than the main memory unit 34. Auxiliary memory unit 25 and auxiliary memory unit 3 5 is, for example, HDD (Hard Disk Drive), SSD (Solid State Drive). It can have a drive, etc. Also, auxiliary storage unit 25 and auxiliary storage unit 3 5 is ReRAM (Resistive Random Access Memory), Also known as resistive random-access memory (PRAM), PRAM (Phase change Random A ccess Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Ra DOM Access Memory (also known as magnetoresistive memory), or flash memory. It may have non-volatile memory such as memory.

[0030] The communication unit 26 transmits and receives data to and from devices located outside the information processing device 20. It has the function of performing the following. The communication unit 36 ​​communicates with devices etc. located outside the information processing device 30. It has the function of sending and receiving data, etc. For example, data from communication unit 26 to communication unit 36. By supplying the above, data etc. is supplied from the information processing device 20 to the information processing device 30. It is possible to do so. In addition, communication units 26 and 36 supply data etc. to the network. It can have functions to retrieve data from a network, and functions to acquire data from a network.

[0031] Here, if the arithmetic unit 23 and the arithmetic unit 33 have the function of performing calculations using machine learning, for example For example, the calculation unit 23 performs learning and supplies the learning results from the information processing device 20 to the information processing device 30. This is possible. For example, the arithmetic unit 23 and the arithmetic unit 33 use a neural network. If it has a function to perform calculations, the calculation unit 23 learns and obtains weight coefficients, etc. Weight coefficients and the like can be supplied from the information processing device 20 to the information processing device 30. Therefore, even if the arithmetic unit 33 provided in the information processing device 30 does not perform learning, the input to the arithmetic unit 33 Based on the data obtained, the calculation unit 23 provided in the information processing device 20 performs the learning results. Therefore, the computational processing power of the arithmetic unit 33 is lower than that of the arithmetic unit 23. It can be considered a good thing.

[0032] The arithmetic unit 23 performs learning and supplies the learning results from the information processing device 20 to the information processing device 30. In this case, the information processing device 20 can be installed, for example, on a server. If 0 is provided on the server, the information processing device 20 does not need to have an imaging unit 21 and a display unit 22. It is also possible to place the imaging unit 21 and the display unit 22 outside the information processing device 20. good.

[0033] Furthermore, the information processing device 30 can be, for example, a smartphone, a tablet, or a personal computer. It can be installed in information terminals such as a computer. Also, at least one component of the information processing device 20 The server may also have a portion of the information processing device 30 and at least a portion of its components. For example, the server may be provided with a calculation unit 23 and a calculation unit 33. In this case, for example, The data acquired by the information terminal is supplied to the processing unit 33 via the network and stored on the server. The calculation unit 33 performs inference on the data. The inference result is then displayed on the network. By supplying the information terminal via the workpiece, the information terminal can acquire the inference results. Cut.

[0034] <An example of an information processing method> The following describes an example of an information processing method using the information processing system 10. Specifically, The user of the information terminal on which the information processing device 30 of the information processing system 10 is installed, This section describes an example of a method for estimating fatigue, drowsiness, etc., using machine learning-based calculations.

[0035] Figures 2 and 3 illustrate an example of a method for estimating fatigue, drowsiness, etc., using machine learning calculations. This is a flowchart illustrating the process. The learning process is shown in Figure 2, and the inference process is shown in Figure 3.

[0036] An example of a learning method will be explained using Figure 2, etc. First, the imaging unit 21 captures a video. For example, capture a video including a human face (step S01). Here, video means 2 This shows a collection of images, more than a frame. As will be explained in more detail later, it is based on the video captured by the imaging unit 21. The processing unit 23 creates training data and performs the learning. Therefore, for example, the imaging unit 21 captures a human face. When capturing video including, the imaging unit 21 captures a large number of people with different genders, races, body types, etc. It is preferable to capture video.

[0037] Furthermore, image processing may be performed on the video captured by the imaging unit 21. For example, noise reduction. It can perform operations such as grayscale conversion, normalization, and contrast adjustment on video. You may perform binarization or other operations on the included images. By performing such operations, The steps can be performed with precision. For example, the first step performed in step S02 described later It can detect objects with high accuracy.

[0038] Next, the calculation unit 23 detects the first object from each of the captured images. The object can be, for example, an eye if a video of the face is captured in step S01. Step S02). The first object can be detected, for example, by a cascade classifier. It can be detected, for example, by Haar Cascades. If an object is considered an eye, and both eyes are included in a single image, then only one eye can be detected. can.

[0039] Subsequently, the calculation unit 23 detects a second object from each of the first objects that were detected. If the first object is the eye, then the second object can be the pupil (Step S03). For example, the pupil can be detected from the eye using circular extraction. Detecting the pupil from the eye Details of the method will be described later.

[0040] Here, the pupil is the hole surrounded by the iris and can be called the "black of the eye." It has the function of adjusting the amount of light projected onto the retina. Also, the iris is, for example, located between the cornea and the lens. It is a thin membrane located on the surface, and can be considered the colored part in the eye.

[0041] Next, the calculation unit 23 calculates the size of each of the detected second objects. (Step S04). For example, if a second object is detected by circular extraction, the second object The radius or diameter of the body can be used as the size of the second object. Also, the shape of the second object... If the shape is extracted as an ellipse, the length of the major axis and the length of the minor axis will be used as the size of the second object. This is possible. Furthermore, the area of ​​the second object can be used as the size of the second object.

[0042] Subsequently, the calculation unit 23 uses the size of the second object to perform learning and obtain the learning result. Step S05). Specifically, the learning results are obtained based on the change in the size of the second object over time. It is advantageous. Learning can be done, for example, using a neural network. In this case, The learning results can be expressed as weight coefficients, as mentioned above. Details of the learning method will be described later. .

[0043] Next, the information processing device 20 supplies the learning results to the information processing device 30 (step S06). Specifically, the learning results acquired by the calculation unit 23 are transmitted to the communication unit 26 via the transmission path 27. Then, the data is supplied from communication unit 26 to communication unit 36. The learning results supplied to communication unit 36 ​​are: The learning results can be stored in the auxiliary storage unit 35. That's fine.

[0044] Next, an example of an inference method based on learning results obtained by the method shown in Figure 2 is shown in Figure 3. This will be explained using the following. First, the imaging unit 31 captures a video. For example, the information processing device 30 The system captures a video including the face of the user of the information terminal that is equipped with the system (step S11). In step S01 shown in Figure 2, the imaging unit 21 performs image processing on the captured video. If this is done, performing the same image processing on the video captured by the imaging unit 31 will refine the inference. It is preferable because it can be done efficiently.

[0045] Next, the calculation unit 33 detects the first object from each image contained in the recorded video. To output. The first object is, for example, the eye if a video of the face was captured in step S11. This can be done (step S12). The first object is used in step S02 shown in Figure 2. It can be detected using the same method as the detection method used.

[0046] Subsequently, the calculation unit 33 detects a second object from each of the first objects that were detected. If the first object is the eye, then the second object can be the pupil (step S13). The second object is detected in the same manner as the detection method used in step S03 shown in Figure 2. It can be released.

[0047] Next, the calculation unit 33 calculates the size of each of the detected second objects. (Step S14). The method for calculating the size is used in Step S04 shown in Figure 2. A similar method can be used.

[0048] Subsequently, based on the change in the size of the second object over time, the calculation unit performs the calculation in step S05 shown in Figure 2. The calculation unit 33, which receives the learning results acquired by 23, performs inference. For example, the imaging unit 31 The captured video includes the face of the user of the information terminal equipped with the information processing device 30. If the second object is the pupil of the user's eye, the calculation unit 33 will determine the user's fatigue. This allows for the estimation of drowsiness, etc. (Step S15). Details of the inference method will be described later.

[0049] Furthermore, pupil size can change not only due to fatigue or drowsiness, but also, for example, due to the brightness of the environment. Therefore, in step S01 shown in Figure 2, for example, a video of the face of the same person, It is preferable to take multiple images with varying ambient brightness. This allows for, for example, different ambient brightness levels. Regardless, the information processing device 30 accurately detects fatigue, drowsiness, etc., of the user of the information terminal equipped with the information processing device 30. It can be estimated fairly well.

[0050] In one aspect of the present invention, an information processing device 3 has the function of estimating fatigue, drowsiness, etc., as described above. The number 0 is placed on information terminals such as smartphones, tablets, and personal computers. This allows for real-time monitoring of user fatigue, drowsiness, etc., without the need for specialized equipment. It can be detected by [unclear].

[0051] [An example of a method for detecting the pupil] Next, an example of a pupil detection method performed in steps S03 and S13 will be described. Figure 4 is a flowchart showing an example of a method for detecting the pupil.

[0052] First, the calculation unit acquires image 41, which is an image containing the detected eye (step S31). Figure 5A is a schematic diagram illustrating step S31. As shown in Figure 5A, the imaging unit The processing unit acquires image 41, which includes the eye detected from the image captured by the camera. Specifically, In step S03, the image including the eye detected by the calculation unit 23 in step S02 is combined with image 41. The calculation unit 23 then obtains the result. Also, in step S13, the calculation unit 33 obtained the result in step S12. The processing unit 33 acquires the image including the detected eye as image 41. If the image is a color image, after the calculation unit acquires image 4, the calculation unit then processes image 4 You may convert 1 to grayscale.

[0053] Next, the calculation unit performs an expansion process on image 41 to obtain image 42, and then performs a contraction process. Go and get image 43 (step S32). That is, close for image 41. By performing the processing, image 43 is obtained. Figure 5B shows the expansion and contraction processes. This is a schematic diagram for explanation.

[0054] Subsequently, the calculation unit subtracts image 43 from image 41 to obtain image 44 (step S33). ). In other words, image 44 is an image represented by the difference between image 41 and image 43. Step In step S33, the calculation unit performs a Black-hat transform using images 41 and 43. This allows us to obtain image 44.

[0055] Next, the calculation unit processes the image 41 acquired in step S31 and the image acquired in step S33. Add 44 and to obtain image 45 (step S34). Note that step S31 If image 41 is converted to grayscale, then in step S34, the image is converted to grayscale. The converted image 41 and image 44 can be added together.

[0056] Furthermore, the processes shown in steps S32 to S34 may be omitted in whole or in part. Alternatively, other processes besides those shown in steps S32 to S34 may be performed. .

[0057] Subsequently, the processing unit performs image processing on image 45 and obtains image 46 (step S 35). For example, the processing unit performs noise reduction, smoothing, and other processing on the image 45. This performs processes such as edge detection and binarization. Specifically, for example, it performs intermediate value processing on image 45. After noise reduction using Ruta and smoothing with a Gaussian filter, the Canny method is applied. Edge detection and binarization are performed using [a specific method / tool]. Noise reduction is performed using, for example, a moving average filter. This may also be done by [method name]. Furthermore, smoothing can be done, for example, by a moving average filter or median filter. This may also be done by a ruta. Furthermore, edge detection can be performed, for example, by a Laplacian filter. That's fine.

[0058] Next, the calculation unit detects the iris 47 from image 46. For example, by using the Hough transform... This allows for the detection of the iris 47. When using the Hough transform, for example, the iris 47 can be measured in a circular shape. It can be detected. Or, for example, the iris 47 can be detected in an elliptical shape. The iris 47 may be detected using a generalized Hough transform.

[0059] Then, the calculation unit acquires an image 49 including the detected iris 47 (step S36). For example, based on the coordinates of the detected iris 47 in image 46, image 49 is obtained from image 46. Extract.

[0060] Figure 5C is a schematic diagram illustrating step S36. For example, image 49 is a schematic diagram of Figure 5C. As shown, it can be a rectangle with all four sides touching the iris 47. For example, a circular shape with the iris 4 If 7 is detected, the image 49 can be made into a square with all four sides touching the iris 47. Oh, each side of image 49 does not have to be in contact with the iris 47. For example, with the iris 47 as the center An image with a predetermined number of pixels may be designated as image 49.

[0061] Next, the calculation unit detects the pupil 48 from the image 49 (step S37). For example, New The pupil 48 is detected from image 49 through calculations using a network.

[0062] Step S37 is performed using a pre-trained generator. Here, A generator is a program that performs calculations using machine learning, and it responds to the input data. It has the function of outputting data. Specifically, the generator learns by The generator can perform inferences on the data input to it.

[0063] Figure 6A is a schematic diagram illustrating the learning process described above. Here, the generator used for learning is Let's call it Generator 50. When using a neural network as Generator 50... The generator 50 is a convolutional neural network. A Neural Network (CNN) can be used. In particular, a type of CNN Therefore, it is preferable to use U-net. In U-net, the input image is convolved After downsampling, the features obtained from downsampling are used for inverse convolution. Upsampling is performed by the metering. The calculation unit 23 and calculation unit 33 are connected to the generator 50 It can be said that it functions as such.

[0064] The generator 50 was trained using supervised learning with data 51 and data 52. It is possible to do so. Data 51 can be considered as a set of images 59. Images 59 include a rainbow. Color 57 and pupil 58 are included. Image 59 shows steps S01 and S02 as shown in Figure 2. , and in the same manner as steps S31 to S36 shown in Figure 4, the information processing device 20 This can be obtained. In step S01, the imaging unit 21 captures a video of the face. Although it is assumed that this will be done, when acquiring image 59, it is not necessary to capture video. For example, The imaging unit 21 may capture one image (one frame) per person.

[0065] Data 52 is data showing the coordinates of the pupil 58 contained in image 59. Specifically, the pupil A binary image can be created by making the color of the area with hole 58 different from the color of the other areas. 52 can be obtained, for example, by filling in the pupil 58 contained in image 59. Alternatively, after obtaining an image including the eye in the same manner as in step S31 shown in Figure 4, the eye By filling in the pupil 58 in the image containing the image, data 52 can be obtained. ru.

[0066] The training of generator 50 involves inputting data 51 into generator 50 and then outputting data This is done so that it approaches data 52. In other words, data 52 is used as the ground truth data for generation. The generator 50 is trained. As the generator 50 learns, 0 generates the learning result 53. A neural network is used as the generator 50. In this case, the learning result 53 can be used as a weight coefficient, etc.

[0067] The learning of the generator 50, that is, the generation of the learning result 53, is performed, for example, by the information processing device 20. The calculation unit 23 can perform the calculation. Then, the learning result 53 is processed from the information processing device 20. By supplying it to the processing device 30, the calculation unit 33 can also perform the same inference as the calculation unit 23. This becomes possible. The learning result 53 generated by the calculation unit 23 is stored, for example, in the auxiliary storage unit 25. Furthermore, the learning results generated by the calculation unit 23 and supplied to the information processing device 30 can be displayed. For example, item 53 can be stored in the auxiliary storage unit 35.

[0068] This completes the training of generator 50.

[0069] Figure 6B is a schematic diagram illustrating step S37. In other words, Figure 6B is a schematic diagram illustrating image 49. This is a schematic diagram illustrating the detection of pupil 48.

[0070] As shown in Figure 6B, in step S37, the generator 5 loaded with the learning result 53 Input the image 49 acquired by the calculation unit in step S36 to 0. This generates Task 50 performs inference on image 49 and outputs data indicating the coordinates of the pupil 48. Yes, it is possible. For example, generator 50 produces a binary image where the color of the pupil 48 is different from the color of the rest of the image. It can output an image.

[0071] By the above method, from among the eyes detected in step S02 or step S12, The pupil can be detected in step S03 or step S13.

[0072] By using machine learning to detect the pupil, for example, pupil detection can be performed by visual inspection. This method allows for pupil detection in a shorter time than other methods. Furthermore, for example, if the surrounding scenery is visible in the pupil... Even if the pupil is visible in the image, it can be detected with high accuracy.

[0073] Note that the method for detecting the pupil 48 in step S37 is limited to the method shown in Figures 6A and 6B. No. For example, if image 49 is a color image, then the same color image 49 is converted to grayscale. After the transformation, edge detection of pupil 48 may be performed. Then, after edge detection, the pupil 48 may be detected.

[0074] The grayscale conversion of Image 49 can be done using, for example, partial least squares (Least Squares) This can be done using Squares(PLS) regression. Image 49 has been converted to grayscale. By doing so, the difference between the brightness of the pupil (48) and the brightness of the iris (47) can be increased. This allows us to emphasize the boundary between pupil 48 and iris 47, thus highlighting the edge of pupil 48. It is possible to detect pupils with high accuracy. Therefore, it is possible to detect pupils with high accuracy. Cut.

[0075] Pupil edge detection can be performed, for example, using the Canny method or a Laplacian filter. This can be done. Also, for detecting the pupil 48 after edge detection, for example, a Hough transform can be used. This can be done by using the Hough transform, for example, detecting the pupil 48 in a circular shape. It is possible to detect the pupil 48 in an oval shape, for example. The pupil 48 may be detected using the F-converter.

[0076] In step S03 or step S13, if the pupil is detected as well as the iris, In step S01 or step S11, imaging can be performed using infrared light. It reflects infrared rays. On the other hand, the pupil does not reflect infrared rays. Therefore, step S01 or In step S11, imaging is performed using infrared light to clearly define the iris and pupil. It can be distinguished from the pupil. Therefore, the pupil can be detected with high accuracy.

[0077] [Example of a method for estimating fatigue, drowsiness, etc._1] Next, the use of the information terminal equipped with the information processing device 30, based on calculations using machine learning. An example of a method for estimating fatigue, drowsiness, etc., of a person will be explained. Specifically, the method performed in step S05 will be described. This explains an example of a learning method using pupil size. Furthermore, it describes the process performed in step S15. Next, we will explain an example of a method for estimating fatigue, drowsiness, etc., based on the above learning results. In the following explanation, we will refer to the second object as the pupil.

[0078] Figure 7A1 is a schematic diagram illustrating step S05. In step S05, the machine The program Generator 60, which performs calculations through learning, will be trained. 0 allows the use of a neural network. Details will be discussed later, but the generator... For 60, you input time-series data, such as the change in pupil size over time. Therefore, When using a neural network as generator 60, Recurrent Neural Network: It is preferable to use RNN. Alternatively, as the generator 60, long- and short-term memory (Lo It is preferable to use Short-Term Memory (LSTM). , a Gated Recurrent Unit (GRU) It is desirable to use it.

[0079] The generator 60 can be trained using data 61 and data 62. Data 61 is data acquired in step S04, and represents the change in pupil size over time. This can be done. As mentioned above, for example, when the pupil is detected by circular extraction, the radius of the pupil Alternatively, the diameter can be used as the size of the pupil. Also, if the pupil is detected as elliptical, The length of the long axis and the length of the short axis can be used to determine the size of the pupil. Also, the area of ​​the pupil can be... This can be the size of the pupil. In Figure 7A1, time 1 to n-1 (where n is an integer of 3 or more) Data 61 shows the change in pupil size over time in the given number (number).

[0080] Data 61 may also represent the change over time in the ratio of pupil size to iris size. In this case, it is preferable that the iris and pupil be extracted into the same type of shape. For example, a rainbow When extracting the iris in a circular shape, it is preferable to extract the pupil in a circular shape as well. When extracting, it is preferable to extract the pupil as an oval shape. Data 61 is the pupil By considering the change over time in the ratio of size to iris size, for example in step S37 When detecting the pupil using the method shown, the resolution of the image 49, which includes the iris 47 and the pupil 48, is compared. It can be made to look different. For example, an image 49 including the iris 47 and pupil 48 of a first human. The resolution of the first image and the resolution of the second image 49, which includes the iris 47 and pupil 48 of a human, are different from each other. It can be made to happen.

[0081] Data 62 is the pupil size at time n. In other words, it is the pupil size included in data 61. This is the pupil size at a time after the time when the size was measured. Note that data 61 is When considering the change over time in the ratio of pupil size to iris size, data 62 also shows the pupil's The ratio of the size to the size of the iris is used.

[0082] Figure 7A2 shows an example of the relationship between pupil diameter and time. In Figure 7A2, the black circles The circle indicates the measured pupil diameter. In other figures, the measured points may also be indicated by black circles. Figure 7A2 As shown, data 62 is from a time later than the time when the pupil size included in data 61 was measured. This can be the pupil size at the given time. For example, data 62 can be the pupil size at the given time. The pupil size included is taken as the pupil size at the time immediately following the last time measurement. It is possible.

[0083] Here, if the generator 60 is to have a function to estimate whether or not fatigue is present, then those who are fatigued will be... The change in pupil size over time is not included in data 61 and data 62. In other words, data 6 Data 1 represents the change in pupil size over time in a person without fatigue, and Data 62 represents the pupil size of a person without fatigue. The size should be as follows. Also, if the generator 60 is to have a function to estimate whether or not the user is drowsy, The temporal changes in pupil size of individuals experiencing drowsiness are not included in Data 61 and Data 62. Data 61 represents the change in pupil size over time in individuals without drowsiness, and Data 62 represents the change in pupil size over time in individuals with drowsiness. Use the pupil size of someone without pupils.

[0084] The training of generator 60 involves inputting data 61 into generator 60 and then outputting data This is done so that it approaches data 62. In other words, data 62 is used as the ground truth data for generation. The generator 60 is trained. As the generator 60 learns, 0 generates the learning result 63. A neural network is used as the generator 60. In this case, the learning result 63 can be used as a weight coefficient, etc.

[0085] Figures 7B1 and 7B2 are schematic diagrams illustrating step S15, and the generator Using 60, the information processing device 30 detects fatigue, drowsiness, etc., in the user of the information terminal where it is installed. This figure shows an example of a method for determining the value. In step S15, first, as shown in Figure 7B1, The data 64 obtained in step S14, which shows the change in pupil size over time, is used for learning. The result 63 is input to the generator 60 from which it was read. For example, in step S05 During the training of generator 60, the time-dependent change in pupil size from time 1 to n-1 is used. When used as input data, the time 1 to n- The input data for 1 is the change in pupil size over time. In other words, data 64 is used as the input data for time The pupil size of the user of the information terminal on which the information processing device 30 is installed, in 1 to n-1. This represents the change in size over time. Based on this, the generator 60 performs inference on the data 64. Next, output data 65. Furthermore, data 65, which is the inference data at time n, will also be output. Using this method, the data at time 2 through n is taken as input data, and the data at time n+1 You may infer that.

[0086] Furthermore, if data 61 represents the change over time in the ratio of pupil size to iris size, Data 64 also represents the change over time in the ratio of pupil size to iris size. By considering the change over time in the ratio of the pore size to the iris size, for example, step S3 When detecting the pupil using the method shown in 7, the resolution of the image 49 including the iris 47 and the pupil 48 These can be made to differ from each other. For example, the size of the pupil 48 at time 1 and the iris 4 The resolution of image 49, acquired by the calculation unit to calculate the size of 7 and the ratio of , and time n-1 The calculation unit obtains the size of the pupil 48 and the size of the iris 47 in order to calculate the ratio. The resolution of image 49 and the other can be made to be different from each other.

[0087] Data 65 is calculated by performing inference on data 64 based on the learning result 63. Furthermore, the pupil size included in data 64 is measured at a time later than the time the pupil size was measured. This is an estimate of the pupil size at time 1 to n-1. For example, data 64 is used to estimate the pupil size at time 1 to n-1. If we consider this as a change over time, data 65 can be the pupil size at time n. In 7B1, the measured pupil size at time 1 to n-1 is x1 to x1, respectively. n -1 It states that the estimated pupil size at time n is given by x n (E) Data 64 represents the time-dependent change in the ratio of pupil size to iris size. In this case, data 65 is the ratio of pupil size to iris size.

[0088] Next, as shown in Figure 7B2, data representing, for example, the measured value of pupil size at time n. We compare 66 with data 65, which is the data output from generator 60. For example, the measured pupil size at time n is compared with the estimated value. Based on the comparison results... The presence or absence of fatigue, drowsiness, etc. is estimated. For example, the generator 60 is a machine that estimates the presence or absence of fatigue. If capable, the generator 60 learns, for example, the pupil size of a person without fatigue over time. The change is used as input data. Therefore, the information terminal on which the information processing device 30 is installed If the user is not fatigued, data 65 will be similar to data 66. Therefore, the difference between data 65 and data 66 will be small. Meanwhile, the information processing device 30 is set If the user of the information terminal being accessed is in a fatigued state, data 66 and data 65 The difference is that the user of the information terminal on which the information processing device 30 is installed is in a fatigue-free state. It will be larger than in some cases. Therefore, by comparing data 66 and data 65, The information processing device 30 can estimate whether or not the user of the information terminal equipped with it is fatigued. The same applies when estimating the presence or absence of drowsiness. Furthermore, data 65 is used to determine pupil size and iris. If we use the size of the pupil and the ratio of the two as estimates, then data 66 will be used as the pupil size and the iris size. This is the measured value of the ratio of ,.

[0089] The function of the generator 60 can be provided to both the calculation unit 23 and the calculation unit 33. In this case, the arithmetic unit 23 of the information processing device 20 learns the generator 60. The learning result 63 is generated, and the learning result 63 is supplied from the information processing device 20 to the information processing device 30. This can be done. As a result, the arithmetic unit 33 provided in the information processing device 30 performs learning. Even without it, the data input to the calculation unit 33 is processed by the calculation device 20. Inference can be performed based on the learning results from unit 23. Therefore, the calculation process of the calculation unit 33 The processing power can be set lower than that of the calculation unit 23. Note that the learning result 63 is an auxiliary memory. The data can be stored in the memory unit 25 and the auxiliary memory unit 35.

[0090] [An example of a method for estimating fatigue, drowsiness, etc._2] Figures 8A and 8B are schematic diagrams illustrating step S05, and differ from the method described above. This is an example of a method for training a generator. Specifically, Figure 8A shows that the training data is Figure 8B shows an example of how data to be input to the generator is created. This figure shows an example of how generator 80 learns. Generator 80 performs calculations using machine learning. This is a program that uses a neural network as generator 80. It is possible.

[0091] The data 81 shown in Figure 8A is data acquired in step S04, and represents the size of the pupil. This can be considered as a change over time. As mentioned above, for example, when the pupil is detected by circular extraction. The size of the pupil can be defined as the radius or diameter of the pupil. Furthermore, the pupil can be defined as elliptical. If detected, the length of the long axis and the length of the short axis can be used as the pupil size. Furthermore, the area of ​​the pupil can be defined as the size of the pupil. Note that the learning method shown in Figure 7A1 is also applicable. Similarly, data 81 can be used to show the change over time in the ratio of pupil size to iris size. can.

[0092] Here, data 82 is generated by performing a Fourier transform on data 81. (Figure) As shown in 8A, the change in pupil diameter over time is converted to the frequency characteristics of pupil diameter using the Fourier transform. It can be exchanged. Furthermore, data 81 is the ratio of pupil size to iris size. When considering changes over time, data 82 represents the frequency characteristics of the ratio of pupil size to iris size. It can be done this way.

[0093] The generator 80 is trained using data 82 and data 83, as shown in Figure 8B. This is possible. Data 82 represents the frequency characteristics of pupil diameter, as mentioned above. Data 83 is This can be used as a label to indicate the presence or absence of fatigue. For example, data 83 could be used to indicate those who are fatigued. Both the frequency characteristics of pupil size in one person and the frequency characteristics of pupil size in a person without fatigue are To include it. And the frequency characteristics of pupil size in fatigued individuals include "fatigue The label "fatigue-free" is associated with the frequency characteristics of pupil size in those without fatigue. A label is associated with this. Additionally, data 83 may be used as a label indicating the presence or absence of drowsiness.

[0094] The training of generator 80 involves inputting data 82 into generator 80 and then outputting data This is done so that it approaches data 83. In other words, data 83 is used as the ground truth data for generation. The generator 80 is trained. As the generator 80 learns, 0 generates the learning result 84. A neural network is used as the generator 80. In this case, the learning result 84 can be used as a weight coefficient, etc.

[0095] By performing a Fourier transform on the change in pupil size over time, the input to generator 80 is obtained. The data being generated can be data that is not time-series data. This allows for generation Even without using an RNN as the lattice 80, the generator 80 can perform learning and inference. can.

[0096] Figures 9A and 9B are schematic diagrams illustrating step S15, and the generator 80 Using this, the information processing device 30 estimates the fatigue, drowsiness, etc., of the user of the information terminal where the information processing device 30 is installed. This figure shows an example of how to do it.

[0097] The data 85 shown in Figure 9A is data acquired in step S04, and represents the size of the pupil. This can be considered as a change over time. As mentioned above, for example, when the pupil is detected by circular extraction. The size of the pupil can be defined as the radius or diameter of the pupil. Furthermore, the pupil can be defined as elliptical. If detected, the length of the long axis and the length of the short axis can be used as the pupil size. Furthermore, the area of ​​the pupil can be defined as the size of the pupil. Note that the data 81 shown in Figure 8A If we consider the change over time in the ratio of pupil size to iris size, then data 85 also shows pupil size. This refers to the change over time between the size of the iris and the ratio of the iris to the size of the iris.

[0098] Here, data 86 is generated by performing a Fourier transform on data 85. (Figure) As shown in 9A, the change in pupil diameter over time is converted to the frequency characteristics of pupil diameter using the Fourier transform. It can be exchanged. Furthermore, data 85 is the ratio of pupil size to iris size. When considering the change over time, data 86 represents the frequency characteristics of the ratio of pupil size to iris size. It can be done this way.

[0099] Then, as shown in Figure 9B, the Fourier-transformed data 86 is input to the generator 80. As a result, the generator 80 performs inference on the data 86 and expresses whether or not fatigue is present. Data 87 can be output. Note that the data 87 shown in Figure 9B indicates the presence or absence of drowsiness. If it is a label, the data 87 output by generator 80 will indicate whether or not the person is drowsy. It can be written as "ta".

[0100] The functions of generator 80 are the same as those of generator 60, and generator 7 Similar to its function as 0, this can be assigned to both the arithmetic unit 23 and the arithmetic unit 33. This allows the arithmetic processing capability of the arithmetic unit 33 to be lower than that of the arithmetic unit 23.

[0101] [Example of a method for estimating fatigue, drowsiness, etc._3] Figure 10A is a schematic diagram illustrating step S05, and shows a different method from the one described above. This is an example of a generator training method. Figure 10A shows the training of generator 70. Generator 70 is a program that performs calculations using machine learning. For example, Generator 7 Assuming 0, a neural network can be used, for example, by using an autoencoder. It is possible.

[0102] When the generator 70 performs learning, data 71 is input to the generator 70. Data 71 is the data acquired in step S04 and represents the change in pupil size over time. This is possible. Here, if the generator 70 is to be equipped with a function to estimate whether or not fatigue is present, The change in pupil size over time in fatigued individuals is not included in Data 71. 1 represents the change in pupil size over time for individuals without fatigue. Also, the generator 70 is used for drowsiness. If the system is designed to estimate the presence or absence of sleepiness, the change in pupil size over time in a sleepy person will be used for data analysis. It will not be included in Data 71. In other words, Data 71 will be the change in pupil size over time for those who are not drowsy. ru.

[0103] As mentioned above, if the pupil is detected by circular extraction, for example, the radius or diameter of the pupil is This can be used to determine the size of the pupil. Also, if the pupil is detected as elliptical, the length of its major axis can be used. The length of the minor axis and the area of ​​the pupil can be used to determine the size of the pupil. It is possible to enlighten them.

[0104] Furthermore, similar to the learning method shown in Figure 7A1, data 71 consists of pupil size and iris size. This can be expressed as the change in the ratio of and over time. Also, as in the case shown in Figure 8A, for example, the pupil The Fourier transform of the change in magnitude over time may be used as data 71.

[0105] The training of generator 70 involves inputting data 71 into generator 70 and then outputting data This is done so that data 72 approaches the input data 71. In other words, data 71 and The generator 70 is trained so that the data 72 become equal. Through learning, the generator 70 generates the learning result 73. When using a neural network with a value of 0, the learning result 73 should be used as the weight coefficient, etc. It is possible.

[0106] Figures 10B1 and 10B2 are schematic diagrams illustrating step S15, and Genere Using the 70, the fatigue, drowsiness, etc. of the user of the information terminal equipped with the information processing device 30. This figure shows an example of a method for estimating [something]. In step S15, first, as shown in Figure 10B1... Furthermore, the data 74 obtained by step S14, which shows the change in pupil size over time, The learning result 73 is then input into the generator 70. 70 performs inference on data 74 and outputs data 75.

[0107] Furthermore, if data 71 represents the change over time in the ratio of pupil size to iris size, then 74 is also defined as the change over time in the ratio of pupil size to iris size. Furthermore, the Fourier transform If the transformed data is used as data 71, then data 74 will also be the Fourier transformed data. Use the following. For example, let data 71 be the Fourier transform of the change in pupil size over time. In that case, data 74 will also be assumed to be the Fourier transform of the change in pupil size over time.

[0108] Next, as shown in Figure 10B2, the data 74 which is the data input to the generator 70 and Then, compare it with data 75, which is the data output from generator 70. Based on the comparison results... It estimates the presence or absence of fatigue, drowsiness, etc. For example, Generator 70 estimates the presence or absence of fatigue. If it has the function, the generator 70 learns, for example, the pupil size of a person without fatigue. This is done using time changes. Therefore, the user of the information terminal equipped with the information processing device 30 If there is no fatigue, then data 75, which is the output data from generator 70, This will be close to data 74, which is the input data to generator 70. In other words, data The difference between 74 and data 75 will be small. Meanwhile, an information processing device 30 is provided. If the user of the information terminal is fatigued, the difference between data 74 and data 75 is... The information processing device 30 is installed in a state where the user of the information terminal is not fatigued. Therefore, by comparing data 74 and data 75, the information processing device It is possible to estimate whether or not the user of the information terminal equipped with 30 is fatigued. The same applies when estimating nothingness.

[0109] The function of the generator 70 is the same as the function of the generator 60, and the calculation unit 23 This can be provided to both the and the arithmetic unit 33. This increases the arithmetic processing capacity of the arithmetic unit 33. This can be set to a value lower than that of the calculation unit 23.

[0110] [Example of a method for estimating fatigue, drowsiness, etc._4] The learning performed in step S05, and the learning results performed in step S15, The reasoning is based on Generative Adversarial Networks. This can also be done using a Network (GAN). For example, AnoGAN (Anorma This can also be done using a GAN. Figure 11 shows a model that can perform the above learning and inference. This is a diagram explaining AnoGAN.

[0111] The AnoGAN shown in Figure 11 has a generator 91 and a discriminator 92. The generator 91 and discriminator 92 are generated by a neural network. It can be constructed in this way.

[0112] The discriminator 92 uses imaging to determine the pupil size of individuals who are not fatigued, drowsy, etc. Data 93, which is time-series data representing the change in size over time, is input. Alternatively, discrete The generator 92 is the time-series data generated by the generator 91, which received data 94 as input. Data 95 is input. The discriminator 92 determines that the input data is used for imaging. Either the data 93 obtained by or the data 95 generated by the generator 91 It has a function to determine whether it exists or not (also called authenticity determination). Data 93 is used in imaging. The time-series data obtained from the analysis of pupil size changes over time in individuals without fatigue, drowsiness, etc. Alternatively, the data could be the result of a Fourier transform.

[0113] The judgment result is output as data 96. Data 96 is, for example, a continuous range between 0 and 1. It can be any value. In this case, for example, discriminator 92, after learning is complete, input If the data obtained is data 93 acquired by imaging, then it will be set as data 96. It outputs a value close to the input data, and the input data is the data 95 generated by generator 91. In that case, you should output a value close to 0 as data 96.

[0114] Data 94 is a multidimensional random number (also called a latent variable). Here, the latent variable represented by Data 94 Let the present variable be the latent variable z. Generator 91 uses this data 94 as a basis. Data that is as similar as possible to data showing the change in pupil size over time in individuals without fatigue, drowsiness, etc. It has the function of generating.

[0115] The learning process alternates between learning the discriminator 92 and learning the generator 91. During the learning process of the discriminator 92, the neural network that constitutes the generator 91 is used. The weight coefficients of the twerk are fixed. Also, during the training of generator 91, discriminants The weight coefficients of the neural network that makes up -92 are fixed.

[0116] During the training of the discriminator 92, the data 93 acquired by imaging, or the generated data The data 95 generated by the data 91 is input to the discriminator 92. The data entered into the data 92 is assigned a correct label. The discriminator 92 outputs For the data 96, the correct labels can be determined as follows. For example, When data 93 is input to discriminator 92, the correct label is set to "1", and when data 95 is input to discriminator 92, the correct label is set to "0". By performing learning in the above manner, discriminator 92 can perform genuine / fake judgment.

[0117] During the learning of generator 91, data 94 representing latent variable z is input to generator 91. Then, based on the input data 94, generator 91 generates data 95. The correct label of data 96 is set to "1". And the learning of generator 91 is performed so that the value of data 96 output from discriminator 92 becomes "1". As the learning of generator 91 progresses, generator 91 can generate data 95 similar to data 93 obtained by imaging.

[0118] When the learning of generator 91 is completed, generator 91 can generate data 95 similar to data 93 obtained by imaging regardless of what latent variable z is input as data 94.

[0119] Next, the operation during inference will be described.

[0120] First, assume that data representing the temporal change in the pupil size of a person without fatigue, sleepiness, etc. is obtained by photography. At this time, search the latent variable space and find latent variable z1 that generates data most similar to the data of the pupil size of the person without fatigue, sleepiness, etc. It has the function of generating data that is very similar to the data. Therefore, the generator is latent Data generated from variable z1 and the pupil size of a person who does not show fatigue, drowsiness, etc., as obtained from the photograph. The data representing the change in size over time will be extremely similar.

[0121] Next, imaging was used to obtain data representing the temporal changes in pupil size of individuals experiencing fatigue, drowsiness, etc. Let's assume this has happened. At this time, we search the space of latent variables and find the pupil size of those who have the above fatigue, drowsiness, etc. The latent variable z2 that generates data closest to the data representing the change in size over time is determined using methods such as gradient descent. This is how it is discovered. Generator 91, through learning, determines the pupil size of individuals who are not fatigued, sleepy, etc. It has the ability to generate data that is very close to representing changes over time, such as fatigue and drowsiness. It does not have the ability to generate data similar to data representing the temporal changes in pupil size of a given individual. Therefore, the data generated by the generator from the latent variable z2 and the data obtained by imaging are different. The data showing the change in pupil size over time in individuals experiencing fatigue, drowsiness, etc., is extremely similar to the data described above. This does not result in fatigue. Based on the above, the presence or absence of fatigue, drowsiness, etc. is estimated by the generator 91. It is possible.

[0122] As shown in Figures 7 to 11 above, the information processing device 30 is provided with This system uses machine learning to estimate the fatigue, drowsiness, and other symptoms of users of information terminals. By using learning, for example, the change in features over time when estimating fatigue, and fatigue itself. Even without manually setting the time-dependent changes in features when estimating "none," fatigue, drowsiness, etc., can be accurately estimated. It can be estimated that, for example, the pupil size when fatigue is estimated to be present. You don't need to manually set the changes over time, or the changes in pupil size over time when fatigue is assumed to be absent. Furthermore, it can accurately estimate fatigue, drowsiness, etc. Also, for example, in the case where fatigue is estimated to be present... The time-dependent changes in the combined feature and the time-dependent changes in the feature when fatigue is estimated to be absent are manually set. Even without it, fatigue, drowsiness, etc. can be estimated, so a simple method is used to estimate fatigue, drowsiness, etc. It is possible.

[0123] If the information processing device 30 estimates that the user is fatigued, sleepy, etc., then, for example, the information processing device 30 The information terminal's display shows an alarm indicating that fatigue, drowsiness, etc., are occurring. This allows, for example, the user of the information terminal to use the information terminal earlier. It is possible to urge them to stop. Alternatively, the information terminal on which the information processing device 30 is installed can be prompted to stop. The power can be turned off. This can cause fatigue, drowsiness, etc., in the user of the information terminal. Nevertheless, it is possible to suppress the occurrence of health damage caused by continued use of information terminals. Cut. [Explanation of Symbols]

[0124] 10: Information processing system, 20: Information processing device, 21: Imaging unit, 22: Display unit, 23: Performance Calculation unit, 24: Main memory unit, 25: Auxiliary memory unit, 26: Communication unit, 27: Transmission line, 30: Information processing unit Processing unit, 31: imaging unit, 32: display unit, 33: calculation unit, 34: main memory unit, 35: auxiliary memory Department, 36: Communications Department, 37: Transmission Line, 41: Image, 42: Image, 43: Image, 44: Image, 45: Image, 46: Image, 47: Iris, 48: Pupil, 49: Image, 50: Generator 51: Data, 52: Data, 53: Learning results, 57: Iris, 58: Pupil, 59: Image 60: Generator, 61: Data, 62: Data, 63: Learning result, 64: Data, 6 5: Data, 66: Data, 70: Generator, 71: Data, 72: Data, 73: Learning results, 74: data, 75: data, 80: generator, 81: data, 82: data DATA, 83: Data, 84: Learning results, 85: Data, 86: Data, 87: Data, 9 1: Generator, 92: Discriminator, 93: Data, 94: Data, 95: Data Data, 96: Data

Claims

[Claim 1] It has an imaging unit and a processing unit that has the function of performing calculations using machine learning, The imaging unit has the function of acquiring a video, which is a collection of two or more frames of images. The calculation unit has the function of detecting a first object from each of two or more images included in the video. The calculation unit has the function of detecting a second object from each of the first objects that have been detected. The calculation unit has a function to calculate the size of each of the detected second objects. The calculation unit is an information processing device having a function to perform learning using the change in the size of the second object over time.

Images