Information processing device

Abstract

To provide an information processing system having a function to detect fatigue and drowsiness, etc. of a user.SOLUTION: An information processing system includes a first information processing device and a second information processing device and has a function to perform calculation by machine learning. The first information processing device has a function to acquire a first video including a face and to detect eyes from two or more first images included in the first video. The first information processing device also has a function to detect pupils from the eyes detected from the first image, calculate the sizes of the pupils, and learn using a temporal change of the sizes of the pupils. The second information processing device has a function to acquire a second video including a face and to detect eyes from two or more second images included in a second video. The second information processing device also has a function to detect pupils from the eyes detected from the second image, to calculate the sizes of the pupils, and to make an inference on the basis of the results of the learning made by the first information processing device with respect to a temporal change of the sizes of the pupils.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] One aspect of the present invention relates to an information processing apparatus. Another aspect of the present invention relates to an information processing system. Another aspect of the present invention relates to an information processing method. Another aspect of the present invention relates to an information terminal.

Background Art

[0002] When an information terminal such as a smartphone or a tablet is used for a long time, the user may feel fatigue, drowsiness, etc. In particular, when the user stares at the screen of the information terminal for a long time, the user may feel eye fatigue. Patent Document 1 discloses an eye fatigue detection device and a detection method.

[0003] The pupil diameter changes depending on the presence or absence of fatigue, drowsiness, etc. For example, when there is fatigue or drowsiness, the pupil diameter is smaller than when there is no fatigue or drowsiness. Also, generally the pupil diameter fluctuates periodically, but when there is fatigue or drowsiness, the fluctuation period of the pupil diameter is longer than when there is no fatigue or drowsiness.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] During the use of an information terminal such as a smartphone or a tablet, if it is possible to detect the user's fatigue, drowsiness, etc. in real time, for example, it is preferable because the operation of the information terminal can be changed according to the presence or absence of the user's fatigue, drowsiness, etc. When detecting fatigue, drowsiness, etc. in real time, it is preferable that the information terminal itself being used has a function of estimating the user's fatigue, drowsiness, etc. However, in order to detect eye fatigue by the method shown in Patent Document 1, a dedicated device is required.

[0006] One aspect of the present invention aims to provide an information processing device having a function for detecting user fatigue, drowsiness, etc. in real time. Alternatively, one aspect of the present invention aims to provide an information processing device having a function for accurately estimating user fatigue, drowsiness, etc. Alternatively, one aspect of the present invention aims to provide an information processing device having a function for estimating user fatigue, drowsiness, etc. in a simple manner. Alternatively, one aspect of the present invention aims to provide an information processing device having a function for estimating user fatigue, drowsiness, etc. in a short time.

[0007] One aspect of the present invention aims to provide an information processing system having a function for detecting user fatigue, drowsiness, etc. in real time. Alternatively, one aspect of the present invention aims to provide an information processing system having a function for accurately estimating user fatigue, drowsiness, etc. Alternatively, one aspect of the present invention aims to provide an information processing system having a function for estimating user fatigue, drowsiness, etc. in a simple manner. Alternatively, one aspect of the present invention aims to provide an information processing system having a function for estimating user fatigue, drowsiness, etc. in a short time.

[0008] Furthermore, the description of multiple problems does not preclude the existence of each other. One embodiment of the present invention does not need to solve all of the exemplified problems. In addition, problems other than those listed will naturally become apparent from the description herein, and such problems may also be problems addressed by one embodiment of the present invention. [Means for solving the problem]

[0009] One aspect of the present invention is an information processing device comprising an imaging unit and a processing unit having a function to perform calculations by machine learning, wherein the imaging unit has a function to acquire a video which is a collection of two or more frames of images, the processing unit has a function to detect a first object from each of two or more images among the images included in the video, the processing unit has a function to detect a second object from each of the detected first objects, the processing unit has a function to calculate the size of each of the detected second objects, and the processing unit has a function to perform machine learning using the change in the size of the second object over time.

[0010] Furthermore, in the above embodiment, machine learning may be performed using a neural network.

[0011] Furthermore, in the above embodiment, the video may include a face, where the first object is an eye and the second object is a pupil.

[0012] Furthermore, one aspect of the present invention is an information processing device having a function to perform inference based on a learning result obtained by learning using the change in size over time of a first object shown in two or more first images included in a first video, wherein the information processing device has a function to acquire a second video, has a function to detect a second object from each of the two or more second images included in the second video, has a function to detect a third object from each of the detected second objects, has a function to calculate the size of each of the detected third objects, and has a function to perform inference based on the learning result with respect to the change in size over time of the third object.

[0013] Furthermore, in the above embodiment, learning and inference are performed by a neural network, and the learning results may include weight coefficients.

[0014] Furthermore, in the above embodiment, the first video may include a first face, the second video may include a second face, the first and third objects may be pupils, and the second object may be an eye.

[0015] Furthermore, in the above embodiment, the information processing device may have a function for estimating the fatigue of a person having a second face. [Effects of the Invention]

[0016] According to one aspect of the present invention, an information processing device can be provided that has the function of detecting a user's fatigue, drowsiness, etc., in real time. Alternatively, according to one aspect of the present invention, an information processing device can be provided that has the function of estimating a user's fatigue, drowsiness, etc., with high accuracy. Alternatively, according to one aspect of the present invention, an information processing device can be provided that has the function of estimating a user's fatigue, drowsiness, etc., in a simple manner. Alternatively, according to one aspect of the present invention, an information processing device can be provided that has the function of estimating a user's fatigue, drowsiness, etc., in a short time.

[0017] According to one aspect of the present invention, an information processing system can be provided that has the function of detecting a user's fatigue, drowsiness, etc., in real time. Alternatively, according to one aspect of the present invention, an information processing system can be provided that has the function of estimating a user's fatigue, drowsiness, etc., with high accuracy. Alternatively, according to one aspect of the present invention, an information processing system can be provided that has the function of estimating a user's fatigue, drowsiness, etc., in a simple manner. Alternatively, according to one aspect of the present invention, an information processing system can be provided that has the function of estimating a user's fatigue, drowsiness, etc., in a short time.

[0018] Furthermore, the description of multiple effects does not preclude the existence of other effects. Also, one aspect of the present invention does not necessarily have to possess all of the exemplified effects. Moreover, any problems, effects, and novel features of one aspect of the present invention other than those described above will become clear from the description and drawings of this specification. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of an information processing system. [Figure 2] Figure 2 is a flowchart showing an example of how an information processing device operates. [Figure 3] Figure 3 is a flowchart showing an example of an operation method of the information processing apparatus. [Figure 4] Figure 4 is a flowchart showing an example of an operation method of the information processing apparatus. [Figure 5] FIG. 5A, FIG. 5B, and FIG. 5C are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 6] FIG. 6A and FIG. 6B are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 7] FIG. 7A1 and FIG. 7A2, and FIG. 7B1 and FIG. 7B2 are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 8] FIG. 8A and FIG. 8B are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 9] FIG. 9A and FIG. 9B are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 10] FIG. 10A, and FIG. 10B1 and FIG. 10B2 are schematic diagrams showing an example of an operation method of the information processing apparatus. [Figure 11] FIG. 11 is a diagram for explaining AnoGAN that can be applied to one aspect of the present invention.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present invention will be described. However, one aspect of the present invention is not limited to the following description, and it is easily understood by those skilled in the art that the form and details can be variously changed without departing from the spirit and scope of the present invention. Therefore, one aspect of the present invention is not construed as being limited to the description of the embodiments shown below.

[0021] Also, in the drawings attached to this specification, the components are classified by function and shown as block diagrams as independent blocks, but in actuality, it is difficult to completely separate the components by function, and one component may be related to multiple functions, or one function may be realized by multiple components.

[0022] (Embodiment 1) This embodiment describes an information processing system and an information processing method using the information processing system according to one aspect of the present invention. The information processing system and information processing method according to one aspect of the present invention can estimate fatigue, drowsiness, etc., of the user of an information terminal such as a smartphone or tablet. In particular, it can detect eye fatigue of the user of the information terminal.

[0023] <Example of an information processing system configuration> Figure 1 is a block diagram showing an example configuration of an information processing system 10, which is an information processing system according to one aspect of the present invention. The information processing system 10 includes an information processing device 20 and an information processing device 30.

[0024] The information processing device 20 includes an imaging unit 21, a display unit 22, a calculation unit 23, a main memory unit 24, an auxiliary memory unit 25, and a communication unit 26. Data and other information can be transmitted between the components of the information processing device 20 via a transmission line 27. The information processing device 30 also includes an imaging unit 31, a display unit 32, a calculation unit 33, a main memory unit 34, an auxiliary memory unit 35, and a communication unit 36. Data and other information can be transmitted between the components of the information processing device 30 via a transmission line 37.

[0025] The imaging unit 21 and imaging unit 31 have the function of capturing images and acquiring image data. The display unit 22 and display unit 32 have the function of displaying images.

[0026] The arithmetic units 23 and 33 have the function of performing arithmetic processing. The arithmetic unit 23 has the function of performing predetermined arithmetic processing on data transmitted to the arithmetic unit 23 via the transmission line 27 from, for example, the imaging unit 21, the main memory unit 24, the auxiliary memory unit 25, or the communication unit 26. The arithmetic unit 33 has the function of performing predetermined arithmetic processing on data transmitted to the arithmetic unit 33 via the transmission line 37 from, for example, the imaging unit 31, the main memory unit 34, the auxiliary memory unit 35, or the communication unit 36. In addition, the arithmetic units 23 and 33 have the function of performing calculations using machine learning. For example, they have the function of performing calculations using a neural network. The arithmetic units 23 and 33 may have, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0027] The main memory units 24 and 34 have the function of storing data and programs. The arithmetic unit 23 can read data and programs stored in the main memory unit 24 and execute arithmetic processing. For example, the arithmetic unit 23 can perform predetermined arithmetic processing on data read from the main memory unit 24 by executing a program read from the main memory unit 24. Similarly, the arithmetic unit 33 can read data and programs stored in the main memory unit 34 and execute arithmetic processing. For example, the arithmetic unit 33 can perform predetermined arithmetic processing on data read from the main memory unit 34 by executing a program read from the main memory unit 34.

[0028] The main memory units 24 and 34 are preferably configured to operate at a higher speed than the auxiliary memory units 25 and 35. The main memory units 24 and 34 may include, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), etc.

[0029] The auxiliary storage units 25 and 35 have the function of storing data and programs for a longer period than the main memory units 24 and 34. The auxiliary storage units 25 and 35 may have, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. Furthermore, the auxiliary storage units 25 and 35 may have non-volatile memory such as ReRAM (Resistive Random Access Memory), PRAM (Phase Change Random Access Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Random Access Memory), or flash memory.

[0030] The communication unit 26 has the function of sending and receiving data to and from devices located outside the information processing device 20. The communication unit 36 ​​has the function of sending and receiving data to and from devices located outside the information processing device 30. For example, by supplying data from the communication unit 26 to the communication unit 36, data can be supplied from the information processing device 20 to the information processing device 30. Furthermore, the communication units 26 and 36 may have the function of supplying data to a network and the function of acquiring data from a network.

[0031] Here, if the arithmetic unit 23 and the arithmetic unit 33 have the function of performing calculations using machine learning, for example, the arithmetic unit 23 can perform learning and supply the learning results from the information processing device 20 to the information processing device 30. For example, if the arithmetic unit 23 and the arithmetic unit 33 have the function of performing calculations using a neural network, the arithmetic unit 23 can perform learning to obtain weight coefficients, etc., and supply said weight coefficients, etc., from the information processing device 20 to the information processing device 30. As a result, even if the arithmetic unit 33 provided in the information processing device 30 does not perform learning, inference can be performed on the data input to the arithmetic unit 33 based on the learning results of the arithmetic unit 23 provided in the information processing device 20. Therefore, the arithmetic processing capability of the arithmetic unit 33 can be lower than that of the arithmetic unit 23.

[0032] When the calculation unit 23 performs learning and supplies the learning results from the information processing device 20 to the information processing device 30, the information processing device 20 can be installed, for example, on a server. Note that if the information processing device 20 is installed on a server, the imaging unit 21 and the display unit 22 do not need to be installed in the information processing device 20. In other words, the imaging unit 21 and the display unit 22 may be installed outside the information processing device 20.

[0033] Furthermore, the information processing device 30 can be installed in an information terminal such as a smartphone, tablet, or personal computer. Alternatively, at least some of the components of the information processing device 20 and at least some of the components of the information processing device 30 may be installed on a server. For example, the arithmetic unit 23 and the arithmetic unit 33 may be installed on the server. In this case, for example, data acquired by the information terminal is supplied to the arithmetic unit 33 via the network, and the arithmetic unit 33 installed on the server performs inference on the data. The inference results are then supplied to the information terminal via the network, allowing the information terminal to obtain the inference results.

[0034] <An example of an information processing method> The following describes an example of an information processing method using the information processing system 10. Specifically, it describes an example of a method for estimating fatigue, drowsiness, etc., of a user of an information terminal equipped with the information processing device 30 of the information processing system 10, using calculations based on machine learning.

[0035] Figures 2 and 3 are flowcharts illustrating an example of a method for estimating fatigue, drowsiness, etc., using machine learning calculations. The learning process is shown in Figure 2, and the inference process is shown in Figure 3.

[0036] An example of a learning method will be explained using Figure 2, etc. First, the imaging unit 21 captures a video. For example, it captures a video that includes a human face (step S01). Here, a video refers to a set of images of two or more frames. As will be described in detail later, the imaging unit 21 creates learning data based on the video it captures, and the calculation unit 23 performs learning. Therefore, for example, when the imaging unit 21 captures a video that includes a human face, it is preferable for the imaging unit 21 to capture videos of a large number of people with different genders, races, body types, etc.

[0037] Furthermore, image processing may be performed on the video captured by the imaging unit 21. For example, noise reduction, grayscale conversion, normalization, contrast adjustment, etc., can be performed. In addition, binarization, etc., may be performed on the images included in the video. By performing such processing, subsequent steps can be carried out with greater accuracy. For example, the detection of the first object performed in step S02, described later, can be carried out with greater accuracy.

[0038] Next, the processing unit 23 detects a first object from each of the captured images. The first object can be, for example, an eye if a video of a face was captured in step S01 (step S02). The first object can be detected by, for example, a cascade classifier. For example, it can be detected by Haar Cascades. If the first object is an eye and both eyes are included in one image, only one eye can be detected.

[0039] Subsequently, the calculation unit 23 detects a second object from each of the detected first objects. For example, if the first object is an eye, the second object can be the pupil (step S03). For example, the pupil can be detected from the eye by circular extraction. Details of the method for detecting the pupil from the eye will be described later.

[0040] Here, the pupil is the opening surrounded by the iris and can be called the "black of the eye." The pupil has the function of adjusting the amount of light projected onto the retina. The iris, on the other hand, is a thin membrane, for example, between the cornea and the lens, and can be considered the colored part of the eye.

[0041] Next, the calculation unit 23 calculates the size of each of the detected second objects (step S04). For example, if the second object is detected by circular extraction, the radius or diameter of the second object can be used as the size of the second object. If the shape of the second object is extracted as an ellipse, the length of the major axis and the length of the minor axis can be used as the size of the second object. Alternatively, the area of ​​the second object can be used as the size of the second object.

[0042] Subsequently, the calculation unit 23 performs learning using the size of the second object and obtains the learning results (step S05). Specifically, the learning results are obtained based on the change in the size of the second object over time. Learning can be performed, for example, using a neural network. In this case, the learning results can be weight coefficients, etc., as described above. Details of the learning method will be described later.

[0043] Next, the information processing device 20 supplies the learning results to the information processing device 30 (step S06). Specifically, the learning results acquired by the arithmetic unit 23 are transmitted to the communication unit 26 via the transmission line 27, and then supplied from the communication unit 26 to the communication unit 36. The learning results supplied to the communication unit 36 ​​can be stored in the auxiliary storage unit 35. Alternatively, the learning results may be stored in the auxiliary storage unit 25.

[0044] Next, an example of an inference method based on learning results obtained by the method shown in Figure 2 will be explained using Figure 3 and other figures. First, the imaging unit 31 captures a video. For example, it captures a video including the face of the user of the information terminal equipped with the information processing device 30 (step S11). Note that if image processing is performed on the video captured by the imaging unit 21 in step S01 shown in Figure 2, it is preferable to perform the same image processing on the video captured by the imaging unit 31, as this allows for more accurate inference.

[0045] Next, the calculation unit 33 detects a first object from each image contained in the captured video. The first object can be, for example, an eye if a video of a face was captured in step S11 (step S12). The first object can be detected in the same way as the detection method used in step S02 shown in Figure 2.

[0046] Subsequently, the calculation unit 33 detects a second object from each of the first objects that were detected. For example, if the first object is an eye, the second object can be a pupil (step S13). The second object can be detected in the same way as the detection method used in step S03 shown in Figure 2.

[0047] Next, for each of the detected second objects, the calculation unit 33 calculates the size of the second object (step S14). The method for calculating the size is the same as the method used in step S04 shown in Figure 2.

[0048] Subsequently, based on the change in the size of the second object over time, the calculation unit 33, which receives the learning results acquired by the calculation unit 23 in step S05 shown in Figure 2, performs inference. For example, if the video captured by the imaging unit 31 includes the face of the user of the information terminal equipped with the information processing device 30, and the second object is the pupil of the user's eye, the calculation unit 33 can estimate the user's fatigue, drowsiness, etc. (step S15). Details of the inference method will be described later.

[0049] Furthermore, pupil size changes not only due to fatigue and drowsiness, but also, for example, the brightness of the environment. Therefore, in step S01 shown in Figure 2, it is preferable to capture multiple videos of the same person's face at different ambient brightness levels. This makes it possible to accurately estimate the fatigue, drowsiness, etc., of the user of the information terminal equipped with the information processing device 30, regardless of the ambient brightness.

[0050] In one aspect of the present invention, an information processing device 30 having the function of estimating fatigue, drowsiness, etc., as described above, is installed in an information terminal such as a smartphone, tablet, or personal computer. This makes it possible to detect the fatigue, drowsiness, etc., of the user of the information terminal in real time without using a dedicated device.

[0051] [An example of a method for detecting the pupil] Next, an example of a pupil detection method performed in steps S03 and S13 will be described. Figure 4 is a flowchart of an example of a pupil detection method.

[0052] First, the calculation unit acquires image 41, which is an image containing the detected eye (step S31). Figure 5A is a schematic diagram illustrating step S31. As shown in Figure 5A, the calculation unit acquires image 41, which contains the detected eye, from the image captured by the imaging unit. Specifically, in step S03, the calculation unit 23 acquires the image containing the eye detected by the calculation unit 23 in step S02 as image 41. Also, in step S13, the calculation unit 33 acquires the image containing the eye detected by the calculation unit 33 in step S12 as image 41. Note that if the image captured by the imaging unit is a color image, the calculation unit may convert image 41 to grayscale after acquiring it.

[0053] Next, the calculation unit performs an expansion process on image 41 to obtain image 42, and then performs a contraction process to obtain image 43 (step S32). In other words, image 43 is obtained by performing a closing process on image 41. Figure 5B is a schematic diagram illustrating the expansion and contraction processes.

[0054] Subsequently, the calculation unit subtracts image 43 from image 41 to obtain image 44 (step S33). In other words, image 44 is an image represented by the difference between image 41 and image 43. In step S33, the calculation unit obtains image 44 by performing a Black-hat transform using image 41 and image 43.

[0055] Next, the calculation unit adds the image 41 acquired in step S31 and the image 44 acquired in step S33 to obtain image 45 (step S34). If image 41 is converted to grayscale in step S31, then in step S34, the grayscale image 41 and image 44 can be added together.

[0056] Furthermore, the processes shown in steps S32 to S34 may be omitted in whole or in part. In addition, processes other than those shown in steps S32 to S34 may be performed.

[0057] Subsequently, the processing unit performs image processing on image 45 and obtains image 46 (step S35). For example, the processing unit performs noise reduction, smoothing, and other processing on image 45. It also performs edge detection and binarization. Specifically, for example, noise reduction using an intermediate value filter and smoothing using a Gaussian filter are performed on image 45, followed by edge detection using the Canny method and binarization. Noise reduction may be performed using, for example, a moving average filter. Smoothing may be performed using, for example, a moving average filter or a median filter. Furthermore, edge detection may be performed using, for example, a Laplacian filter.

[0058] Next, the calculation unit detects the iris 47 from image 46. For example, the iris 47 can be detected by using the Hough transform. When using the Hough transform, the iris 47 can be detected as a circle, for example. Alternatively, the iris 47 can be detected as an ellipse, for example. Note that the iris 47 may also be detected using a generalized Hough transform.

[0059] Then, the calculation unit acquires an image 49 that includes the detected iris 47 (step S36). For example, based on the coordinates of the detected iris 47 in image 46, image 49 is extracted from image 46.

[0060] Figure 5C is a schematic diagram illustrating step S36. For example, as shown in Figure 5C, image 49 can be a rectangle with all four sides touching the iris 47. For example, if the iris 47 is detected in a circular shape, image 49 can be a square with all four sides touching the iris 47. Note that each side of image 49 does not necessarily have to touch the iris 47. For example, an image with a predetermined number of pixels centered on the iris 47 may be used as image 49.

[0061] Next, the processing unit detects the pupil 48 from the image 49 (step S37). For example, the pupil 48 is detected from the image 49 by a calculation using a neural network.

[0062] Step S37 is performed using a pre-trained generator. Here, the generator is a program that performs calculations using machine learning and has the function of outputting data corresponding to the input data. Specifically, by training, the generator can perform inference on the data input to it.

[0063] Figure 6A is a schematic diagram illustrating the learning process described above. Here, the generator performing the learning is referred to as generator 50. When a neural network is used as generator 50, generator 50 can be a convolutional neural network (CNN). In particular, it is preferable to use a U-net as a type of CNN. In a U-net, the input image is downsampled by convolution, and then upsampled by deconvolution using the features obtained by downsampling. The arithmetic units 23 and 33 can be said to have the function of generator 50.

[0064] The generator 50 can be trained using supervised learning with data 51 and data 52. Data 51 can be a set of images 59. Images 59 include the iris 57 and pupil 58. Images 59 can be acquired by the information processing device 20 in the same manner as steps S01 and S02 shown in Figure 2, and steps S31 to S36 shown in Figure 4. In step S01, the imaging unit 21 is assumed to capture a video of the face, but when acquiring images 59, it is not necessary to capture a video. For example, the imaging unit 21 may capture one image (one frame) per person.

[0065] Data 52 is data indicating the coordinates of the pupil 58 contained in image 59. Specifically, it can be a binary image in which the color of the pupil 58 is different from the color of the other parts. Data 52 can be obtained, for example, by filling in the pupil 58 contained in image 59. Alternatively, after obtaining an image containing an eye in the same manner as in step S31 shown in Figure 4, data 52 can be obtained by filling in the pupil 58 in the image containing the eye.

[0066] The generator 50 is trained so that when data 51 is input to the generator 50, the output data approaches data 52. In other words, the generator 50 is trained using data 52 as the ground truth data. Through this training, the generator 50 generates the training result 53. If a neural network is used as the generator 50, the training result 53 can be the weight coefficients, etc.

[0067] The learning of the generator 50, that is, the generation of the learning result 53, can be performed, for example, by the arithmetic unit 23 of the information processing device 20. By supplying the learning result 53 from the information processing device 20 to the information processing device 30, the arithmetic unit 33 can also perform the same inference as the arithmetic unit 23. The learning result 53 generated by the arithmetic unit 23 can be stored, for example, in the auxiliary storage unit 25. Also, the learning result 53 generated by the arithmetic unit 23 and supplied to the information processing device 30 can be stored, for example, in the auxiliary storage unit 35.

[0068] This completes the training of generator 50.

[0069] Figure 6B is a schematic diagram illustrating step S37. In other words, Figure 6B is a schematic diagram illustrating the detection of the pupil 48 from image 49.

[0070] As shown in Figure 6B, in step S37, the image 49 acquired by the calculation unit in step S36 is input to the generator 50, which has the learning result 53 loaded into it. As a result, the generator 50 can perform inference on the image 49 and output data indicating the coordinates of the pupil 48. For example, the generator 50 can output a binary image in which the color of the pupil 48 is different from the color of the rest of the image.

[0071] By the above method, the pupil can be detected in step S03 or step S13 from among the eyes detected in step S02 or step S12.

[0072] By using machine learning to detect the pupil, it is possible to detect the pupil in a shorter time than, for example, by visual inspection. Furthermore, even if the surrounding scenery is reflected in the pupil, the pupil can be detected with high accuracy.

[0073] The method for detecting the pupil 48 in step S37 is not limited to the method shown in Figures 6A and 6B. For example, the image 49 may be a color image, and the color image 49 may be converted to grayscale before performing edge detection of the pupil 48. Then, the pupil 48 may be detected after edge detection.

[0074] The grayscale conversion of image 49 can be performed, for example, using Partial Least Squares (PLS) regression. By converting image 49 to grayscale, the difference between the brightness of the pupil 48 and the brightness of the iris 47 can be increased. This allows the boundary between the pupil 48 and the iris 47 to be emphasized, enabling accurate edge detection of the pupil 48. Therefore, the pupil 48 can be detected with high accuracy.

[0075] The edge detection of the pupil 48 can be performed, for example, by the Canny method or a Laplacian filter. Furthermore, the detection of the pupil 48 after edge detection can be performed, for example, by using the Hough transform. When using the Hough transform, the pupil 48 can be detected as, for example, a circle, or as an ellipse. Alternatively, the pupil 48 may be detected using a generalized Hough transform.

[0076] If the pupil is to be detected in step S03 or step S13, imaging can be performed using infrared light in step S01 or step S11. The iris reflects infrared light, while the pupil does not. Therefore, by performing imaging using infrared light in step S01 or step S11, the iris and pupil can be clearly distinguished. Consequently, the pupil can be detected with high accuracy.

[0077] [Example of a method for estimating fatigue, drowsiness, etc._1] Next, we will describe an example of a method for estimating fatigue, drowsiness, etc., of a user of an information terminal equipped with an information processing device 30, using machine learning calculations. Specifically, we will describe an example of a learning method using pupil size, which is performed in step S05. We will also describe an example of a method for estimating fatigue, drowsiness, etc., by inference based on the above learning results, which is performed in step S15. In the following, the second object will be described as the pupil.

[0078] Figure 7A1 is a schematic diagram illustrating step S05. In step S05, the generator 60, a program that performs calculations using machine learning, is trained. The generator 60 can use a neural network. As will be explained in detail later, time-series data, such as the change in pupil size over time, is input to the generator 60. Therefore, when using a neural network as the generator 60, it is preferable to use a recurrent neural network (RNN). Alternatively, it is preferable to use long short-term memory (LSTM) as the generator 60. Alternatively, it is preferable to use a gated recurrent unit (GRU).

[0079] The generator 60 can be trained using data 61 and data 62. Data 61 is the data acquired in step S04 and can represent the change in pupil size over time. As mentioned above, for example, if the pupil is detected by circular extraction, the radius or diameter of the pupil can be used as the pupil size. If the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be used as the pupil size. Alternatively, the area of ​​the pupil can be used as the pupil size. In Figure 7A1, data 61 represents the change in pupil size over time from time 1 to n-1 (where n is an integer greater than or equal to 3).

[0080] Data 61 may also be the time-dependent change in the ratio of pupil size to iris size. In this case, it is preferable to extract the iris and pupil into the same type of shape. For example, if the iris is extracted as a circle, it is preferable to extract the pupil as a circle as well. Also, if the iris is extracted as an ellipse, it is preferable to extract the pupil as an ellipse as well. By using the time-dependent change in the ratio of pupil size to iris size as data 61, when detecting the pupil using the method shown in step S37, for example, the resolution of the image 49 including the iris 47 and the pupil 48 can be made different from each other. For example, the resolution of the image 49 including the iris 47 and pupil 48 of a first person and the resolution of the image 49 including the iris 47 and pupil 48 of a second person can be made different from each other.

[0081] Data 62 represents the pupil size at time n. In other words, it is the pupil size at a time later than the time at which the pupil size included in Data 61 was measured. If Data 61 represents the change over time in the ratio of pupil size to iris size, then Data 62 will also represent the ratio of pupil size to iris size.

[0082] Figure 7A2 shows an example of the relationship between pupil diameter and time. In Figure 7A2, the black circles indicate the measured pupil diameter. In other figures as well, measured points may be indicated by black circles. As shown in Figure 7A2, data 62 can be the pupil size at a time later than the time when the pupil size included in data 61 was measured. For example, data 62 can be the pupil size at the time immediately following the last time when the pupil size included in data 61 was measured.

[0083] If the generator 60 is given the function to estimate the presence or absence of fatigue, the change in pupil size over time for fatigued individuals will not be included in data 61 and data 62. In other words, data 61 will be the change in pupil size over time for non-fatigued individuals, and data 62 will be the pupil size for non-fatigued individuals. Also, if the generator 60 is given the function to estimate the presence or absence of drowsiness, the change in pupil size over time for drowsy individuals will not be included in data 61 and data 62. In other words, data 61 will be the change in pupil size over time for non-drowsy individuals, and data 62 will be the pupil size for non-drowsy individuals.

[0084] The generator 60 is trained so that when data 61 is input to the generator 60, the output data approaches data 62. In other words, the generator 60 is trained using data 62 as the ground truth data. Through this training, the generator 60 generates the training result 63. If a neural network is used as the generator 60, the training result 63 can be the weight coefficients, etc.

[0085] Figures 7B1 and 7B2 are schematic diagrams illustrating step S15, illustrating an example of a method for estimating fatigue, drowsiness, etc., of a user of an information terminal equipped with an information processing device 30 using a generator 60. In step S15, first, as shown in Figure 7B1, data 64 showing the change in pupil size over time, acquired in step S14, is input to the generator 60, which has the learning result 63 loaded. For example, if the change in pupil size over time from time 1 to n-1 was used as input data during the learning of the generator 60 in step S05, then the change in pupil size over time from time 1 to n-1 is also used as input data during inference in step S15. In other words, data 64 is set to the change in pupil size over time of a user of an information terminal equipped with an information processing device 30, from time 1 to n-1. As a result, the generator 60 performs inference on data 64 and outputs data 65. Furthermore, using data 65, which is the inference data at time n, the data from times 2 to n may be used as input data, and the data at time n+1 may be inferred.

[0086] If data 61 is the time-dependent change in the ratio of pupil size to iris size, then data 64 will also be the time-dependent change in the ratio of pupil size to iris size. By making data 64 the time-dependent change in the ratio of pupil size to iris size, for example, when detecting the pupil using the method shown in step S37, the resolution of the image 49 including the iris 47 and pupil 48 can be made different from each other. For example, the resolution of the image 49 acquired by the calculation unit to calculate the ratio of pupil size 48 to iris size 47 at time 1 can be made different from the resolution of the image 49 acquired by the calculation unit to calculate the ratio of pupil size 48 to iris size 47 at time n-1.

[0087] Data 65 is an estimate of pupil size at a time after the time when the pupil size included in Data 64 was measured, calculated by performing inference on Data 64 based on the learning result 63. For example, if Data 64 is the change in pupil size over time from time 1 to n-1, then Data 65 can be the pupil size at time n. In Figure 7B1, the measured values ​​of pupil size at time 1 to n-1 are x1 to x, respectively. n-1 It states that the estimated pupil size at time n is given by x n (E) is indicated. If data 64 represents the change over time in the ratio of pupil size to iris size, then data 65 represents the ratio of pupil size to iris size.

[0088] Next, as shown in Figure 7B2, data 66, which represents, for example, the measured value of pupil size at time n, is compared with data 65, which is the data output from generator 60. In other words, for example, the measured value of pupil size at time n is compared with the estimated value. Based on the comparison result, the presence or absence of fatigue, drowsiness, etc., is estimated. For example, if generator 60 has a function to estimate the presence or absence of fatigue, generator 60 is trained using, for example, the change in pupil size over time of a person without fatigue as input data. Therefore, if the user of the information terminal equipped with the information processing device 30 is not fatigued, data 65 will be close to data 66. In other words, the difference between data 65 and data 66 will be small. On the other hand, if the user of the information terminal equipped with the information processing device 30 is fatigued, the difference between data 66 and data 65 will be larger than when the user of the information terminal equipped with the information processing device 30 is not fatigued. From the above, by comparing data 66 and data 65, it is possible to estimate the presence or absence of fatigue in the user of the information terminal equipped with the information processing device 30. The same applies when estimating the presence or absence of drowsiness. If data 65 is used as an estimated value of the ratio of pupil size to iris size, then data 66 will be used as the measured value of the ratio of pupil size to iris size.

[0089] The function of the generator 60 can be provided to both the arithmetic unit 23 and the arithmetic unit 33. In this case, the arithmetic unit 23 of the information processing device 20 can learn the generator 60 and generate the learning result 63, and the learning result 63 can be supplied from the information processing device 20 to the information processing device 30. As a result, even if the arithmetic unit 33 provided in the information processing device 30 does not perform learning, inference can be performed on the data input to the arithmetic unit 33 based on the learning result from the arithmetic unit 23 provided in the information processing device 20. Therefore, the arithmetic processing capability of the arithmetic unit 33 can be lower than that of the arithmetic unit 23. The learning result 63 can be stored in the auxiliary storage unit 25 and the auxiliary storage unit 35.

[0090] [An example of a method for estimating fatigue, drowsiness, etc._2] Figures 8A and 8B are schematic diagrams illustrating step S05 and are examples of generator learning methods different from the method described above. Specifically, Figure 8A shows an example of how to create data to be input to the generator as training data, and Figure 8B shows an example of how to learn generator 80. Generator 80 is a program that performs calculations using machine learning. For example, a neural network can be used as generator 80.

[0091] The data 81 shown in Figure 8A is the data acquired in step S04 and can represent the change in pupil size over time. As mentioned above, if the pupil is detected by circular extraction, for example, the radius or diameter of the pupil can be used as the pupil size. If the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be used as the pupil size. Alternatively, the area of ​​the pupil can be used as the pupil size. Similar to the learning method shown in Figure 7A1, the data 81 can represent the change in the ratio of pupil size to iris size over time.

[0092] Here, data 82 is generated by performing a Fourier transform on data 81. As shown in Figure 8A, the Fourier transform can convert the change in pupil diameter over time into the frequency characteristics of pupil diameter. If data 81 represents the change in the ratio of pupil size to iris size over time, then data 82 can be the frequency characteristics of the ratio of pupil size to iris size.

[0093] The generator 80 can be trained using data 82 and data 83, as shown in Figure 8B. Data 82 represents the frequency characteristics of pupil diameter, as described above. Data 83 can be a label indicating the presence or absence of fatigue. For example, data 83 can include both the frequency characteristics of pupil size of fatigued individuals and the frequency characteristics of pupil size of non-fatigued individuals. The frequency characteristics of pupil size of fatigued individuals can be associated with the label "fatigued," and the frequency characteristics of pupil size of non-fatigued individuals can be associated with the label "non-fatigued." Data 83 may also be a label indicating the presence or absence of drowsiness.

[0094] The generator 80 is trained so that when data 82 is input to the generator 80, the output data approaches data 83. In other words, the generator 80 is trained using data 83 as the ground truth data. Through this training, the generator 80 generates a training result 84. If a neural network is used as the generator 80, the training result 84 can be used as weight coefficients, etc.

[0095] By performing a Fourier transform on the change in pupil size over time, the data input to the generator 80 can be changed from time-series data to non-time-series data. As a result, the generator 80 can perform learning and inference even without using an RNN as the generator 80.

[0096] Figures 9A and 9B are schematic diagrams illustrating step S15, and show an example of a method for estimating fatigue, drowsiness, etc., of a user of an information terminal equipped with an information processing device 30, using a generator 80.

[0097] The data 85 shown in Figure 9A is the data acquired in step S04 and can represent the change in pupil size over time. As mentioned above, if the pupil is detected by circular extraction, for example, the radius or diameter of the pupil can be used as the pupil size. If the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be used as the pupil size. Alternatively, the area of ​​the pupil can be used as the pupil size. If the data 81 shown in Figure 8A represents the change in the ratio of pupil size to iris size over time, then data 85 should also represent the change in the ratio of pupil size to iris size over time.

[0098] Here, data 86 is generated by performing a Fourier transform on data 85. As shown in Figure 9A, the Fourier transform can convert the change in pupil diameter over time into the frequency characteristics of pupil diameter. If data 85 represents the change in the ratio of pupil size to iris size over time, then data 86 can be the frequency characteristics of the ratio of pupil size to iris size.

[0099] Then, as shown in Figure 9B, the Fourier-transformed data 86 is input to the generator 80. This allows the generator 80 to perform inference on the data 86 and output data 87 indicating the presence or absence of fatigue. If the data 87 shown in Figure 9B is a label indicating the presence or absence of drowsiness, then the data 87 output by the generator 80 can be data indicating the presence or absence of drowsiness.

[0100] The function of generator 80, like the functions of generator 60 and generator 70, can be provided to both the arithmetic unit 23 and the arithmetic unit 33. This allows the arithmetic processing power of the arithmetic unit 33 to be lower than that of the arithmetic unit 23.

[0101] [Example of a method for estimating fatigue, drowsiness, etc._3] Figure 10A is a schematic diagram illustrating step S05, and is an example of a generator learning method different from the method described above. In Figure 10A, the generator 70 is trained. The generator 70 is a program that performs calculations using machine learning. For example, a neural network can be used as the generator 70, or an autoencoder can be used.

[0102] When the generator 70 performs learning, data 71 is input to the generator 70. Data 71 is the data obtained in step S04 and can be the change in pupil size over time. If the generator 70 is given the function to estimate the presence or absence of fatigue, the change in pupil size over time of fatigued individuals is not included in data 71. In other words, data 71 will be the change in pupil size over time of non-fatigued individuals. Also, if the generator 70 is given the function to estimate the presence or absence of drowsiness, the change in pupil size over time of drowsy individuals is not included in data 71. In other words, data 71 will be the change in pupil size over time of non-drowsy individuals.

[0103] As mentioned above, if the pupil is detected by circular extraction, for example, the radius or diameter of the pupil can be used as the pupil size. If the pupil is detected as an ellipse, the length of the major axis and the length of the minor axis can be used as the pupil size. Alternatively, the area of ​​the pupil can be used as the pupil size.

[0104] Furthermore, similar to the learning method shown in Figure 7A1, data 71 can be the time-dependent change in the ratio of pupil size to iris size. Also, as in the case shown in Figure 8A, for example, the Fourier transform of the time-dependent change in pupil size may be used as data 71.

[0105] The generator 70 is trained so that when data 71 is input to the generator 70, the output data 72 approaches the input data 71. In other words, the generator 70 is trained so that data 71 and data 72 become equal. Through this training, the generator 70 generates a training result 73. When a neural network is used as the generator 70, the training result 73 can be a weight coefficient or the like.

[0106] Figures 10B1 and 10B2 are schematic diagrams illustrating step S15, illustrating an example of a method for estimating fatigue, drowsiness, etc., of a user of an information terminal equipped with an information processing device 30, using a generator 70. In step S15, first, as shown in Figure 10B1, the data 74 showing the change in pupil size over time, acquired in step S14, is input to the generator 70, which has the learning result 73 loaded. As a result, the generator 70 performs inference on the data 74 and outputs data 75.

[0107] If data 71 represents the change over time of the ratio of pupil size to iris size, then data 74 should also represent the change over time of the ratio of pupil size to iris size. Furthermore, if the Fourier transformed data is used as data 71, then the Fourier transformed data should also be used for data 74. For example, if data 71 is the Fourier transformed result of the change over time of pupil size, then data 74 should also be the Fourier transformed result of the change over time of pupil size.

[0108] Next, as shown in Figure 10B2, data 74, which is the data input to the generator 70, and data 75, which is the data output from the generator 70, are compared. Based on the comparison result, the presence or absence of fatigue, drowsiness, etc., is estimated. For example, if the generator 70 has a function to estimate the presence or absence of fatigue, the generator 70 is trained using, for example, the change in pupil size over time of a person without fatigue. Therefore, if the user of the information terminal equipped with the information processing device 30 is not fatigued, data 75, which is the output data from the generator 70, will be close to data 74, which is the input data to the generator 70. In other words, the difference between data 74 and data 75 will be small. On the other hand, if the user of the information terminal equipped with the information processing device 30 is fatigued, the difference between data 74 and data 75 will be larger than when the user of the information terminal equipped with the information processing device 30 is not fatigued. From the above, by comparing data 74 and data 75, it is possible to estimate the presence or absence of fatigue in the user of the information terminal equipped with the information processing device 30. The same applies when estimating the presence or absence of drowsiness.

[0109] The function of the generator 70 can be provided to both the arithmetic unit 23 and the arithmetic unit 33, similar to the function of the generator 60. This allows the arithmetic processing power of the arithmetic unit 33 to be lower than that of the arithmetic unit 23.

[0110] [Example of a method for estimating fatigue, drowsiness, etc._4] The learning performed in step S05 and the inference based on the learning results performed in step S15 may be carried out using a Generative Adversarial Network (GAN). For example, AnoGAN (Anormally GAN) may be used. Figure 11 is a diagram illustrating an AnoGAN capable of performing the above learning and inference.

[0111] The AnoGAN shown in Figure 11 comprises a generator 91 and a discriminator 92. The generator 91 and the discriminator 92 can be constructed using neural networks.

[0112] The discriminator 92 receives data 93, which is time-series data representing the change over time of pupil size in a person who is not fatigued, sleepy, etc., obtained by imaging. Alternatively, the discriminator 92 receives data 95, which is time-series data generated by the generator 91, which receives data 94 as input. The discriminator 92 has the function of determining (also called authenticity determination) whether the input data is data 93 obtained by imaging or data 95 generated by the generator 91. Data 93 may be data obtained by Fourier transforming time-series data representing the change over time of pupil size in a person who is not fatigued, sleepy, etc., obtained by imaging.

[0113] The judgment result is output as data 96. Data 96 can be a continuous value between 0 and 1, for example. In this case, for example, after the discriminator 92 has finished learning, if the input data is data 93 acquired by imaging, it should output a value close to 1 as data 96, and if the input data is data 95 generated by the generator 91, it should output a value close to 0 as data 96.

[0114] Data 94 is a multidimensional random number (also called a latent variable). Here, the latent variable represented by Data 94 is denoted as latent variable z. Generator 91 has the function of generating data that is as similar as possible to data representing the change in pupil size over time for a person who is not fatigued, sleepy, etc., based on such Data 94.

[0115] The learning process alternates between training the discriminator 92 and training the generator 91. Specifically, the weight coefficients of the neural network constituting the generator 91 are fixed during the training of the discriminator 92. Similarly, the weight coefficients of the neural network constituting the discriminator 92 are fixed during the training of the generator 91.

[0116] During the training of the discriminator 92, data 93 acquired by imaging or data 95 generated by the generator 91 is input to the discriminator 92. A correct label is assigned to the data input to the discriminator 92. The correct label can be determined for the data 96 output by the discriminator 92 as follows. For example, if data 93 is input to the discriminator 92, the correct label is set to "1", and if data 95 is input to the discriminator 92, the correct label is set to "0". By training in this manner, the discriminator 92 becomes capable of determining authenticity.

[0117] During the training of generator 91, data 94 representing the latent variable z is input to generator 91. Then, generator 91 generates data 95 based on the input data 94. The correct label for data 96 is set to "1". Then, generator 91 is trained so that the value of data 96 output from discriminator 92 is "1". As generator 91's training progresses, it becomes able to generate data 95 that is similar to data 93 acquired by imaging.

[0118] Once the generator 91 has finished training, it will be able to generate data 95 that is similar to the data 93 obtained by imaging, regardless of what latent variable z is input as data 94.

[0119] Next, I will explain the process during inference.

[0120] First, let's assume that data representing the temporal change in pupil size of individuals without fatigue or drowsiness is obtained through imaging. At this time, we search the space of latent variables and find a latent variable z1 that generates data most similar to the above data on pupil size of individuals without fatigue or drowsiness using gradient descent or similar methods. The generator 91 has the function of generating data that is extremely similar to the data representing the temporal change in pupil size of individuals without fatigue or drowsiness through learning. Therefore, the data generated by the generator from the latent variable z1 and the data representing the temporal change in pupil size of individuals without fatigue or drowsiness obtained through imaging will be extremely similar.

[0121] Next, let's assume that imaging has obtained data representing the temporal change in pupil size of individuals experiencing fatigue, drowsiness, etc. At this time, we search the space of latent variables and find a latent variable z2 that generates data closest to the data representing the temporal change in pupil size of the individuals experiencing fatigue, drowsiness, etc., using methods such as gradient descent. Through learning, the generator 91 has the ability to generate data that is very close to the data representing the temporal change in pupil size of individuals without fatigue, drowsiness, etc., but it does not have the ability to generate data similar to the data representing the temporal change in pupil size of individuals experiencing fatigue, drowsiness, etc. Therefore, the data generated by the generator from the latent variable z2 and the data representing the temporal change in pupil size of the individuals experiencing fatigue, drowsiness, etc., obtained through imaging will not be very similar. Thus, the presence or absence of fatigue, drowsiness, etc., can be estimated using the generator 91.

[0122] As shown in Figures 7 to 11 above, the information processing device 30 estimates the fatigue, drowsiness, etc., of the user of the information terminal on which the information processing device 30 is installed, using calculations based on machine learning. By using machine learning, fatigue, drowsiness, etc., can be estimated with high accuracy without having to manually set, for example, the temporal changes in feature quantities when fatigue is estimated to be present and the temporal changes in feature quantities when fatigue is estimated to be absent. Specifically, fatigue, drowsiness, etc., can be estimated with high accuracy without having to manually set, for example, the temporal changes in pupil size when fatigue is estimated and the temporal changes in pupil size when fatigue is estimated and the temporal changes in pupil size when fatigue is estimated and the temporal changes in pupil size when fatigue is estimated and the temporal changes in feature quantities are absent.

[0123] If the information processing device 30 estimates that the user is experiencing fatigue, drowsiness, etc., it can display an alarm on the display unit of the information terminal equipped with the information processing device 30, indicating that fatigue, drowsiness, etc., is occurring. This allows the user of the information terminal to be prompted to stop using the terminal as soon as possible, or the power to the information terminal equipped with the information processing device 30 to be turned off. This helps to prevent health problems caused by the user continuing to use the information terminal despite experiencing fatigue, drowsiness, etc. [Explanation of Symbols]

[0124] 10: Information processing system, 20: Information processing device, 21: Imaging unit, 22: Display unit, 23: Calculation unit, 24: Main memory unit, 25: Auxiliary memory unit, 26: Communication unit, 27: Transmission line, 30: Information processing device, 31: Imaging unit, 32: Display unit, 33: Calculation unit, 34: Main memory unit, 35: Auxiliary memory unit, 36: Communication unit, 37: Transmission line, 41: Image, 42: Image, 43: Image, 44: Image, 45: Image, 46: Image, 47: Iris, 48: Pupil, 49: Image, 50: Generator, 51: Data, 52: Data, 53: Learning result, 57 : Iris, 58: Pupil, 59: Image, 60: Generator, 61: Data, 62: Data, 63: Learning result, 64: Data, 65: Data, 66: Data, 70: Generator, 71: Data, 72: Data, 73: Learning result, 74: Data, 75: Data, 80: Generator, 81: Data, 82: Data, 83: Data, 84: Learning result, 85: Data, 86: Data, 87: Data, 91: Generator, 92: Discriminator, 93: Data, 94: Data, 95: Data, 96: Data

Claims

1. An information processing device having a function to perform inference based on a learning result obtained by learning using the change in pupil size over time shown in two or more first images included in a first video including a face, The aforementioned information processing device has a function to acquire a second video including the user's face, The information processing device has a function to detect eyes from each of the two or more second images included in the second video, The aforementioned information processing device has a function to detect the pupil from each of the detected eyes. The information processing device has a function to calculate the size of each of the detected pupils. The information processing device has a function to perform inference based on the learning results in response to the change in pupil size over time. An information processing device having a function to infer whether or not fatigue is present by comparing the measured value of the pupil size calculated from the second video with the estimated value of the pupil size based on the learning results.

2. In Claim 1, The first video described above is an information processing device that uses the temporal changes in the pupils of a person without fatigue as input data.

3. In claim 1 or 2, The learning and inference described above are performed by a neural network. The learning results include an information processing device with weight coefficients.

Images