Information processing system and information processing method

Abstract

To provide an information processing system that can detect fatigue and the like by using a neural network.SOLUTION: An information processing method includes: acquiring a reference image on the basis of first to n-th (n is an integer of 2 or more) images; inputting the first to n-th images and the reference image to an input layer of a neural network, and outputting first to n-th estimated ages and a reference estimated age from an output layer and first to n-th data and reference data from an intermediate layer; acquiring first to n-th coordinates, with values corresponding to the differences between the first to n-th estimated ages and the reference estimated age as an x-coordinate and values corresponding to the similarities between the first to n-th data and the reference data as a Y-coordinate; inputting a query image to the input layer and outputting a query estimated age from the output layer and query data from the intermediate layer, and acquiring query coordinates by using an output result; and determining the presence or absence of fatigue and the like of a person having the face included in the query image on the basis of the first to n-th coordinates and the query coordinates.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] 1. Field of the Invention One aspect of the present invention relates to an information processing system and an information processing method. [Background technology]

[0002] Portable information terminals have been developed that have the function of detecting the fatigue and stress of a user. For example, Patent Document 1 discloses a portable information terminal that detects the fatigue of a user based on the pulse rate. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-86524 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when detecting the fatigue of a user of an electronic device such as a portable information terminal based on the pulse rate, the electronic device needs to be held in close contact with the user for a certain period of time, which poses a problem of lower convenience compared to detecting fatigue without the electronic device being held in close contact with the user.

[0005] An object of one embodiment of the present invention is to provide a highly convenient information processing system. Another object is to provide an information processing system that can detect fatigue, stress, and the like in a short time. Another object is to provide an information processing system that can detect fatigue, stress, and the like by using a neural network. Another object is to provide an information processing system that can detect fatigue, stress, and the like with high accuracy. Another object is to provide an information processing system that can detect fatigue, stress, and the like in a simple manner.

[0006] Another object of the present invention is to provide a highly convenient information processing method. Another object of the present invention is to provide an information processing method that can detect fatigue, stress, etc. in a short time. Another object of the present invention is to provide an information processing method that can detect fatigue, stress, etc. using a neural network. Another object of the present invention is to provide an information processing method that can detect fatigue, stress, etc. with high accuracy. Another object of the present invention is to provide an information processing method that can detect fatigue, stress, etc. in a simple manner.

[0007] It should be noted that the description of multiple problems does not preclude the existence of each other's problems. One embodiment of the present invention does not necessarily solve all of the problems exemplified. Furthermore, problems other than those listed will become apparent from the description in this specification, and such problems may also be problems of one embodiment of the present invention. [Means for solving the problem]

[0008] One aspect of the present invention is an information processing system and information processing method that acquires a query image including a face image of a person and determines whether or not the person is fatigued from the query image. That is, first to nth (n is an integer greater than or equal to 2) images including a face image are acquired, and these are input into a neural network to acquire first to nth estimated ages output from the output layer and first to nth data output from the intermediate layer. Also, a query image including a face image is acquired and input into the neural network to acquire a query estimated age output from the output layer and query data output from the intermediate layer. Then, by comparing the query estimated age with the first to nth estimated ages and comparing the query data with the first to nth data, it is possible to determine whether or not the person included in the query image is fatigued.

[0009] Specifically, one aspect of the present invention includes an imaging unit, a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit, wherein the imaging unit has a function of acquiring first to nth (n is an integer equal to or greater than 2) images including an image of a person's face and a query image including an image of the person's face, the first processing unit has a function of acquiring a reference image based on the first to nth images, the second processing unit has a function of performing processing using a neural network having an input layer, an intermediate layer, and an output layer, and when the first to nth images or the reference image are input to the input layer, the second processing unit has a function of outputting the first to nth estimated ages or the reference estimated ages from the output layer, respectively, and outputting the first to nth data or the reference data from the intermediate layer, respectively, when the query image is input. and a fourth processing unit that performs clustering based on the first to nth coordinates and determines whether a person included in the query image is fatigued or not based on the clustering results and the query coordinates.

[0010] Alternatively, one aspect of the present invention is to acquire first to n-th (n is an integer of 2 or more) images including an image of a person's face, acquire a reference image based on the first to n-th images, input the first to n-th images and the reference image to an input layer of a neural network having an input layer, an intermediate layer, and an output layer, output the first to n-th estimated ages and the reference estimated ages from the output layer, and output the first to n-th data and the reference data from the intermediate layer, respectively, and set the difference between the first to n-th estimated ages and the reference estimated ages as x coordinates, respectively, and set the value of the similarity between the first to n-th data and the reference data as y coordinates, respectively. This is an information processing method in which first to nth coordinates are acquired as coordinates, a query image including an image of a person's face is acquired, the query image is input to an input layer, a query estimated age is output from the output layer, and query data is output from an intermediate layer, the difference between the query estimated age and the reference estimated age is set as the x coordinate, and the similarity value between the query data and the reference data is set as the y coordinate to acquire query coordinates, clustering is performed based on the first to nth coordinates, and based on the clustering results and the query coordinates, it is determined whether or not a person included in the query image is fatigued. [Effects of the Invention]

[0011] According to one embodiment of the present invention, a highly convenient information processing system can be provided. Alternatively, an information processing system capable of detecting fatigue, stress, and the like in a short time can be provided. Alternatively, an information processing system capable of detecting fatigue, stress, and the like using a neural network can be provided. Alternatively, an information processing system capable of detecting fatigue, stress, and the like with high accuracy can be provided. Alternatively, an information processing system capable of detecting fatigue, stress, and the like in a simple manner can be provided.

[0012] Alternatively, it is possible to provide a highly convenient information processing method. Alternatively, it is possible to provide an information processing method that can detect fatigue, stress, etc. in a short time. Alternatively, it is possible to provide an information processing method that can detect fatigue, stress, etc. using a neural network. Alternatively, it is possible to provide an information processing method that can detect fatigue, stress, etc. with high accuracy. Alternatively, it is possible to provide an information processing method that can detect fatigue, stress, etc. in a simple manner.

[0013] The description of multiple effects does not preclude the existence of other effects. Furthermore, one embodiment of the present invention does not necessarily have all of the exemplified effects. Furthermore, problems, effects, and novel features of one embodiment of the present invention other than those described above will become apparent from the description and drawings in this specification. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system. [Figure 2] 2A and 2B are schematic diagrams showing examples of the configuration of a neural network. [Figure 3] FIG. 3 is a flowchart showing an example of an information processing method. [Figure 4] 4A and 4B are schematic diagrams showing an example of an information processing method. [Figure 5] 5A and 5B are schematic diagrams showing an example of an information processing method. [Figure 6] FIG. 6 is a graph showing an example of an information processing method. [Figure 7] FIG. 7 is a flowchart showing an example of an information processing method. [Figure 8] FIG. 8 is a schematic diagram showing an example of an information processing method. [Figure 9] 9A to 9C, 9D1 and 9D2 are graphs showing an example of an information processing method. [Figure 10]10A to 10D are diagrams showing an example of an electronic device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described. However, one aspect of the present invention is not limited to the following description, and it will be readily understood by those skilled in the art that various changes in form and details can be made without departing from the spirit and scope of the present invention. Therefore, one aspect of the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0016] In addition, in the drawings attached to this specification, components are classified by function and shown as block diagrams that are independent of each other, but in reality, it is difficult to completely separate components by function, and one component may be involved in multiple functions, or one function may be realized by multiple components.

[0017] (Embodiment 1) In this embodiment, an information processing system according to one embodiment of the present invention and an information processing method using the information processing system will be described. The information processing system and the information processing method according to one embodiment of the present invention can determine whether a user of a mobile information terminal such as a smartphone or a tablet is fatigued, stressed, or the like. Specifically, the presence or absence of fatigue, stress, or the like can be determined using a neural network that has a function of estimating age.

[0018] <Example of information processing system configuration> 1 is a block diagram showing an example of a configuration of an information processing system 10, which is an information processing system according to one embodiment of the present invention. The information processing system 10 can be incorporated into electronic devices, such as mobile information terminals such as smartphones and tablets.

[0019] The information processing system 10 includes an imaging unit 11, a storage unit 12, a processing unit 20, and an output unit 13. The processing unit 20 includes an image calculation unit 21, an age estimation unit 22, a comparison unit 23, and a determination unit 24.

[0020] In this specification, the image calculation unit 21, the age estimation unit 22, the comparison unit 23, and the determination unit 24, which are components of the processing unit 20, may also be referred to as processing units. For example, the image calculation unit 21 may be referred to as a first processing unit, the age estimation unit 22 may be referred to as a second processing unit, the comparison unit 23 may be referred to as a third processing unit, and the determination unit 24 may be referred to as a fourth processing unit.

[0021] In Fig. 1, arrows indicate the exchange of data and the like between the components of the information processing system 10. Note that the data exchange shown in Fig. 1 is an example, and for example, data and the like can be exchanged between components that are not connected by arrows. Also, even between components that are connected by arrows, data exchange may not be performed.

[0022] The imaging unit 11 has a function of acquiring an image. For example, the imaging unit 11 has pixels, each having a photoelectric conversion element, arranged in a matrix, and can acquire an image by capturing an image using the pixels. The image acquired by the imaging unit 11 can be an image including a person, for example, an image including a user of an electronic device in which the information processing system 10 is incorporated. Specifically, the image acquired by the imaging unit 11 can be an image including, for example, a face, for example, an image including the face of a user of an electronic device in which the information processing system 10 is incorporated.

[0023] In this specification, a user of an electronic device in which an information processing system is incorporated may be referred to as a user of the information processing system. For example, a user of an electronic device in which information processing system 10 is incorporated may be referred to as a user of information processing system 10.

[0024] The storage unit 12 has a function of storing images acquired by the imaging unit 11. The images stored in the storage unit 12 can be output to the processing unit 20 as needed. The images stored in the storage unit 12 can be output to, for example, the image calculation unit 21, the age estimation unit 22, etc.

[0025] The storage unit 12 also has a function of storing data and the like output by the processing unit 20. For example, the processing unit 20 reads out and processes an image stored in the storage unit 12, and the storage unit 12 has a function of storing data and the like acquired by the processing unit 20 through the processing.

[0026] The storage unit 12 may include, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The storage unit 12 may also include, for example, a nonvolatile memory such as a flash memory, a resistive random access memory (ReRAM), a phase change random access memory (PRAM), a ferroelectric random access memory (FeRAM), or a magnetoresistive random access memory (MRAM). Furthermore, the storage unit 12 may also include, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0027] The image calculation unit 21 has a function of acquiring a new image based on multiple images. For example, it has a function of acquiring an average image of multiple images. For example, if multiple images including faces are stored in the storage unit 12, the image calculation unit 21 first extracts faces from the multiple images. The extracted face images are called face images. Next, after aligning the resolution of the face images, the image calculation unit 21 acquires an average image of the face images. The images acquired by the image calculation unit 21 are stored in the storage unit 12.

[0028] In this specification, the image acquired by the image calculation unit 21 is referred to as a reference image. Also, an average image of face images may be referred to as an average face image. As described above, the image calculation unit 21 has a function of acquiring, for example, an average face image. Therefore, the reference image may be, for example, an average face image.

[0029] The age estimation unit 22 has a function of performing processing using a neural network NN. Specifically, it has a function of performing processing using a neural network NN on an image input to the age estimation unit 22. The processing result is output to the comparison unit 23. Alternatively, the processing result is stored in the storage unit 12.

[0030] A neural network NN has the function of estimating the age of a person when an image including the person is input. For example, when an image including a face is input, a neural network NN has the function of estimating the age based on the features of the face. For example, age can be estimated based on wrinkles, sagging skin, age spots, nasolabial folds, etc. For example, age can be estimated based on wrinkles around the mouth or at the corners of the eyes.

[0031] 2A is a diagram showing an example of the configuration of a neural network NN. The neural network NN has layers L[1] to L[m] (m is an integer of 2 or more).

[0032] Layers L[1] to L[m] have neurons, and the neurons in each layer are connected to each other. For example, a neuron in layer L[1] is connected to a neuron in layer L[2]. A neuron in layer L[2] is connected to a neuron in layer L[1] and a neuron in layer L[3]. In other words, layers L[1] to L[m] form a hierarchical neural network.

[0033] An image is input to layer L[1], which outputs data corresponding to the input image. This data is input to layer L[2], which outputs data corresponding to the input data. Data output from layer L[m-1] is input to layer L[m], which outputs data corresponding to the input data. From the above, layer L[1] can be considered the input layer, layers L[2] to L[m-1] can be considered intermediate layers, and layer L[m] can be considered the output layer.

[0034] The neural network NN is trained in advance so that data output from layers L[1] to L[m] corresponds to the features of the image input to the neural network NN. The training can be performed by unsupervised learning, supervised learning, etc. Whether the training is performed by unsupervised learning or supervised learning, a backpropagation method or the like can be used as a training algorithm.

[0035] The neural network NN can be a convolutional neural network (CNN). Figure 2B is a diagram showing an example of the configuration of a neural network NN when a CNN is applied as the neural network NN. Here, the neural network NN to which a CNN is applied is referred to as a neural network NNa.

[0036] The neural network NNa has a convolutional layer CL, a pooling layer PL, and a fully connected layer FCL. FIG. 2B shows an example in which the neural network NNa has m convolutional layers CL and m pooling layers PL (m is an integer equal to or greater than 1), and two fully connected layers FCL. Note that the neural network NNa may have only one fully connected layer FCL, or may have three or more fully connected layers FCL.

[0037] The convolutional layer CL has a function of performing convolution on data input to the convolutional layer CL. For example, the convolutional layer CL[1] has a function of performing convolution on an image input to the age estimation unit 22. The convolutional layer CL[2] has a function of performing convolution on data output from the pooling layer PL[1]. The convolutional layer CL[m] has a function of performing convolution on data output from the pooling layer PL[m-1].

[0038] Convolution is performed by repeatedly performing product-sum operations on the data input to the convolution layer CL and the weight filter. Through convolution in the convolution layer CL, image features corresponding to the image input to the neural network NNa are extracted.

[0039] The convolved data is transformed by an activation function and then output to the pooling layer PL. As the activation function, ReLU (Rectified Linear Units) or the like can be used. ReLU is a function that outputs "0" when the input value is negative and outputs the input value as is when the input value is "0" or greater. In addition, as the activation function, a sigmoid function, a tanh function, or the like can also be used.

[0040] The pooling layer PL has the function of pooling the data input from the convolutional layer CL. Pooling is a process of dividing the data into multiple regions, extracting predetermined data for each region, and arranging it in a matrix. Pooling can reduce the amount of data while retaining the features extracted by the convolutional layer CL. It can also increase robustness against small deviations in the input data. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as pooling.

[0041] The fully connected layer FCL has the function of combining input data, transforming the combined data using an activation function, and outputting it. The activation function can be ReLU, sigmoid function, tanh function, or the like. The fully connected layer FCL has a configuration in which all nodes in a given layer are connected to all nodes in the next layer. The data output from the convolution layer CL or pooling layer PL is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. The vector obtained by inference using the fully connected layer FCL is then output from the fully connected layer FCL.

[0042] In the neural network NNa, one fully connected layer FCL can be used as the output layer. For example, in the neural network NNa shown in FIG. 2B, the fully connected layer FCL[2] can be used as the output layer. Here, in the neural network NNa shown in FIG. 2B, the fully connected layer FCL[1] can be used as the hidden layer. Furthermore, if the neural network NNa has only the fully connected layer FCL[1] as the fully connected layer FCL, the fully connected layer FCL[1] can be used as the output layer. Furthermore, if the neural network NNa has fully connected layers FCL[1] to FCL[3], the fully connected layer FCL[3] can be used as the output layer, and the fully connected layer FCL[1] and the fully connected layer FCL[2] can be used as hidden layers. Similarly, if the neural network NNa has four or more fully connected layers FCL, one fully connected layer FCL can be used as the output layer, and the remaining fully connected layers FCL can be used as hidden layers.

[0043] The configuration of the neural network NNa is not limited to the configuration shown in FIG. 2B. For example, a pooling layer PL may be provided for each of a plurality of convolution layers CL. In other words, the number of pooling layers PL in the neural network NNa may be less than the number of convolution layers CL. Furthermore, if it is desired to preserve as much position information of extracted features as possible, it is not necessary to provide a pooling layer PL.

[0044] The neural network NNa can optimize the filter values ​​of the weight filters, the weight coefficients of the fully connected layer FCL, and the like by performing learning.

[0045] When an image including a person is input to the input layer of a neural network NN, the output layer of the neural network NN outputs the estimated age of the person. For example, if the neural network NN has the configuration shown in FIG. 2A, when an image including a person is input to the input layer, layer L[1], the estimated age of the person is output from the output layer, layer L[m]. Also, if the neural network NN is a neural network NNa with the configuration shown in FIG. 2B, when an image including a person is input to the input layer, convolution layer CL[1], the fully connected layer FCL[2], the output layer, the estimated age of the person is output.

[0046] The comparison unit 23 has a function of comparing data output by the output layer of the neural network NN. For example, it has a function of comparing estimated ages output by the output layer of the neural network NN. Specifically, it has a function of comparing the estimated age of a person included in an image acquired by the imaging unit 11 with the estimated age acquired by inputting a reference image into the neural network NN. For example, the comparison can be performed by calculating the difference between the estimated age of a person included in an image acquired by the imaging unit 11 and the estimated age acquired by inputting a reference image into the neural network NN.

[0047] The comparison unit 23 also has a function of comparing data output by intermediate layers of the neural network NN. For example, if the neural network NN has the configuration shown in FIG. 2A, the comparison unit 23 has a function of comparing data output by the layer L[m-1]. Specifically, the comparison unit 23 has a function of comparing data output by the layer L[m-1] when an image acquired by the imaging unit 11 is input to the neural network NN with data output by the layer L[m-1] when a reference image is input to the neural network NN. If the neural network NN is the neural network NNa having the configuration shown in FIG. 2B, the comparison unit 23 has a function of comparing data output by the fully connected layer FCL[1], for example. Alternatively, the comparison unit 23 has a function of comparing data output by the pooling layer PL[m]. The comparison of data output by intermediate layers of the neural network NN can be performed by calculating similarity, for example. For example, similarity can be calculated using cosine similarity, covariance, unbiased covariance, Pearson's product-moment correlation coefficient, etc. In particular, it is preferable to use cosine similarity.

[0048] Furthermore, the comparison unit 23 has a function of acquiring coordinates based on the comparison results. For example, the comparison result of the estimated age output from the output layer of the neural network NN is the x coordinate, and the comparison result of the data output from the intermediate layer of the neural network NN is the y coordinate.

[0049] Note that output data may be compared for two or more intermediate layers of the neural network NN. For example, if the neural network NN has the configuration shown in FIG. 2A, it has a function of comparing the data output from layer L[m-1] and the data output from layer L[m-2]. Specifically, it has a function of comparing the data output from layer L[m-1] and the data output from layer L[m-2] when an image acquired by the imaging unit 11 is input to the neural network NN with the data output from layer L[m-1] and the data output from layer L[m-2] when a reference image is input to the neural network NN. Furthermore, if the neural network NN is a neural network NNa with the configuration shown in FIG. 2B, it has a function of comparing the data output from fully connected layer FCL[1] and the data output from pooling layer PL[m]. Specifically, for example, when an image acquired by the imaging unit 11 is input to the neural network NNa, the data output by the fully connected layer FCL[1] and the data output by the pooling layer PL[m] are compared with the data output by the fully connected layer FCL[1] and the data output by the pooling layer PL[m] when a reference image is input to the neural network NNa.

[0050] The determination unit 24 has a function of performing clustering on the coordinates acquired by the comparison unit 23. The determination unit 24 also has a function of making a determination based on the results of the clustering. For example, when a person is included in the image acquired by the imaging unit 11, the determination unit 24 has a function of determining whether the person is fatigued or stressed. The clustering method and the method of determining whether fatigue or the like is present will be described in detail later.

[0051] The processing unit 20 can perform processing using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. For example, the image calculation unit 21, the comparison unit 23, and the determination unit 24 can perform processing using a CPU. Furthermore, since the age estimation unit 22 is configured using a neural network NN, it is preferable to use a GPU because it can perform processing at high speed.

[0052] The output unit 13 has a function of outputting the determination result by the determination unit 24. The output unit 13 has, for example, a display unit, and can display the determination result of fatigue, stress, etc. on the display unit. The output unit 13 also has, for example, a speaker, and can emit a warning sound when it is determined that fatigue, stress, etc. is present.

[0053] <Example of information processing method> The following describes an example of an information processing method using the information processing system 10. Specifically, the following describes an example of a method for determining the presence or absence of fatigue using the information processing system 10.

[0054] 3 is a flowchart illustrating an example of a method for providing the information processing system 10 with a function for determining whether or not fatigue is present. First, the imaging unit 11 acquires images 31[1] to 31[n] (n is an integer equal to or greater than 2) (step S01). Images 31[1] to 31[n] are assumed to be images that include the same person. Images 31[1] to 31[n] are assumed to be images that include the face of the same person, as shown in FIG. 4A, for example. The person included in images 31[1] to 31[n] may be, for example, the user of the information processing system 10.

[0055] The people included in the images 31[1] to 31[n] are assumed to be in a state where they are not fatigued. For example, the imaging unit 11 acquires the images 31[1] to 31[n] when the user of the information processing system 10 is not fatigued.

[0056] Images 31[1] to 31[n] are acquired within a certain period of time. For example, it is preferable to acquire images 31[1] to 31[n] within one month, three months, six months, or one year. For example, if one image 31 is acquired every day from January 1 to January 31, n is 31. Also, if ten images 31 are acquired every January for six months, n is 60.

[0057] Next, the image calculation unit 21 acquires the reference image 32 based on the images 31[1] to 31[n] (step S02). FIG. 4B is a schematic diagram showing an example of the operation in step S02. For example, the reference image 32 is acquired by calculating the average of the images 31[1] to 31[n]. For example, if the images 31[1] to 31[n] include faces, first, n facial images are acquired by extracting faces from each of the images 31[1] to 31[n]. Next, the resolutions of the n facial images are made uniform, and then an average image of the n facial images is acquired. This average image can be used as the reference image 32.

[0058] Thereafter, images 31[1] to 31[n] are input to the age estimation unit 22, respectively (step S03). Fig. 5A is a schematic diagram showing an example of the operation in step S03. In Fig. 5A, the age estimation unit 22 has a function of performing processing using the neural network NN having the configuration shown in Fig. 2A. Note that in the following figures, the age estimation unit 22 also has a function of performing processing using the neural network NN having the configuration shown in Fig. 2A.

[0059] As shown in FIG. 5A, images 31[1] to 31[n] are input to layer L[1], which functions as an input layer. As a result, estimated ages 33[1] to 33[n] are output from layer L[m], which functions as an output layer. For example, when image 31[i] (i is an integer between 1 and n) is input to layer L[1], estimated age 33[i] is output from layer L[m]. In FIG. 5A, estimated age 33[1] is aa, estimated age 33[2] is bb, and estimated age 33[n] is cc. Furthermore, data is output from the intermediate layer. This data may represent, for example, the feature values ​​of image 31 input to layer L[1]. In FIG. 5A, when image 31[i] is input to layer L[1], data 34[i] is output from layer L[m-1].

[0060] Data 34[1] to data 34[n] can be, for example, vectors. Fig. 5A shows components included in data 34[1] to data 34[n], which are vectors. In Fig. 5A, data 34[1] includes components Va1 and Va2, data 34[2] includes components Vb1 and Vb2, and data 34[n] includes components Vc1 and Vc2.

[0061] Further, a reference image 32 is input to the age estimation unit 22 (step S04). FIG. 5B is a schematic diagram illustrating an example of the operation in step S04. As shown in FIG. 5B, the reference image 32 is input to a layer L[1] functioning as an input layer. As a result, a reference estimated age 35 is output from a layer L[m] functioning as an output layer. In FIG. 5B, the estimated age 35 is kk. Furthermore, data is output from the intermediate layer. This data may represent, for example, the feature amount of the reference image 32 input to the layer L[1]. In FIG. 5B, when the reference image 32 is input to the layer L[1], the data output from the layer L[m-1] is referred to as reference data 36. The reference data 36 may be, for example, a vector. FIG. 5B shows the components included in the reference data 36, ​​which is a vector. In FIG. 5B, the reference data 36 includes components Vk1, Vk2, and the like.

[0062] In this specification, the estimated age output from the age estimation unit 22 when a reference image is input to the age estimation unit 22 is referred to as the reference estimated age. Also, the data output from the intermediate layer when a reference image is input to the age estimation unit 22 is referred to as reference data.

[0063] The reference data 36 may be, for example, a vector. Figure 5B shows the components of the reference data 36.

[0064] After steps S03 and S04 are completed, the comparison unit 23 obtains values ​​XV[1] to XV[n] based on the estimated ages 33[1] to 33[n] and the reference estimated age 35 (step S05). Specifically, the comparison unit 23 obtains value XV[i] based on the estimated age 33[i] and the reference estimated age 35. For example, the difference between the estimated age 33[i] and the reference estimated age 35 can be set as value XV[i].

[0065] Furthermore, the comparison unit 23 acquires values ​​YV[1] to YV[n] based on the data 34[1] to 34[n] and the reference data 36 (step S06). Specifically, the value YV[i] is acquired based on the data 34[i] and the reference data 36. For example, the similarity between the data 34[i], which is a vector, and the reference data 36 can be set as the value YV[i]. The similarity can be calculated using cosine similarity, covariance, unbiased covariance, Pearson's product moment correlation coefficient, etc. In particular, it is preferable to use cosine similarity.

[0066] After steps S05 and S06 are completed, the comparison unit 23 acquires coordinates with values ​​XV[1] to XV[n] as the x coordinates and values ​​YV[1] to YV[n] as the y coordinates (step S07). Specifically, coordinates (XV[i], YV[i]) with value XV[i] as the x coordinate and value YV[i] as the y coordinate are acquired. Here, the coordinate (XV[i], YV[i]) is referred to as the i-th coordinate. FIG. 6 shows an x-y coordinate system. In the coordinate system shown in FIG. 6, coordinates are shown as plots. Note that similar notations are used for coordinate systems shown in other figures. It is assumed that first to n-th coordinates are shown in FIG. 6. In other words, it is assumed that n coordinates (plots) are shown.

[0067] After the comparison unit 23 acquires the first to n-th coordinates, for example, the determination unit 24 performs clustering on the first to n-th coordinates. For example, one cluster is formed based on the first to n-th coordinates. The region containing this cluster is defined as region 30. Clustering can be performed using, for example, the Local Outlier Factor (LOF) method. When clustering is performed using LOF, coordinates outside region 30 can be considered to be outliers.

[0068] 7 is a flowchart showing an example of a method for determining the presence or absence of fatigue using the information processing system 10 that has performed the processes shown in FIGS. 3 to 6. First, the imaging unit 11 acquires an image 41 (step S11). The image 41 is an image that includes the same person as the person included in the images 31[1] to 31[n]. For example, if a face is included in the images 31[1] to 31[n], the image 41 is an image that includes the face of the same person as the person who has the face.

[0069] In this specification, image 41 may be referred to as a query image. Furthermore, when images 31[1] to 31[n] are referred to as the first to n-th images, image 41 may be referred to as the (n+1)th image.

[0070] Next, an image 41 is input to the age estimation unit 22 (step S12). FIG. 8 is a schematic diagram showing an example of the operation in step S12. As shown in FIG. 8, the image 41 is input to a layer L[1] that functions as an input layer. As a result, an estimated age 42 is output from a layer L[m] that functions as an output layer. In FIG. 8, the estimated age 42 is assumed to be qq. Furthermore, data is output from the intermediate layer. In FIG. 8, when the image 41 is input to the layer L[1], the data output from the layer L[m-1] is assumed to be data 43. The data 43 can be, for example, a vector. FIG. 8 shows the components included in the data 43, which is a vector. In FIG. 8, the reference data 36 includes components Vq1, Vq2, etc.

[0071] The data 43 may be, for example, a vector. Components of the data 43 are shown in FIG.

[0072] For example, if the image 41 includes a face, the face can be extracted from the image 41 to obtain a facial image, and the resolution of the facial image can be adjusted to match the resolution of the facial images obtained from the images 31[1] to 31[n] before being input to the age estimation unit 22. This process can be performed by the image calculation unit 21, for example.

[0073] In this specification, for example, when image 41 is referred to as the (n+1)th image, estimated age 42 may be referred to as the (n+1)th estimated age, and data 43 may be referred to as the (n+1)th data. Also, when image 41 is referred to as a query image, estimated age 42 may be referred to as a query estimated age, and data 43 may be referred to as query data.

[0074] The comparison unit 23 then obtains the value XVq based on the estimated age 42 and the reference estimated age 35 (step S13). Specifically, the value XVq is obtained using the same method as that used to obtain the values ​​XV[1] to XV[n]. For example, if the difference between the estimated age 33[i] and the reference estimated age 35 is the value XV[i], the difference between the estimated age 42 and the reference estimated age 35 is the value XVq. Note that if the image 41 is obtained a long time after the images 31[1] to 31[n] are obtained, the value XVq may be obtained based on, for example, the value obtained by subtracting the time period from the estimated age 42 and the reference estimated age 35. For example, if the image 31[n] is the most recently obtained image among the images 31[1] to 31[n] and the image 41 is obtained one year after the image 31[n] is obtained, the value XVq may be obtained by subtracting one year from the difference between the estimated age 42 and the reference estimated age 35. This makes it possible to determine with high accuracy whether fatigue or the like is present, even if image 41 is acquired a long time after images 31[1] to 31[n] are acquired.

[0075] The comparator 23 also obtains a value YVq based on the data 43 and the reference data 36 (step S14). Specifically, the value YVq is obtained using the same method as that used to obtain the values ​​YV[1] to YV[n]. For example, if the cosine similarity between the data 43[i] and the reference data 36 is the value YV[i], the cosine similarity between the data 43 and the reference data 36 is the value YVq.

[0076] After steps S13 and S14 are completed, the comparison unit 23 acquires coordinates with the value XVq as the x coordinate and the value YVq as the y coordinate (step S15). Coordinates (XVq, YVq) are shown in FIGS. 9A to 9C. Here, FIG. 9 also shows the coordinates shown in FIG. 6. When the image 41 is referred to as the (n+1)th image, the coordinates (XVq, YVq) may be referred to as the (n+1)th coordinates. When the image 41 is referred to as a query image, the coordinates (XVq, YVq) may be referred to as query coordinates.

[0077] Next, the determination unit 24 determines whether or not a person included in the image 41 is fatigued based on the coordinates (XV[1], YV[1]) through (XV[n], YV[n]) and the coordinates (XVq, YVq) (step S16). For example, if the image 41 includes a face, the determination unit 24 determines whether or not the person having the face is fatigued. Specifically, if the coordinates (XVq, YVq) are included in the area 50 shown in Figures 9A to 9C, etc., it is determined that the person is fatigued, and if they are not included, it is determined that the person is not fatigued.

[0078] Region 50 will be described below. As described above, the people included in images 31[1] to 31[n] are assumed to be in a state of not being fatigued. Therefore, if coordinates (XVq, YVq) are included in region 30 formed using LOF or the like based on coordinates (XV[1], YV[1]) to (XV[n], YV[n]), the person included in image 41 can be determined to be not fatigued. Furthermore, fatigue tends to increase wrinkles and sagging skin on the face, making people appear older than those without fatigue. In other words, a person with fatigue has more characteristics that cause the age estimation unit 22 to estimate a higher age than a person without fatigue, even if they are the same person of the same age. Therefore, for example, if the value XVq obtained from image 41 is the difference between the estimated age 42 obtained from image 41 and the reference estimated age 35 obtained from reference image 32, if the value XVq is a negative value, the person included in image 41 can be determined to be not fatigued. From the above, as shown in FIGS. 9A to 9C, the region 50 can be, for example, a region not included in the region 30, where the coordinate XVq is 0 or greater.

[0079] FIG. 9A shows a case where coordinates (XVq, YVq) are included in region 50. In this case, the person included in image 41 can be determined to be fatigued. FIG. 9B shows a case where coordinates (XVq, YVq) are included in region 30. In this case, the person included in image 41 can be determined to be not fatigued. FIG. 9C shows a case where coordinates (XVq, YVq) are included in neither region 30 nor region 50. Even in this case, as in the case shown in FIG. 9B, the person included in image 41 can be determined to be not fatigued.

[0080] 9A to 9C, for example, an area not included in area 30 and having a coordinate XVq of 0 or greater is defined as area 50, but the range of area 50 is not limited to this. In FIGS. 9D1 and 9D2, area 50 is added to the xy coordinate system shown in FIG. 6, and the range of area 50 differs from that shown in FIGS. 9A to 9C. Note that coordinates (XVq, YVq) are not shown in FIGS. 9D1 and 9D2.

[0081] 9D1, an area whose x coordinate is equal to or larger than the largest x coordinate among the x coordinates of the boundary of area 30 is defined as area 50. Note that any y coordinate is included in area 50 as long as the x coordinate is equal to or larger than the largest x coordinate among the x coordinates of the boundary of area 30.

[0082] In FIG. 9D1, region 50 is rectangular, but this is not a limitation of the present invention. In FIG. 9D2, the larger the x coordinate, the larger the y coordinate range included in region 50. Specifically, the larger the x coordinate, the larger the y coordinate, even if the y coordinate is larger, it is still included in region 50. In FIG. 9D2, an example is shown in which region 50 is a trapezoid with its upper and lower sides parallel to the x axis, and one leg of region 50 abuts the boundary of region 30. Note that region 50 may also be a right-angled triangle, with its hypotenuse abutting the boundary of region 30, for example.

[0083] In the cases shown in FIGS. 9A to 9C and 9D1 and 9D2, region 50 is defined based on region 30 and the size of the x-coordinate. As described above, the x-coordinate can be, for example, the difference between the estimated age and the reference estimated age. Therefore, it is preferable that images 31[1] to 31[n] used to form region 30 be acquired within one year. For example, if, among images 31[1] to 31[n], image 31[1] is the image acquired in the earliest period and image 31[n] is the image acquired in the latest period, it is preferable that image 31[n] be acquired within one year of image 31[1] being acquired.

[0084] The above is an example of a method for determining whether or not a user of the information processing system 10 is fatigued. The presence or absence of stress, etc., of a user of the information processing system 10 can also be determined by a method similar to the methods shown in FIGS.

[0085] In an information processing method using the information processing system 10, for example, it is possible to determine whether or not a user of the information processing system 10 is fatigued, even if the user does not place any part of the body in close contact with an electronic device in which the information processing system 10 is incorporated for a certain period of time. Therefore, it can be said that the information processing system 10 is highly convenient. Specifically, it can be said that an electronic device in which the information processing system 10 is incorporated is highly convenient. Furthermore, the information processing system 10 can determine whether or not a user is fatigued, if the imaging unit 11 captures an image and acquires an image that includes a face, etc. Therefore, the information processing system 10 can determine whether or not a user is fatigued, etc. in a short period of time.

[0086] Furthermore, in the information processing method using the information processing system 10, it is possible to determine the presence or absence of fatigue, etc. using a neural network. Therefore, the information processing system 10 can determine the presence or absence of fatigue, etc. with high accuracy.

[0087] One possible method for determining the presence or absence of fatigue using a neural network is to input an image including a face or the like into the neural network, and have the output layer of the neural network directly output an estimation result of the presence or absence of fatigue. However, this method requires that both images of a state without fatigue and images of a state with fatigue be prepared as training data. On the other hand, in an information processing method according to one aspect of the present invention, for example, images 31[1] to 31[n] used to provide the information processing system 10 with the function of determining the presence or absence of fatigue can all be images of a state without fatigue. Therefore, the information processing system 10 can determine the presence or absence of fatigue in a simple manner while using a neural network.

[0088] Furthermore, in an information processing method using the information processing system 10, for example, the presence or absence of fatigue, etc., of a person included in an image 41 is determined using the difference between the person's estimated age 42 and the reference estimated age 35. Therefore, the estimated age 42 itself is not used in the determination. Therefore, the accuracy of age estimation by the neural network NN does not need to be high. Therefore, for example, it is not necessary to customize the weights, etc. of the neural network NN for each person whose fatigue, etc. is to be determined. Therefore, it is not necessary to prepare learning data for each person whose fatigue, etc. is to be determined. As described above, the information processing system 10 can determine the presence or absence of fatigue, etc., in a simple manner while using a neural network.

[0089] This embodiment mode can be implemented by appropriately combining at least a part thereof with other embodiment modes described in this specification.

[0090] (Embodiment 2) In this embodiment, examples of an electronic device to which an information processing system and an information processing method according to one embodiment of the present invention can be applied will be described with reference to drawings.

[0091] Examples of electronic devices to which the information processing system and information processing method according to one embodiment of the present invention can be applied include display devices, smartphones, tablets, personal computers, image storage devices or image playback devices equipped with recording media, mobile phones, game consoles including portable types, portable data terminals, e-book terminals, cameras such as video cameras and digital still cameras, goggle-type displays (head-mounted displays), etc. Specific examples of these electronic devices are shown in Figures 10A to 10D.

[0092] 10A shows an example of a mobile phone 910, which may be, for example, a smartphone. The mobile phone 910 has a housing 911, a display 912, operation buttons 913, an external connection port 914, a speaker 915, a socket 916, a camera 917, an earphone socket 918, and the like. The mobile phone 910 may be provided with a touch sensor on the display 912. Various operations, such as making a call or entering text, can be performed by touching the display 912 with a finger or a stylus. In addition, various removable storage devices, such as a memory card such as an SD card, a USB memory, or an SSD (solid state drive), can be inserted into the socket 916.

[0093] By applying the information processing system and the information processing method of one embodiment of the present invention to the mobile phone 910, the mobile phone 910 can determine whether or not a user is fatigued, stressed, or the like with high convenience in a short time. Note that when the information processing system 10 is applied to the mobile phone 910, the imaging unit 11 shown in FIG. 1 can be said to include a camera 917. Furthermore, the output unit 13 shown in FIG. 1 can be said to include a display unit 912, a speaker 915, or the like.

[0094] 10B shows an example of a portable data terminal 920, which may be, for example, a tablet. The portable data terminal 920 includes a housing 921, a display unit 922, a speaker 923, a camera 924, and the like. Information can be input and output using a touch panel function of the display unit 922. Furthermore, characters and the like can be recognized from an image acquired by the camera 924, and the characters can be output as voice through the speaker 923.

[0095] By applying the information processing system and information processing method of one embodiment of the present invention to the portable data terminal 920, the portable data terminal 920 can determine whether or not a user is fatigued, stressed, or the like with high convenience in a short time. Note that when the information processing system 10 is applied to the portable data terminal 920, the imaging unit 11 shown in FIG. 1 can be said to include a camera 924. Furthermore, the output unit 13 shown in FIG. 1 can be said to include a display unit 922, a speaker 923, or the like.

[0096] 10C shows an example of a wristwatch-type information terminal 930, which includes a housing / wristband 931, a display unit 932, operation buttons 933, an external connection port 934, a camera 935, etc. The display unit 932 is provided with a touch panel for operating the information terminal 930. The housing / wristband 931 and the display unit 932 are flexible and have excellent wearability on the body.

[0097] By applying the information processing system and the information processing method of one embodiment of the present invention to the information terminal 930, the information terminal 930 can determine whether or not a user is fatigued, stressed, or the like with high convenience in a short time. Note that when the information processing system 10 is applied to the information terminal 930, the imaging unit 11 shown in FIG. 1 can be said to include a camera 935. Furthermore, the output unit 13 shown in FIG. 1 can be said to include a display unit 932, etc.

[0098] 10D shows a laptop personal computer 940. The laptop personal computer 940 includes a housing 941, a keyboard 942, a pointing device 943, an external connection port 944, and speakers 945. A display unit 946 and a camera 947 are incorporated in the housing 941.

[0099] By applying the information processing system and information processing method of one embodiment of the present invention to the notebook personal computer 940, the notebook personal computer 940 can determine whether or not a user, etc., is fatigued, stressed, or the like with high convenience in a short time. Note that when the information processing system 10 is applied to the notebook personal computer 940, the imaging unit 11 shown in Fig. 1 can be said to include a camera 947. Furthermore, the output unit 13 shown in Fig. 1 can be said to include a display unit 946, etc.

[0100] Although the electronic devices in this embodiment have a display portion, one embodiment of the present invention can also be applied to electronic devices that do not have a display portion.

[0101] This embodiment mode can be implemented by appropriately combining at least a part thereof with other embodiment modes described in this specification. [Explanation of symbols]

[0102] 10: Information processing system, 11: Imaging unit, 12: Memory unit, 13: Output unit, 20: Processing unit, 21: Image calculation unit, 22: Age estimation unit, 23: Comparison unit, 24: Determination unit, 30: Area, 31: Image, 32: Reference image, 33: Estimated age, 34: Data, 35: Reference estimated age, 36: Reference data, 41: Image, 42: Estimated age, 43: Data, 50: Area, 910: Mobile phone, 911: Housing, 912: Display unit, 913: Operation button, 914: External connection port, 915: Speaker, 916: Plug-in mouth, 917: camera, 918: earphone jack, 920: portable data terminal, 921: housing, 922: display unit, 923: speaker, 924: camera, 930: information terminal, 931: housing / wristband, 932: display unit, 933: operation buttons, 934: external connection port, 935: camera, 940: notebook personal computer, 941: housing, 942: keyboard, 943: pointing device, 944: external connection port, 945: speaker, 946: display unit, 947: camera

Claims

1. an imaging unit, a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit; the imaging unit has a function of acquiring first to n-th images (n is an integer equal to or greater than 2) including an image of a person's face and a query image; the first processing unit has a function of acquiring a reference image based on the first to n-th images, the second processing unit has a function of performing processing using a neural network having an input layer, an intermediate layer, and an output layer; the second processing unit has a function of outputting, when the first to nth images or the reference images are input to the input layer, first to nth estimated ages or reference estimated ages from the output layer, respectively, and outputting, from the intermediate layer, first to nth data or reference data, respectively; the second processing unit has a function of outputting a query estimated age from the output layer and outputting query data from the intermediate layer when the query image is input to the input layer; the third processing unit has a function of acquiring first to n-th coordinates, respectively, by setting values ​​of differences between the first to n-th estimated ages and the reference estimated ages as x-coordinates and values ​​of similarity between the first to n-th data and the reference data as y-coordinates; the third processing unit has a function of acquiring query coordinates by setting a difference value between the query estimated age and the reference estimated age as an x ​​coordinate and a similarity value between the query data and the reference data as a y coordinate; The fourth processing unit is an information processing system having a function of performing clustering based on the first to nth coordinates and determining whether or not the person included in the query image is stressed based on the results of the clustering and the query coordinates.

2. acquiring first to n-th images (n is an integer equal to or greater than 2) including an image of a person's face; acquiring a reference image based on the first to n-th images; a neural network having an input layer, an intermediate layer, and an output layer, inputting the first to n-th images and the reference image into the input layer, and outputting the first to n-th estimated ages and reference estimated ages from the output layer, and outputting the first to n-th data and reference data from the intermediate layer; a difference between the first to n-th estimated ages and the reference estimated ages as x-coordinates, and a value of similarity between the first to n-th data and the reference data as y-coordinates, thereby obtaining first to n-th coordinates, respectively; Obtain a query image containing an image of a person's face; The query image is input to the input layer, and a query estimated age is output from the output layer and query data is output from the intermediate layer; a difference value between the query estimated age and the reference estimated age is defined as an x-coordinate, and a similarity value between the query data and the reference data is defined as a y-coordinate, thereby obtaining query coordinates; An information processing method for performing clustering based on the first to n-th coordinates, and determining whether or not a person included in the query image is stressed based on the clustering result and the query coordinates.

Images