Electronic apparatus

Abstract

To provide an information processing system and method that detect a fatigue, etc., by using a neural network.SOLUTION: A method obtains a reference image based on first to n-th (where n is an integer greater than or equal to two) images, inputs the first to n-th images and the reference image in an input layer of a neural network, outputs first to n-th estimated ages and a reference estimated age from an output layer, and outputs first to n-th data and reference data from an intermediate layer, takes, as an x coordinate, a value corresponding to a difference between the first to n-th estimated ages and the reference estimated age, takes, as a y coordinate, a value corresponding to the similarity between the first to n-th data and the reference data, obtains first to n-th coordinates, inputs a query image in the input layer, outputs a query estimated age from the output layer, and outputs query data from the intermediate layer, and obtains a query coordinate by using output results. Based on the first to n-th coordinates and on the query coordinate, the presence or absence of the fatigue, etc., of a person who has a face included in the query image is determined.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] One aspect of the present invention relates to an information processing system. Another aspect of the present invention relates to an information processing method. Regarding. [Background technology]

[0002] Portable information terminals have been developed that have the function of detecting the fatigue and stress of the user. Patent Document 1 discloses a mobile information terminal that detects the user's fatigue based on the pulse rate. . [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-86524 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when detecting the fatigue of a user of an electronic device such as a portable information terminal based on the pulse rate, The device must be in close contact with the user for a certain period of time. However, there is a problem in that it is less convenient than detecting the above.

[0005] An object of one embodiment of the present invention is to provide a highly convenient information processing system. To provide an information processing system capable of detecting fatigue, stress, etc. in a short time. Alternatively, detecting fatigue, stress, etc. using a neural network. It is an object of the present invention to provide an information processing system that can accurately calculate fatigue. One of the objectives of the present invention is to provide an information processing system that can detect stress, etc. Or, to provide an information processing system that can detect fatigue, stress, etc. in a simple manner. One of the goals is to

[0006] Another object of the present invention is to provide a highly convenient information processing method. It is an object of the present invention to provide an information processing method capable of detecting stress, etc. is an information processing system that can detect fatigue, stress, etc. using neural networks. Another object of the present invention is to provide a method for detecting fatigue, stress, etc. with high accuracy. Another object of the present invention is to provide an information processing method that can easily measure fatigue, stress, and other An object of the present invention is to provide an information processing method capable of detecting traces, etc.

[0007] It should be noted that the description of multiple problems does not preclude the existence of each problem. It is not necessary to solve all of the problems exemplified above. This problem is self-evident from the description of the above, and such a problem may also be one aspect of the present invention. . [Means for solving the problem]

[0008] In one aspect of the present invention, a query image including an image of a person's face is acquired, and The present invention provides an information processing system and an information processing method for determining whether a person is fatigued or not. First to nth (n is an integer equal to or greater than 2) images including a face image are acquired, and these are processed by neural network. The input to the network is the first to nth estimated ages output from the output layer, and the middle layer is The first to nth data to be output are acquired. Also, a query image including a face image is acquired. The input to the neural network is the query estimated age output from the output layer, and the middle layer Then, the query estimated age is calculated as the first to nth estimated ages. The query image is obtained by comparing the query data with the first to nth data. It is possible to determine whether or not the person included in the list is fatigued.

[0009] Specifically, one aspect of the present invention is a digital camera including an imaging unit, a first processing unit, a second processing unit, and a third processing unit. The imaging unit captures first to nth images (n being an integer) including an image of a person's face. a function of acquiring an image of a target object (an integer equal to or greater than 2) and a query image including an image of a person's face; The first processing unit has a function of acquiring a reference image based on the first to n-th images, and the second processing unit The logic unit performs processing using a neural network having an input layer, an intermediate layer, and an output layer. The second processing unit has a function of performing the above-described processing when the first to n-th images or the reference image are input to the input layer. When the first to nth estimated ages or the reference estimated ages are output from the output layer, The intermediate layer has a function of outputting the first to nth data or reference data, and the second The processing unit outputs the query estimated age from the output layer when a query image is input to the input layer. The third processing unit has a function of outputting query data from the intermediate layer, and the third processing unit outputs the first to n-th guesses. The difference between the retirement age and the reference estimated age is set as the x-coordinate, and the first to nth data are The value of the similarity with the reference data is taken as the y coordinate, and the first to nth coordinates are taken. The third processing unit calculates the difference between the query estimated age and the reference estimated age by x Let the coordinate be y, and the value of the similarity between the query data and the reference data be y. Let the query coordinate be The fourth processing unit performs clustering based on the first to n-th coordinates. Based on the clustering results and the query coordinates, the fatigue level of the person included in the query image is calculated. It is an information processing system that has the function of determining whether or not someone is working.

[0010] Alternatively, one aspect of the present invention is to provide a method for detecting a face of a person by using a plurality of images, each of which includes a first image to an nth image (n is an integer of 2 or more). An image is acquired, and a reference image is acquired based on the first to nth images. The input layer, the intermediate layer, and the output and an input layer of the neural network having a first to nth image and a reference image The first to nth estimated ages and the reference estimated age are input from the output layer, and the first to nth estimated ages are input from the intermediate layer. to n-th data and reference data, and output the first to n-th estimated ages and the reference data. The difference between the estimated age and the nth data is set as the x-coordinate, and the first to nth data and the reference data are classified as The similarity values ​​are set as y coordinates, and the first to nth coordinates are acquired, respectively, to form an image of the person's face. A query image containing the image is obtained, and the query image is input to the input layer, and the query is estimated from the output layer. The query data is output from the middle layer, and the query estimated age and the reference estimated age are calculated. The difference between the x-coordinates is used as the x-coordinate, and the similarity between the query data and the reference data is calculated as The query coordinates are obtained as the y coordinates, and clustering is performed based on the first to nth coordinates. Based on the clustering results and the query coordinates, the number of people included in the query image is calculated. This is an information processing method for determining whether or not fatigue is present. [Effects of the Invention]

[0011] According to one embodiment of the present invention, a highly convenient information processing system can be provided. It is possible to provide an information processing system that can detect fatigue, stress, etc. in a short time. Alternatively, neural networks can be used to detect fatigue, stress, etc. It is possible to provide an information processing system that can detect fatigue, stress, etc. with high accuracy. It is possible to provide an information processing system that can easily measure fatigue, stress, and other It is possible to provide an information processing system that can detect traces, etc.

[0012] Alternatively, a highly convenient information processing method can be provided. Alternatively, fatigue and stress can be alleviated in a short time. It is possible to provide an information processing method capable of detecting neural networks, etc. To provide an information processing method capable of detecting fatigue, stress, etc. using a network. Alternatively, an information processing method that can detect fatigue, stress, etc. with high accuracy can be provided. Alternatively, information that can detect fatigue, stress, etc. in a simple manner can be provided. It is possible to provide an information processing method.

[0013] It should be noted that the description of multiple effects does not preclude the existence of other effects. The embodiment does not necessarily have to have all of the effects exemplified. Problems, effects, and novel features other than those described above can be understood from the description and drawings of this specification. It will become clear by itself. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system. [Figure 2] 2A and 2B are schematic diagrams showing examples of the configuration of a neural network. [Figure 3] FIG. 3 is a flowchart showing an example of an information processing method. [Figure 4] 4A and 4B are schematic diagrams showing an example of an information processing method. [Figure 5]5A and 5B are schematic diagrams showing an example of an information processing method. [Figure 6] FIG. 6 is a graph showing an example of an information processing method. [Figure 7] FIG. 7 is a flowchart showing an example of an information processing method. [Figure 8] FIG. 8 is a schematic diagram showing an example of an information processing method. [Figure 9] 9A to 9C, 9D1 and 9D2 are graphs showing an example of an information processing method. [Figure 10] 10A to 10D are diagrams showing an example of an electronic device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described. However, one aspect of the present invention is not limited to the following description. The present invention is not limited to the above embodiments, and various changes and modifications may be made in form and detail without departing from the spirit and scope of the present invention. It will be readily understood by those skilled in the art that the present invention can be implemented by the following method. The present invention is not to be construed as being limited to the description of the embodiment shown in the accompanying drawings.

[0016] In addition, in the drawings attached to this specification, the components are classified by function and are shown as independent blocks. Although the block diagram is shown as a block, the actual components are not completely separated by function. It is difficult to achieve this, and one component may be involved in multiple functions, or one function may be involved in multiple components. It may be possible to achieve this without any effort.

[0017] (Embodiment 1) In this embodiment, an information processing system according to one embodiment of the present invention and a method for using the information processing system will be described. An information processing system and an information processing method according to one embodiment of the present invention will be described. In this method, fatigue of users of mobile information terminals such as smartphones or tablets is reduced. It can determine whether or not there is stress. Specifically, it has the function of estimating age. Neural networks can be used to determine whether fatigue, stress, etc. are present.

[0018] <Example of information processing system configuration> FIG. 1 shows an example of the configuration of an information processing system 10, which is an information processing system according to one embodiment of the present invention. The information processing system 10 can be incorporated into an electronic device. For example, it can be incorporated into mobile information terminals such as smartphones and tablets.

[0019] The information processing system 10 includes an imaging unit 11, a storage unit 12, a processing unit 20, an output unit 13, and The processing unit 20 includes an image calculation unit 21, an age estimation unit 22, a comparison unit 23, and a determination unit 24 and has.

[0020] In this specification, the components of the processing unit 20 include an image calculation unit 21, an age estimation unit 22, The comparison unit 23 and the determination unit 24 may also be referred to as processing units. The image calculation unit 21 is referred to as a first processing unit, the age estimation unit 22 is referred to as a second processing unit, and the comparison unit 2 3 may be referred to as a third processing unit, and the determination unit 24 may be referred to as a fourth processing unit.

[0021] In FIG. 1, arrows indicate the exchange of data and the like between the components of the information processing system 10. The data exchange shown in Figure 1 is an example, and the data connected by arrows, for example, In some cases, data can be exchanged between components that are not connected to the arrows. Therefore, even if the components are coupled, there may be cases where data is not exchanged between them.

[0022] The imaging unit 11 has a function of acquiring an image. For example, the imaging unit 11 includes a photoelectric conversion element. The pixels are arranged in a matrix, and by capturing an image using the pixels, The image captured by the imaging unit 11 can be an image including a person. For example, an image including a user of an electronic device incorporating the information processing system 10 can be obtained. Specifically, the image captured by the imaging unit 11 may include a face, for example. For example, the face of a user of an electronic device incorporating the information processing system 10 can be It may be an image containing

[0023] In this specification, a user of an electronic device in which an information processing system is incorporated is referred to as an information processing For example, the user of the information processing system 10 may be a A user of the electronic device may be referred to as a user of the information processing system 10.

[0024] The storage unit 12 has a function of storing images acquired by the imaging unit 11. The image stored in the memory unit 12 can be output to the processing unit 20 as needed. The image can be output to, for example, the image calculation unit 21, the age estimation unit 22, etc.

[0025] The storage unit 12 also has a function of storing data output by the processing unit 20. For example, The processing unit 20 reads out the image stored in the storage unit 12 and processes it, and the processing The storage unit 12 has a function of storing data acquired by the processing unit 20 and the like.

[0026] The storage unit 12 is, for example, a DRAM (Dynamic Random Access Memory ory), SRAM (Static Random Access Memory), etc. The storage unit 12 may include, for example, a flash memory, a ReRAM (Re Resistive Random Access Memory (RRAM) ), PRAM (Phase change Random Access Memor y), FeRAM (Ferroelectric Random Access Mem ory), MRAM (Magnetoresistive Random Access It can have non-volatile memory such as magnetoresistive memory (MR memory). Furthermore, the storage unit 12 may be, for example, a hard disk drive (HDD), a solid-state drive (SSD), A solid state drive (SD) may also be included.

[0027] The image calculation unit 21 has a function of acquiring a new image based on a plurality of images. For example, For example, the memory unit 12 stores images including faces. When a plurality of images are stored, the image calculation unit 21 first extracts faces from the plurality of images. The extracted face image is called a face image. Next, after aligning the resolution of the face image, the face image is averaged. The image calculation unit 21 acquires an average image. The image acquired by the image calculation unit 21 is stored in the storage unit 12.

[0028] In this specification, the image acquired by the image calculation unit 21 is referred to as a reference image. As described above, the image calculation unit 21 calculates, for example, the average image of the face. It has a function to acquire an average face image. Therefore, the reference image can be, for example, an average face image. can be done.

[0029] The age estimation unit 22 has a function of performing processing using a neural network NN. In this example, the image input to the age estimation unit 22 is processed by a neural network NN. The processing result is output to the comparison unit 23. Alternatively, the processing result is stored in the memory. The data is stored in unit 12.

[0030] When an image containing a person is input, the neural network NN calculates the age of the person. For example, when an image containing a face is input, the neural network NN When a person is identified, the system has the function of estimating the person's age based on the facial features. For example, wrinkles, Age can be estimated based on the appearance of wrinkles, age spots, and nasolabial folds. It is possible to estimate a person's age based on wrinkles around the eyes, etc.

[0031] FIG. 2A is a diagram showing an example of the configuration of a neural network NN. The network NN has layers L[1] to L[m] (m is an integer of 2 or more).

[0032] Layers L[1] to L[m] have neurons, and the neurons in each layer For example, a neuron in layer L[1] is connected to a neuron in layer L[2]. The neurons in layer L[2] are connected to the neurons in layer L[3]. The neurons are arranged in layer L[1] and layer L[3]. In other words, layers L[1] to L[m] form a hierarchical neural network. The network is configured.

[0033] An image is input to layer L[1], which outputs data corresponding to the input image. The data is input to layer L[2], which stores the data corresponding to the input data. The layer L[m] receives the data output from the layer L[m-1], and the layer L[m ] outputs data corresponding to the input data. From the above, we can consider layer L[1] as the input layer Layers L[2] to L[m-1] can be intermediate layers, and layer L[m] can be the output layer.

[0034] The neural network NN uses data output from layers L[1] to L[m], for example. The neural network NN is designed to have a feature set that corresponds to the image input. The learning can be done by unsupervised learning, supervised learning, etc. Whether learning is performed using unsupervised or supervised learning methods, the learning algorithm As the algorithm, an error back propagation method or the like can be used.

[0035] Neural network NN is a convolutional neural network (CNN). Figure 2B shows the neural network. Neural network NN when CNN is applied as a neural network NN Here, the neural network NN to which CNN is applied is called Let us call the neural network NNa.

[0036] The neural network NNa consists of a convolutional layer CL, a pooling layer PL, and a fully connected layer F In Figure 2B, the neural network NNa has a convolutional layer CL and a pulley Each layer has m layers of coupling layers PL (m is an integer greater than or equal to 1) and two layers of fully connected layers FCL. The neural network NNa has only one fully connected layer FCL. It may have three or more layers.

[0037] The convolution layer CL has the function of performing convolution on the data input to the convolution layer CL. For example, the convolution layer CL[1] calculates the following for the image input to the age estimation unit 22: It has the function of performing convolution. Also, the convolution layer CL[2] is connected to the pooling layer PL[1] It has the function of performing convolution on the data output from the convolution layer CL[m ] is a function that performs convolution on the data output from the pooling layer PL[m-1]. Has.

[0038] Convolution is performed by repeating the multiplication and accumulation of the data input to the convolution layer CL and the weight filter. The convolution in the convolution layer CL is used to generate the neural network. Image features corresponding to the image input to the network NNa are extracted.

[0039] The convolved data is transformed by the activation function and then output to the pooling layer PL. The activation function is ReLU (Rectified Linear Unit ts) can be used. ReLU outputs "0" if the input value is negative, If the input value is "0" or greater, the function outputs the input value as is. As the number, a sigmoid function, a tanh function, etc. can also be used.

[0040] The pooling layer PL performs pooling on the data input from the convolution layer CL. Pooling divides data into multiple regions and allocates a predetermined amount of data to each region. The pooling process extracts the data and arranges them in a matrix. This allows the amount of data to be reduced while retaining the extracted features. This can improve robustness against small deviations in the Large pooling, average pooling, Lp pooling, etc. can be used.

[0041] The fully connected layer FCL combines the input data and transforms the combined data using an activation function. The activation function can be ReLU, sigmoid function, or tanh function. The fully connected layer FCL is a layer in which all nodes in a layer are connected to all nodes in the next layer. The output from the convolution layer CL or the pooling layer PL is The data is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. Then, the vector obtained by inference using the fully connected layer FCL is It is output from L.

[0042] In the neural network NNa, one fully connected layer FCL can be used as the output layer. For example, in the neural network NNa shown in Figure 2B, the fully connected layer FCL[2 ] can be used as the output layer. Here, in the neural network NNa shown in Figure 2B, The fully connected layer FCL[1] can be used as an intermediate layer. If NNa has only the fully connected layer FCL[1] as the fully connected layer FCL, CL[1] can be used as the output layer. Furthermore, the neural network NNa If there are connection layers FCL[1] to FCL[3], the fully connected layer FCL[3] is The output layer can be the fully connected layer FCL[1] and the fully connected layer FCL[2] can be the intermediate layer. Similarly, when the neural network NNa has four or more fully connected layers FCL, One fully connected layer FCL can be used as the output layer, and the remaining fully connected layers FCL can be used as intermediate layers. .

[0043] The configuration of the neural network NNa is not limited to that shown in FIG. A neural network layer PL may be provided for each of a plurality of convolution layers CL. The number of pooling layers PL in the neural network NNa is less than the number of convolutional layers CL. Also, if you want to retain as much position information of the extracted features as possible, you can use the pooling layer PL. It does not have to be provided.

[0044] The neural network NNa learns the filter values ​​of the weight filters, The weighting coefficients of the coupling layer FCL can be optimized.

[0045] When an image containing a person is input to the input layer of the neural network NN, the neural network The estimated age of the person is output from the output layer of the neural network NN. When the network NN has the configuration shown in Figure 2A, the input layer, layer L[1], contains images containing people. When an image is input, the estimated age of the person is output from the output layer L[m]. The neural network NN is a neural network NNa having the configuration shown in FIG. 2B. If an image containing a person is input to the input layer, the convolution layer CL[1], the output The fully connected layer FCL[2] outputs the estimated age of the person.

[0046] The comparison unit 23 has a function of comparing data output from the output layer of the neural network NN. For example, it is possible to compare the estimated ages output by the output layer of the neural network NN. Specifically, for example, the estimated age of a person included in an image acquired by the imaging unit 11 and The estimated age obtained by inputting the reference image into the neural network NN, and For example, the estimated age of a person included in an image acquired by the imaging unit 11 is compared with The estimated age obtained by inputting the reference image into the neural network NN, and Comparisons can be made by calculating the difference.

[0047] The comparison unit 23 compares the data output from the intermediate layer of the neural network NN. For example, when the neural network NN has the configuration shown in FIG. 2A, the layer L[m-1]. When the image acquired by is input to the neural network NN, the layer L[m-1] outputs When the data and the reference image are input to the neural network NN, the layer L[m-1] It has the function of comparing the output data with the neural network NN shown in Figure 2. In the case of the neural network NNa with the configuration shown in B, for example, the fully connected layer FCL[1] It has the function of comparing the data output by the pooling layer PL[m]. It has the function of comparing the data output by the intermediate layer of the neural network NN. The comparison can be performed by calculating the similarity, for example, cosine similarity Similarity can be calculated using covariance, unbiased covariance, Pearson's product moment correlation coefficient, etc. In particular, it is preferable to use the cosine similarity.

[0048] Furthermore, the comparison unit 23 has a function of acquiring coordinates based on the comparison result. The comparison result of the estimated age output by the output layer of the neural network NN is expressed as the x-coordinate, The coordinate is obtained by comparing the data output by the intermediate layer of the network NN as the y coordinate. It has a function.

[0049] In addition, for the intermediate layers of the neural network NN, the output data is For example, if the neural network NN has the configuration shown in Figure 2A, In this case, the data output by layer L[m-1] and the data output by layer L[m-2] are compared. Specifically, for example, the image acquired by the imaging unit 11 is processed by a neural network. When input to the NN, the data output by layer L[m-1] and the data output by layer L[m-2] are When the data and the reference image are input to the neural network NN, the layer L[m-1] It has the function of comparing the output data with the data output by layer [m-2]. The neural network NN is the neural network NNa having the configuration shown in FIG. 2B. For example, if the data output by the fully connected layer FCL[1] and the pooling layer PL[m] are Specifically, for example, the image captured by the imaging unit 11 is compared with the output data. The data output by the fully connected layer FCL[1] when the neural network NNa is input is The data output by the pooling layer PL[m] and the reference image are fed to the neural network. The data output by the fully connected layer FCL[1] when input to the NNa, and the data output by the pooling layer P It has the function of comparing the data output by L[m] with .

[0050] The determination unit 24 has a function of performing clustering on the coordinates acquired by the comparison unit 23. The determination unit 24 also has a function of making a determination based on the result of clustering. For example, if a person is included in the image acquired by the imaging unit 11, it is possible to determine whether the person is tired or not, or whether the person is stressed. It has the function to determine whether or not there is a response, etc. Clustering method and determination of whether or not there is fatigue, etc. The method will be described in detail below.

[0051] The processing unit 20 includes a CPU (Central Processing Unit), a GPU ( Processing can be performed using a Graphics Processing Unit, etc. For example, the image calculation unit 21, the comparison unit 23, and the determination unit 24 perform processing using a CPU. The age estimation unit 22 is configured by a neural network NN. Therefore, it is preferable to use a GPU since it allows high-speed processing.

[0052] The output unit 13 has a function of outputting the determination result by the determination unit 24. For example, the output unit 13 For example, a display unit may be provided, and the results of the assessment of fatigue, stress, etc. may be displayed on the display unit. The output unit 13 has, for example, a speaker, and outputs a message when it is determined that the user is fatigued, stressed, or the like. , a warning sound can be emitted.

[0053] <Example of information processing method> An example of an information processing method using the information processing system 10 will be described below. Specifically, An example of a method for determining the presence or absence of fatigue using the information processing system 10 will be described.

[0054] FIG. 3 illustrates an example of a method for providing the information processing system 10 with a function for determining whether or not a user is fatigued. First, the imaging unit 11 captures images 31[1] to 31[n]. (n is an integer equal to or greater than 2) (Step S01). ] are images containing the same person. Images 31[1] to 31[n] are, for example, images shown in FIG. As shown in Figure 1, the images 31[1] to 31[n] contain the face of the same person. The person included can be, for example, a user of the information processing system 10.

[0055] The people in the images 31[1] to 31[n] are assumed to be in a state of no fatigue, for example. For example, when the user of the information processing system 10 is not fatigued, the image capturing unit 11 captures the image 31 [1 ] to image 31[n] are acquired.

[0056] Images 31[1] to 31[n] are acquired within a certain period of time, for example, within one month. Within three months, six months, or one year, images 31[1] to 31[n] are to be acquired. For example, one image 31 is acquired every day from January 1st to January 31st. If you want to save 31 images over a 6-month period, say 10 images per month, n is 31. If you get , n will be 60.

[0057] Next, the image calculation unit 21 obtains the reference image 32 based on the images 31[1] to 31[n]. FIG. 4B is a schematic diagram showing an example of the operation in step S02. For example, by calculating the average of images 31[1] to 31[n], the reference Acquire image 32. For example, if images 31[1] to 31[n] contain a face, First, by extracting faces from each of the images 31[1] to 31[n], n Next, after aligning the resolution of the n facial images, the average of the n facial images is calculated. The average image can be used as the reference image 32.

[0058] Then, the images 31[1] to 31[n] are input to the age estimation unit 22 (step Step S03). FIG. 5A is a schematic diagram showing an example of the operation in step S03. In 5A, the age estimation unit 22 performs processing using a neural network NN having the configuration shown in FIG. In the following figures, the age estimation unit 22 is also the same as that shown in FIG. It has the function of performing processing using the neural network NN configured as shown in A.

[0059] As shown in FIG. 5A, images 31[1] to 31[n] are each a function of the input layer. This allows the layer L[m ], the estimated age 33[1] to estimated age 33[n] are output. For example, i] (i is an integer between 1 and n) is input to layer L[1], the estimated age is 33[i] is output from layer L[m]. In Figure 5A, the estimated age 33[1] is aa, and the estimated age 33[2] is bb, and the estimated age 33[n] is cc. The data is output from the layer L[1]. In Figure 5A, image 31[i] is input to layer L[1]. In this case, the data output from layer L[m-1] is defined as data 34[i].

[0060] The data 34[1] to 34[n] may be, for example, vectors. shows the components contained in the vector data 34[1] to data 34[n]. In FIG. 5A, the data 34[1] includes components Va1, Va2, etc. 34[2] contains Vb1, Vb2, etc. as components, and data 34[n] contains Vb1, Vb2, etc. as components. It is said that these include Vc1, Vc2, etc.

[0061] Furthermore, the reference image 32 is input to the age estimation unit 22 (step S04). 5B is a schematic diagram showing an example of the operation in step S04. is input to the layer L[1], which functions as an input layer. The layer L[m] with the function outputs a reference estimated age of 35. In Figure 5B, the estimated age of 3 5 is kk. Data is output from the hidden layer. For example, For example, it can represent the feature amount of the reference image 32 input to layer L[1]. In B, when the reference image 32 is input to the layer L[1], the output from the layer L[m-1] is The data is referred to as reference data 36. The reference data 36 can be, for example, a vector. FIG. 5B shows the components contained in the reference data 36, ​​which is a vector. , and the reference data 36 includes components Vk1, Vk2, and the like.

[0062] In this specification, when a reference image is input to the age estimation unit 22, the output from the age estimation unit 22 is The estimated age inputted is called a reference estimated age. In this case, the data output from the intermediate layer is called reference data.

[0063] The reference data 36 can be, for example, a vector. Shows the ingredients.

[0064] After steps S03 and S04 are completed, the comparison unit 23 compares the estimated ages 33[1] to 33[2]. Obtain values ​​XV[1] to XV[n] based on age 33[n] and reference estimated age 35. (Step S05). Specifically, based on the estimated age 33[i] and the reference estimated age 35, the value Obtain XV[i]. For example, the difference between the estimated age 33[i] and the reference estimated age 35 is calculated as the value XV[i].

[0065] Furthermore, the comparison unit 23 compares the data 34[1] to 34[n] based on the reference data 36. , values ​​YV[1] to YV[n] are acquired (step S06). 4[i] and the reference data 36 to obtain the value YV[i]. For example, the vector The similarity between the data 34[i] and the reference data 36 can be set as a value YV[i]. The similarity is calculated using cosine similarity, covariance, unbiased covariance, Pearson's product moment correlation coefficient, etc. In particular, it is preferable to use the cosine similarity.

[0066] After steps S05 and S06 are completed, the comparison unit 23 compares the values ​​XV[1] to XV[n ] is the x-coordinate and the values ​​YV[1] to YV[n] are the y-coordinate (step S 07). Specifically, the coordinate (XV[i] is the x-coordinate and the value YV[i] is the y-coordinate. Here, we obtain the coordinates (XV[i],YV[i]) of the i-th coordinate. In the coordinate system shown in Figure 6, the coordinates are expressed as The coordinate systems shown in other figures are also represented in the same way. In other words, n coordinates (plots) are represented. Let's say.

[0067] After the comparison unit 23 acquires the first to n-th coordinates, for example, the determination unit 24 For example, one cluster is created based on the first to nth coordinates. The area containing the cluster is defined as area 30. For example, the Local Outlier Factor (LOF) method is used. When clustering is performed by LOF, the coordinates outside the region 30 are It can be said that this is the value.

[0068] FIG. 7 shows a case where the presence or absence of fatigue is detected using the information processing system 10 that has performed the processes shown in FIGS. 3 to 6. 1 is a flowchart showing an example of a method for determining whether or not a (Step S11). Image 41 is a group of people included in images 31[1] to 31[n]. For example, if image 31[1] to image 31[n] contain a face, In this case, the image 41 is an image containing the face of the same person as the person having the face.

[0069] In this specification, the image 41 may be referred to as a query image. If images 31[n] through 31[n] are referred to as the first through n-th images, image 41 is referred to as the n+1-th image. There are cases where this happens.

[0070] Next, the image 41 is input to the age estimation unit 22 (step S12). 8 is a schematic diagram showing an example of the operation in the image processing unit 12. As shown in FIG. 8, an image 41 is This is input to the layer L[1], which functions as an output layer. The layer L[m] outputs the estimated age 42. In Figure 8, the estimated age 42 is assumed to be qq. In addition, data is output from the hidden layer. In Figure 8, image 41 is input to layer L[1]. When the data is input, the data output from the layer L[m-1] is assumed to be data 43. can be a vector, for example. In FIG. 8, the reference data 36 includes components Vq1, Vq2, etc. It is said that it will be included.

[0071] The data 43 can be, for example, a vector. FIG. 8 shows the components of the data 43. are.

[0072] For example, if the image 41 contains a face, the face image is extracted from the image 41. Then, the resolution of the face image is adjusted to the resolution of the face image obtained from the images 31[1] to 31[n]. After adjusting the resolution, the image can be input to the age estimation unit 22. This can be done by the image calculation unit 21.

[0073] In this specification, for example, when the image 41 is referred to as the (n+1)th image, the estimated age 42 is referred to as the (n+1)th image. This is called the estimated age of n+1, and data 43 is sometimes called the n+1 data. If image 41 is called a query image, estimated age 42 is called a query estimated age, and data 43 is sometimes called query data.

[0074] Then, the comparison unit 23 obtains the value XVq based on the estimated age 42 and the reference estimated age 35. (Step S13) Specifically, the values ​​XV[1] to XV[n] are obtained by The value XVq is obtained in the same way as the estimated age 33[i] and the reference estimated age 33[i]. If the difference in retirement age of 35 is the value XV[i], then the difference between the estimated age of 42 and the reference estimated age of 35 is the value X Vq. Note that after a long period of time has passed since the acquisition of images 31[1] to 31[n] When acquiring an image 41, for example, the estimated age 42 minus the period is used as the reference estimated age. The value XVq may be obtained based on the age 35. For example, the image 31[1] to the image 31[2] Among [n], image 31[n] is the most recently acquired image, and If an image 41 is acquired one year after the initial age, subtract one year from the difference between the estimated age 42 and the reference estimated age 35. The value obtained by multiplying the image 31[1] to the image 31[n] can be set as the value XVq. Even if an image41 is acquired long after the previous image, the presence or absence of fatigue, etc. can be detected with high accuracy. It can be determined with high accuracy.

[0075] Also, the comparison unit 23 obtains the value YVq based on the data 43 and the reference data 36 (step Specifically, the method used to obtain the values ​​YV[1] to YV[n] For example, the value YVq is obtained by using the same method as the data 34[i] and the reference data 36 If the cosine similarity of data 43 and reference data 36 is set to the value YV[i], The similarity is represented by a value YVq.

[0076] After steps S13 and S14 are completed, the comparison unit 23 compares the value XVq with the x-coordinate and the value YVq 9A to 9C, the coordinates (X, V, q) are obtained (step S15). , YVq). Here, Fig. 9 also shows the coordinates shown in Fig. 6. When the image 41 is called the n+1th image, the coordinates (XVq, YVq) are called the n+1th coordinates. In addition, when the image 41 is called a query image, the coordinates (XVq, YVq) are called query coordinates. It is sometimes called a mark.

[0077] Next, the determination unit 24 calculates the coordinates (XV[1], YV[1]) to (XV[n], YV[n ]) and coordinates (XVq, YVq), it is determined whether the person in the image 41 is fatigued or not. For example, if the image 41 includes a face, the fatigue of the person having the face is calculated (step S16). Specifically, the coordinates (XVq, YVq) are determined to be the coordinates shown in Figs. 9A to 9C. If the result is included in the area 50, it is determined that fatigue exists, and if not included, it is determined that fatigue does not exist.

[0078] The area 50 will be described below. As mentioned above, the images 31[1] to 31[n] The person included is assumed to be in a state of no fatigue, for example. Therefore, the coordinates (XV[1], Area formed using LOF etc. based on coordinates (XV[n],YV[n]) to coordinates (XV[n],YV[n]) If the coordinates (XVq, YVq) are included in the area 30, the person in the image 41 is not fatigued. Furthermore, when a person is fatigued, wrinkles and sagging of the face increase, People who are fatigued tend to look older than those who are not. The estimated age by the age estimation unit 22 is higher than that of a person without the work, even if the person is the same age. Therefore, for example, the value obtained from the image 41 XVq is the estimated age 42 obtained from image 41 and the reference estimated age 32 obtained from reference image 32. If the difference in retirement age is 35, and the value XVq is negative, the person in image 41 is From the above, it can be determined that there is no fatigue. For example, it can be an area not included in the area 30 and having a coordinate XVq of 0 or more.

[0079] FIG. 9A shows a case where the coordinates (XVq, YVq) are included in the area 50. In this case, The person in the image 41 can be determined to be fatigued. In this case, the image 41 includes the image Vq, YVq) in the area 30. It can be determined that the person is not fatigued. Furthermore, FIG. 9C shows that the coordinates (XVq, YVq) This shows a case where the area is not included in either area 30 or area 50. Even in this case, as shown in FIG. As shown, the person in image 41 can be determined to be free of fatigue.

[0080] 9A to 9C, for example, a region not included in the region 30 and having a coordinate XVq of 0 or more is designated as a region. Although the area 50 is shown, the range of the area 50 is not limited to this. 6, and the range of the area 50 is the same as that shown in FIGS. 9A to 9C. It should be noted that the coordinates (XVq, YVq) in Fig. 9D1 and Fig. 9D2 are Not shown.

[0081] In FIG. 9D1, the x coordinate is equal to or greater than the largest x coordinate of the x coordinates of the boundary of the region 30. The area having the same size as the area 50 is the area 50. The x-coordinate of the boundary of the area 30 is Any y coordinate that is equal to or greater than the largest x coordinate is included in region 50. It states that:

[0082] In FIG. 9D1, the region 50 is rectangular, but this is not a limitation of one embodiment of the present invention. In D2, the larger the x coordinate, the larger the range of the y coordinate included in the area 50. Specifically, the larger the x coordinate, the larger the y coordinate, but the larger the area 50. In 9D2, the area 50 is a trapezoid whose upper and lower sides are parallel to the x-axis, and one leg is the same as that of the area 30. For example, if the area 50 is a right triangle, and the hypotenuse is the area 3, It may be configured to contact the boundary of 0.

[0083] In the cases shown in FIGS. 9A to 9C, 9D1 and 9D2, the area 30 and the x-coordinate As mentioned above, the x-coordinate is determined based on, for example, the estimated age and The difference between the image and the reference estimated age can be used to form the region 30. It is preferable that images 31[1] to 31[n] be acquired within one year. For example, image 3 Among images 1[1] to 31[n], image 31[1] is the image acquired in the earliest period. Let image 31[n] be the image acquired in the latest period. It is preferable to acquire the image 31[n] within one year of the date of the acquisition.

[0084] The above is an example of a method for determining whether or not fatigue is present using the information processing system 10. The presence or absence of stress, etc., of the user of the information processing system 10 is also determined in the same manner as in the methods shown in FIGS. It can be determined by the following method.

[0085] In the information processing method using the information processing system 10, for example, However, if you do not place a part of your body in close contact with the electronic device in which the information processing system 10 is installed for a certain period of time, Therefore, the information processing system can determine whether the user is fatigued or not. Specifically, the information processing system 10 is incorporated The electronic devices that are used in the information processing system 10 are highly convenient. If the image capturing unit 11 captures an image containing a face or the like, it is possible to determine whether the person is fatigued or not. Therefore, the information processing system 10 can determine whether or not the user is fatigued in a short time. It is possible.

[0086] In addition, in the information processing method using the information processing system 10, a neural network is used. Therefore, the information processing system 10 can determine whether or not the user is fatigued. Absence can be determined with high accuracy.

[0087] As a method for determining whether fatigue is present or absent using a neural network, we have investigated the neural network by analyzing images containing faces, etc. The output layer of the neural network determines whether fatigue, etc. is present or absent. It is possible to directly output the estimation results. However, this method requires a lot of effort, especially when there is no fatigue. It is necessary to prepare training data from both images in a normal state and images in a state where fatigue, etc. On the other hand, in an information processing method according to one aspect of the present invention, for example, the information processing system 10 is provided with a function such as fatigue. The images 31[1] to 31[n] used to provide the function of determining whether or not an object exists are all Therefore, the information processing system 10 can obtain an image of a state in which the subject is not fatigued. Using a neural network, it is possible to determine the presence or absence of fatigue in a simple manner.

[0088] In addition, in the information processing method using the information processing system 10, for example, a person included in the image 41 The presence or absence of fatigue is determined using the difference between the person's estimated age of 42 and the reference estimated age of 35. Therefore, the estimated age42 itself is not used for the judgment. The accuracy of age estimation by neural network NN does not need to be high. There is no need to customize the weights of the network NN for each person who is to be judged for fatigue, etc. Therefore, there is no need to prepare learning data for each person whose fatigue, etc. is to be determined. Therefore, the information processing system 10 can estimate fatigue in a simple manner using a neural network. It is possible to determine whether or not there is a problem.

[0089] This embodiment may be appropriately combined with at least a part of another embodiment described in this specification. It can be implemented in combination.

[0090] (Embodiment 2) In this embodiment, an information processing system and an information processing method according to one embodiment of the present invention are applied. An example of an electronic device that can do this will be described with reference to the drawings.

[0091] An information processing system according to one embodiment of the present invention, and an electronic device to which the information processing method can be applied. It is equipped with display devices, smartphones, tablets, personal computers, and recording media. Image storage devices or image playback devices, mobile phones, game consoles including portable types, portable data terminals , e-book terminals, video cameras, digital still cameras, goggle-type displays Examples of such electronic devices include head-mounted displays (HRDs). A to 10D.

[0092] FIG. 10A shows an example of a mobile phone 910, which may be, for example, a smartphone. The mobile phone 910 includes a housing 911, a display unit 912, operation buttons 913, and an external connection port. 914, a speaker 915, a jack 916, a camera 917, an earphone jack 918, etc. The mobile phone 910 can be provided with a touch sensor on the display unit 912. Any operation such as entering text or characters can be performed by touching the display unit 912 with a finger or a stylus. The slot 916 can also be used to insert memory cards such as SD cards. We also handle various types of data storage, including USB memory sticks and SSDs (Solid State Drives). Removable storage devices can be plugged in.

[0093] The information processing system and the information processing method according to one embodiment of the present invention are applied to the mobile phone 910. As a result, the mobile phone 910 is highly convenient and can alleviate fatigue, stress, etc. of the user in a short time. It is possible to determine whether or not the information processing system 10 is present. In this case, the imaging unit 11 shown in FIG. 1 can be said to include the camera 917. The output unit 13 shown in FIG. 1 includes a display unit 912, a speaker 915, etc. can.

[0094] FIG. 10B is an example of a portable data terminal 920, which may be, for example, a tablet. The portable data terminal 920 includes a housing 921, a display unit 922, a speaker 923, a camera 924, and a The display unit 922 has a touch panel function, which allows input and output of information. In addition, characters and the like can be recognized from an image acquired by a camera 924, and the corresponding characters can be played back through a speaker 923. The characters can be output as voice.

[0095] The information processing system and the information processing method according to one aspect of the present invention are applied to the portable data terminal 920. By doing so, the portable data terminal 920 is highly convenient and reduces fatigue and stress of the user in a short time. It is possible to determine whether or not there is a response, etc. 10 is applied, the imaging unit 11 shown in FIG. 1 may include a camera 924. The output unit 13 shown in FIG. 1 includes a display unit 922, a speaker 923, etc. It can be said that.

[0096] FIG. 10C shows an example of a wristwatch-type information terminal 930, which includes a housing / wristband 931, a display The display unit 932 includes an operation button 933, an external connection port 934, a camera 935, etc. A touch panel 932 is provided for operating the information terminal 930. The band 931 and the display unit 932 are flexible and have excellent wearability on the body.

[0097] Applying the information processing system and the information processing method according to one embodiment of the present invention to the information terminal 930 As a result, the information terminal 930 can easily and quickly determine whether the user is fatigued, stressed, etc. When the information processing system 10 is applied to the information terminal 930, 1 includes a camera 935. It can be said that the output unit 13 shown includes the display unit 932 and the like.

[0098] FIG. 10D shows a notebook personal computer 940. The computer 940 includes a housing 941, a keyboard 942, a pointing device 943, and an external The housing 941 includes a connection port 944, a speaker 945, and the like. It incorporates the La947.

[0099] The notebook personal computer 940 is provided with an information processing system according to one embodiment of the present invention, and By applying the processing method, the notebook personal computer 940 can be used with high convenience. Moreover, it is possible to determine whether the user is fatigued or stressed in a short time. When the information processing system 10 is applied to a personal computer 940, the photographing device shown in FIG. It can be said that the imaging unit 11 includes the camera 947. Also, the output unit 13 shown in FIG. It can be said that the display unit 946 and the like are included.

[0100] Although the electronic device of this embodiment has a display unit, the present invention can also be applied to electronic devices that do not have a display unit. One aspect of the present invention can be applied.

[0101] This embodiment may be appropriately combined with at least a part of another embodiment described in this specification. It can be implemented in combination. [Explanation of symbols]

[0102] 10: Information processing system, 11: Imaging unit, 12: Storage unit, 13: Output unit, 20: Processing unit, 21: Image calculation unit, 22: Age estimation unit, 23: Comparison unit, 24: Determination unit, 30: Region, 31 :Image, 32:Reference image, 33:Estimated age, 34:Data, 35:Reference estimated age, 36: Reference data, 41: Image, 42: Estimated age, 43: Data, 50: Area, 910: Mobile phone number Handset, 911: housing, 912: display unit, 913: operation buttons, 914: external connection port, 915: Speaker, 916: Outlet, 917: Camera, 918: Earphone jack, 920 : Portable data terminal, 921: Housing, 922: Display unit, 923: Speaker, 924: Camera 930: Information terminal, 931: Housing and wristband, 932: Display, 933: Operation buttons 934: External connection port 935: Camera 940: Notebook personal computer 941: Housing, 942: Keyboard, 943: Pointing device, 944: External connection port, 945: speaker, 946: display, 947: camera

Claims

[Claim 1] An electronic device incorporating an information processing system, the information processing system includes an imaging unit, a storage unit, a processing unit, and an output unit; the processing unit includes an image calculation unit, an age estimation unit, a comparison unit, and a determination unit; the imaging unit has a function of capturing an image of a face of a user of the electronic device, the storage unit has a function of storing first to n-th images (n is an integer of 2 or more) acquired by the imaging unit, and a function of outputting the first to n-th images (n is an integer of 2 or more) to the image calculation unit, the image calculation unit has a function of acquiring a calculated image calculated by averaging the first to n-th (n is an integer of 2 or more) images output from the storage unit, the age estimation unit has a function of performing processing using a neural network, the neural network has a function of acquiring a first estimated age of the user from the first to n-th images (n is an integer of 2 or more) and a function of estimating a second estimated age of the user from the calculated image; the comparison unit has a function of comparing the first estimated age with the second estimated age, The determination unit has a function of determining whether the user is fatigued or stressed based on the comparison result of the comparison unit.

Images