Information processing device, information processing method, and recording medium

Abstract

Provided is an information processing device, wherein: a face image acquisition means acquires a face image of a subject; a wakefulness state determination means determines the wakefulness state of the subject; a calculation means calculates, from the face image, information related to the eye-opening degree of the eyes during an interval for which the wakefulness state has been determined by the wakefulness state determination means; and a cognitive function estimation means estimates the cognitive function of the subject on the basis of the information related to the eye-opening degree.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to a technique for estimating a cognitive function of a subject.

[0002] Managing the health of the mind, body, and brain is important for extending healthy lifespans, but it is difficult to recognize declines in cognitive function. There are approximately 36 million people aged 65 and over, and it is estimated that approximately 6 million of them have dementia.

[0003] Patent Document 1 describes an automatic driving system for a vehicle that processes facial image data of a driver to extract information about the driver's eye behavior and determine the driver's level of alertness and drowsiness.

[0004] JP 2017-30390 A

[0005] However, only about 1.9 million people aged 75 or older take cognitive function tests when renewing their driver's licenses, and it is difficult for elderly people to get the opportunity to take cognitive function tests unless they voluntarily visit a hospital. Therefore, there is a need to increase opportunities to measure cognitive function, and there is a growing demand for easy, stress-free testing outside of medical institutions.

[0006] One of the purposes of the present disclosure is to estimate cognitive function simply and easily without placing a burden on the subject.

[0007] In order to solve the above problem, in one aspect of the present invention, an information processing device comprises: a facial image acquisition means for acquiring a facial image of a subject; an awake state determination means for determining the awake state of the subject; a calculation means for calculating, from the facial image, information regarding the degree of eye opening during a section determined to be an awake state by the awake state determination means; and a cognitive function estimation means for estimating the cognitive function of the subject based on the information regarding the degree of eye opening.

[0008] In another aspect of the present invention, an information processing method includes acquiring a facial image of a subject, determining the subject's state of wakefulness, calculating information regarding the degree of eye opening during a section determined to be the wakefulness state from the facial image, and estimating the subject's cognitive function based on the information regarding the degree of eye opening.

[0009] In yet another aspect of the present invention, a recording medium records a program that causes a computer to execute a process of acquiring a facial image of a subject, determining the subject's state of wakefulness, calculating information regarding the degree of eye opening from the facial image during a section determined to be the wakefulness state, and estimating the subject's cognitive function based on the information regarding the degree of eye opening.

[0010] According to the present disclosure, cognitive function can be estimated easily and without burdening the subject.

[0011] 1 shows an example of a schematic configuration of a cognitive function estimation system. FIG. 2 shows an example of classification according to MMSE scores. FIG. 3 shows an example of the hardware configuration of a server and a user terminal. FIG. 4 is a block diagram showing an example of the functional configuration of a server. FIG. 5 is an example of a graph showing time series information on eye opening degree. FIG. 6 is an example of a graph showing the average percentage of closed eyes of subjects classified as cognitively normal. FIG. 7 is an example of a graph showing time series information on eye opening degree and time fluctuation. FIG. 8 is an example of a graph showing time series information on eye opening degree of the left eye and right eye and a difference between the left and right eyes. FIG. 9 is an example of a graph showing the average and standard deviation of the percentage of closed eyes. FIG. 10 is an example of a graph showing the average speed of movement. FIG. 11 is a table showing the correlation between cognitive function by purpose and eyelid fluctuation feature amounts. FIG. 12 is a flowchart of a cognitive function estimation process. FIG. 13 shows the functional configuration of a server that uses a machine learning model. FIG. 14 is a flowchart of a learning process for training a machine learning model. FIG. 15 is a flowchart of a cognitive function estimation process using a machine learning model.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. [Embodiment] (Configuration) Fig. 1 shows an example of a schematic configuration of a cognitive function estimation system 100 to which an information processing device of the present disclosure is applied. The cognitive function estimation system 100 is a system that estimates cognitive function based on time-series information regarding the degree of eye opening and outputs an estimation result.

[0013] In the cognitive function estimation system 100, a server 1 and a user terminal 2 are communicatively connected via a network 5 such as the Internet. The user terminal 2 is a smartphone, tablet, PC, or the like used by a user (hereinafter also referred to as a "subject") whose cognitive function is to be estimated, and captures an image of the subject's face and transmits it as a facial image D1 to the server 1. For example, the subject may be an elderly person with concerns about their cognitive function. The server 1 is an information processing device that processes, stores, and transmits and receives various data, and analyzes the facial image D1 to estimate the subject's cognitive function.

[0014] In the present disclosure, the cognitive function estimation system 100 outputs, as a cognitive function estimation result, a score equivalent to the Mini-Mental State Examination (MMSE), which is one of the cognitive function assessments, or a classification based on the MMSE score equivalent. The MMSE is a widely used dementia test, and is a cognitive function test with a maximum score of 30 points, consisting of 11 items: time orientation, place orientation, immediate and delayed recall of three words, calculation, object naming, sentence repetition, three-stage verbal command, written command, sentence writing, and drawing copying. Furthermore, classification based on the MMSE score equivalent, as shown in FIG. 2 , classifies subjects with an MMSE score of 28 or more as "cognitively normal," subjects with a score of 24 to 27 as "suspected mild cognitive impairment," and subjects with a score of 23 or less as "suspected dementia."

[0015] 3A is a block diagram showing an example of the hardware configuration of the server 1. As shown in the figure, the server 1 includes an interface 11, a processor 12, a memory 13, a recording medium 14, a display unit 15, and an input unit 16.

[0016] The interface 11 exchanges data with the user terminal 2. The interface 11 is used when receiving the facial image D1 from the user terminal 2. The interface 11 is also used when the server 1 exchanges data with a predetermined device connected by wire or wirelessly.

[0017] The processor 12 is a computer such as a CPU (Central Processing Unit), and executes a prepared program to control the entire server 1. The processor 12 may be a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0018] The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 stores programs executed by the processor 12. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0019] The recording medium 14 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the server 1. The recording medium 14 records various programs to be executed by the processor 12. When the server 1 executes the cognitive function estimation process, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12.

[0020] The display unit 15 displays a predetermined image on, for example, an LCD (Liquid Crystal Display), etc. The input unit 16 includes a keyboard, a mouse, a touch panel, etc., and is used by an operator who manages the server 1.

[0021] 3(b) is a block diagram showing an example of the hardware configuration of the user terminal 2. As shown in the figure, the user terminal 2 includes an interface 21, a processor 22, a memory 23, a recording medium 24, a display unit 25, an input unit 26, and an imaging unit 27.

[0022] The interface 21 exchanges data with the server 1 via the network 5. The interface 21 is used to transmit the facial image D1 to the server 1 and to receive the estimation result of the cognitive function of the subject from the server 1.

[0023] The processor 22 is a computer such as a CPU, and executes a program prepared in advance to control the entire user terminal 2. The processor 22 may be a CPU, a GPU, a DSP, an MPU, an FPU, a PPU, a TPU, a quantum processor, a microcontroller, or a combination thereof.

[0024] The memory 23 is composed of a ROM, a RAM, etc. The memory 23 stores programs executed by the processor 22. The memory 23 is also used as a working memory while the processor 22 is executing various processes.

[0025] The recording medium 24 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the user terminal 2. The recording medium 24 records various programs executed by the processor 22. The display unit 25 is, for example, an LCD, and displays predetermined images. The input unit 26 is, for example, a touch panel, and is used when the user performs predetermined operations. The imaging unit 27 is equipped with a camera and acquires captured still image data and video data.

[0026] 4 is a block diagram showing an example of the functional configuration of the server 1. Functionally, the server 1 includes a face image acquisition unit 40, an arousal state determination unit 45, a feature calculation unit 50, a cognitive function estimation unit 60, and an output unit 70. The face image acquisition unit 40, the arousal state determination unit 45, the feature calculation unit 50, the cognitive function estimation unit 60, and the output unit 70 are realized by the processor 12 executing a program.

[0027] The server 1 acquires time-series information on the degree of eye opening during periods when the subject is awake from the facial image D1, and estimates the subject's cognitive function based on feature values ​​calculated from the acquired information. The server 1 also estimates the subject's cognitive function and transmits the estimation result to the user terminal 2 for display.

[0028] The facial image acquisition unit 40 acquires a facial image D1 of the subject from the user terminal 2. When the subject wishes to measure their cognitive function, they operate their own user terminal 2 to launch a predetermined application. When the predetermined application is launched, the user terminal 2 outputs questions to the subject in an automated voice, such as "What's the date today?" or "What's the weather like today?" The user terminal 2 then captures a facial image of the subject while they are talking in response to the questions in the automated voice, and transmits the captured image to the server 1 as a facial image D1.

[0029] It is desirable that the facial image D1 be an image of approximately 1 to 10 minutes of free conversation during which the subject is considered to be awake. In this embodiment, a predetermined application is launched and facial image D1 is acquired while the subject is conversing by answering questions provided by an automated voice. However, the present disclosure is not limited to this. A specific video may be displayed and a facial image may be acquired while the subject is watching the video, or a simple game may be displayed and a facial image may be acquired while the subject is playing the game. In this way, when capturing facial image D1, it is possible to design the facial image D1 so that the subject is less aware that it is a test of cognitive function.

[0030] Furthermore, the facial image D1 may be a facial image of the subject captured during an online call with a stranger or family member, or may be a facial image of the subject captured during an online medical consultation with a doctor or psychologist. Furthermore, the facial image may be captured from a home party or while traveling, for example. In other words, the facial image D1 does not have to be a facial image captured for a cognitive function test, and may be a facial image of the subject captured for any purpose.

[0031] Furthermore, in this embodiment, the facial image acquisition unit 40 of the server 1 acquires the facial image D1 from the user terminal 2, but the present disclosure is not limited to this, and the facial image D1 may be acquired from a terminal used by the subject's family or a specified DB, or the facial image D1 may be acquired on the cloud.

[0032] The wakefulness state determination unit 45 determines the wakefulness state of the subject. Here, a specific example of a method for determining the wakefulness state will be described. In one example, the wakefulness state determination unit 45 determines whether or not the subject is talking to a person or an automated voice based on the facial image D1 using voice recognition, and automatically determines that the subject is wakeful if the subject is talking. In another example, the wakefulness state determination unit 45 may automatically determine whether or not the subject is wakeful based on the facial image D1 using a technology for estimating drowsiness.

[0033] Note that a technology for estimating drowsiness is described in, for example, Patent Document "WO 2019 / 123569," which is incorporated herein by reference. Specifically, this document describes a drowsiness estimation device that analyzes a facial image of a subject, calculates feature amounts from a time-series signal of eye opening degree, and estimates the subject's drowsiness from the feature amounts.

[0034] In yet another example, the wakefulness determination unit 45 may determine whether or not the subject is in an wakeful state by obtaining a subjective evaluation of the facial image D1 by a family member or other person who knows the subject well. In yet another example, the wakefulness determination unit 45 may determine whether or not the subject is in an wakeful state by obtaining an evaluation of the facial image D1 by a third party, referring to an index such as an awake facial image used in existing technology for estimating drowsiness. In yet another example, the wakefulness determination unit 45 may determine whether or not the subject is in an wakeful state by having the subject answer whether or not they are wakeful during, or before, or after, capturing the facial image D1, and obtaining the subjective value of the answer.

[0035] The feature amount calculation unit 50 first calculates, from the facial image D1, time-series information regarding the degree of eye opening during the interval determined to be an awake state. Fig. 5 is an example of a graph showing time-series information regarding the degree of eye opening, with the vertical axis representing the degree of eye opening and the horizontal axis representing time. The degree of eye opening is a measure of the degree to which the eyes are open, and is also called the eyelid opening degree. An eye opening value of "0" indicates that the subject's eyes are completely closed, and a value of "1" indicates that the subject's eyes are completely open.

[0036] The feature calculation unit 50 also calculates feature amounts from time-series information regarding the degree of eye opening. Specifically, the feature calculation unit 50 calculates feature amounts of eyelid fluctuation (hereinafter also referred to as "eyelid fluctuation feature amounts") from the time-series information regarding the degree of eye opening, for example, using a one-minute window width with a 10-second shift, and then calculates the average and standard deviation of the section determined to be in an awake state. Note that a technique for analyzing a face video and calculating feature amounts from time-series information regarding the degree of eye opening is described, for example, in the aforementioned patent document "WO 2019 / 123569."

[0037] Here, the eyelid fluctuation feature calculated from the time-series information on the degree of eye opening will be described. The feature calculation unit 50 calculates five feature amounts as eyelid fluctuation feature amounts from the time-series information on the degree of eye opening: eye closure rate (PERCLOS), movement speed, blink count, time fluctuation, and left-right asymmetry.

[0038] The eye closure percentage is the percentage of eyes that are open below a threshold. FIG. 6 is an example of a graph showing the average eye closure percentage for multiple subjects classified as cognitively normal. The vertical axis represents the eye closure percentage, with values ​​increasing in the direction of the arrow. Bar 31 in FIG. 6 represents the lower limit of the eye closure percentage, bar 32 represents the upper limit of the eye closure percentage, box 33 represents the range of eye closure percentages that includes 25% to 75% of subjects, and thick line 34 represents the average eye closure percentage for all subjects. Note that white circles on the graph represent outliers.

[0039] The movement speed is the speed from closing the eyes to opening them, that is, the blink speed.

[0040] The number of blinks is the number of blinks in a given period of time.

[0041] Time fluctuation is the degree of dispersion of the time-series information of the degree of eye opening. Figure 7 is an example of a graph with the vertical axis representing the degree of eye opening and the horizontal axis representing time, with the solid line representing the time-series information of the degree of eye opening and the thick line representing the time fluctuation. Generally, when a person is awake, they keep their eyelids constant except for blinking, so as shown in Figure 7, the time fluctuation is located at the top with little dispersion.

[0042] The left-right difference is the value of the difference in movement between the time-series information on the eye opening degree of the left eye and the time-series information on the eye opening degree of the right eye. Fig. 8 is an example of a graph with the vertical axis representing the eye opening degree and the horizontal axis representing time, where the solid line represents the time-series information on the eye opening degree of the left eye, the dotted line represents the time-series information on the eye opening degree of the right eye, and the thick line represents the left-right difference.

[0043] The cognitive function estimation unit 60 estimates the cognitive function of the subject based on the time-series information regarding the degree of eye opening calculated by the feature calculation unit 50. Specifically, the cognitive function estimation unit 60 estimates the cognitive function of the subject based on the eyelid fluctuation feature calculated by the feature calculation unit 50.

[0044] For example, the cognitive function estimation unit 60 determines whether the subject has a significant decline in cognitive function and whether dementia is suspected, based on the average and standard deviation of the percentage of eyes closed, based on experimental results described below. As will be described in detail later, the cognitive function estimation unit 60 determines whether the subject has a significant decline in cognitive function and whether dementia is suspected, when the average and standard deviation of the percentage of eyes closed of the subject are increased or fluctuate significantly compared to the average and standard deviation of the percentage of eyes closed of healthy subjects.

[0045] Furthermore, the cognitive function estimation unit 60 determines whether the subject has a mild decline in cognitive function or a suspected mild cognitive impairment based on the average and standard deviation of the movement speed. As will be described in detail later, the cognitive function estimation unit 60 determines whether the subject has a mild decline in cognitive function or a suspected mild cognitive impairment if the average and standard deviation of the movement speed of the subject are increased or fluctuate significantly compared to the average and standard deviation of the movement speed of healthy subjects. Furthermore, the cognitive function estimation unit 60 determines whether the subject has normal cognitive function if there are no particular problems with the various feature amounts.

[0046] The output unit 70 outputs the estimation result by the cognitive function estimation unit 60. Specifically, the output unit 70 transmits the classification determined by the cognitive function estimation unit 60, such as "cognitively normal," "mild cognitive impairment," or "dementia," as the estimation result to the user terminal 2 and displays it. The estimation result may include phrases such as "significant decline in cognitive function" or "mild decline in cognitive function."

[0047] In the present disclosure, the cognitive function estimation unit 60 determines whether the subject is "cognitively normal," "suspected mild cognitive impairment," or "suspected dementia" based on the eyelid fluctuation feature amount, and the output unit 70 outputs the determined classification as an estimated result of cognitive function, but the present disclosure is not limited to this. The cognitive function estimation unit 60 may calculate an MMSE score equivalent based on the eyelid fluctuation feature amount, and the output unit 70 may output the calculated MMSE score equivalent as an estimated result of cognitive function. Furthermore, the cognitive function estimation unit 60 may calculate a score equivalent of any test that evaluates cognitive function, such as the Japanese version of the Mild Cognitive Impairment Assessment (MoCA-J), without being limited to the MMSE, and output the calculated score equivalent as an estimated result.

[0048] In the above configuration, the face image acquisition unit 40, the wakefulness state determination unit 45, and the feature calculation unit 50 of the server 1 are examples of the face image acquisition means, the wakefulness state determination means, and the calculation means of the present disclosure, respectively. The feature calculation unit 50 and the cognitive function estimation unit 60 of the server 1 are examples of the cognitive function estimation means of the present disclosure.

[0049] (Experiment) Here, we will explain an experiment to determine the correlation between eyelid fluctuation features calculated from time-series information on the degree of eye opening and cognitive function. The experimental subjects were multiple dementia patients, mild cognitive impairment patients, and healthy individuals. In the experiment, facial images of each subject in an awake state were analyzed, and the following features were calculated from the time-series information on the degree of eye opening: eye closure rate, movement speed, blink count, time fluctuation, and laterality. Then, the average and standard deviation of each feature were calculated for each group of dementia patients, mild cognitive impairment patients, and healthy individuals.

[0050] FIG. 9( a) is an example of a graph showing the average eye closure percentage based on experimental results, with the vertical axis representing the percentage of closed eyes and the horizontal axis representing the group. As shown in FIG. 9( a), a comparison between a group of healthy subjects and a group of dementia patients revealed that the average eye closure percentage was higher in the group of dementia patients compared to the group of healthy subjects, and that the variability, which is the difference between the upper and lower limits, also increased. FIG. 9( b) is an example of a graph showing the standard deviation of the eye closure percentage based on experimental results, with the vertical axis representing the percentage of closed eyes and the horizontal axis representing the group. As shown in FIG. 9( b), a comparison between a group of healthy subjects and a group of dementia patients revealed that, similar to FIG. 9( a), the standard deviation of the eye closure percentage was higher and the variability also increased in the group of dementia patients compared to the group of healthy subjects. In other words, a comparison between the two groups, the group of healthy subjects and the group of dementia patients, revealed significant differences in the average eye closure percentage and standard deviation. Furthermore, when the time-series information of the eye opening degree was divided into multiple frames to calculate eyelid fluctuation features, the same trends were found for both the average eye closure percentage and standard deviation, even when the difference from the previous frame was calculated.

[0051] Figure 10 is an example of a graph showing the average movement speed based on experimental results, with the vertical axis representing the movement speed value and the horizontal axis representing the group. Here, the movement speed increases in the direction of the arrow on the vertical axis. In Figure 10, a comparison between a group of healthy subjects and a group of mild cognitive impairment patients revealed that the average movement speed increased in the group of mild cognitive impairment patients compared to the group of healthy subjects, and the variability, which is the difference between the upper and lower limits, also increased. In other words, the speed at which the eyelids opened and closed fluctuated. In other words, a comparison between the group of healthy subjects and the group of mild dementia patients revealed a significant difference in the average movement speed. When the time series information of eye opening degree was divided into multiple frames and feature values ​​were calculated, the average movement speed showed the same trend, even when the difference from the previous frame was taken. The standard deviation of the movement speed also showed the same trend.

[0052] From the above, the experimental results showed that when cognitive function declines significantly, the average percentage of eyes closed increases, and the variability in the percentage of eyes closed also increases.In addition, when cognitive function declines slightly, the average speed of movement increases, and the variability in speed of movement also increases.It is assumed that these results are related to the fact that when apathy, one of the early symptoms of dementia, occurs, people lose motivation, their brains become less excited, and they become more prone to drowsiness, and that when people develop dementia, their sleep rhythm is more likely to be disrupted, and they become more prone to drowsiness due to lack of sleep at night.

[0053] 9 and 10 , the cognitive function estimation unit 60 of the server 1 determines that the subject's cognitive function is significantly impaired and that dementia is suspected if the average and standard deviation of the subject's eye closure percentage are increased or fluctuate significantly compared to the average and standard deviation of healthy subjects. Furthermore, the cognitive function estimation unit 60 determines that the subject's cognitive function is slightly impaired and that mild cognitive impairment is suspected if the average and standard deviation of the subject's movement speed are increased or fluctuate significantly compared to the average and standard deviation of healthy subjects.

[0054] Fig. 11 is a table showing the correlation between cognitive function by purpose and eyelid fluctuation feature amount based on the experimental results. In the table of Fig. 11, the vertical axis represents cognitive function by purpose, and the horizontal axis represents the average and standard deviation of eye closure rate, movement speed, time fluctuation, laterality, and blink count, and the correlation is indicated by four levels: very high correlation, slightly high correlation, slightly low correlation, and low correlation using symbols such as circles and triangles.

[0055] Here, examples of objective-specific cognitive functions include cognitive synthesis, memory, Alzheimer's disease symptom index, frontal lobe function, language function, depression scale, behavioral / psychological index, activity, and executive function. Cognitive synthesis is a comprehensive cognitive function and is evaluated, for example, by MMSE, MoCA-J, or the like. Memory is evaluated, for example, from the results of a delayed recall test. Furthermore, Alzheimer's disease symptom index is evaluated, for example, by ADAS (Alzheimer's Disease Assessment Scale), or the like. Frontal lobe function is evaluated, for example, by FAB (Frontal Assessment Battery), or the like. Language function is evaluated, for example, from the results of a conversation test. Depression scales are evaluated, for example, by GDS (Geriatric Depression Scale), or the like. Behavioral and psychological indicators are evaluated by, for example, the Neuropsychiatric Inventory (NPI), activity is evaluated by, for example, the Functional Activities Questionnaire (FAQ), and executive function is evaluated by, for example, the Trail Making Test.

[0056] 11 , for example, it can be seen that there is a high correlation between language function in cognitive functions and the average and standard deviation of eyelid fluctuation feature amounts, such as eye closure rate, time fluctuation, and blink count. It can also be seen that there is a high correlation between executive function in cognitive functions and the average and standard deviation of eyelid fluctuation feature amounts, such as eye closure rate, left-right difference, and blink count.

[0057] Based on the experimental results shown in Figure 11, the cognitive function estimation unit 60 of the server 1 can improve the accuracy of estimating cognitive function by referring to the correlation between cognitive function for each purpose and eyelid fluctuation feature and using various feature values ​​in combination.

[0058] (Cognitive Function Estimation Processing) Next, a description will be given of the cognitive function estimation processing performed by the server 1. Fig. 12 is a flowchart of the cognitive function estimation processing performed by the server 1. This processing is realized by the processor 12 shown in Fig. 2(a) executing a program prepared in advance.

[0059] First, the server 1 acquires a facial image D1 capturing a facial image of the subject (step S101). The server 1 then determines whether the subject captured in the facial image D1 is in an awake state (step S102). If the subject is not in an awake state (step S102; No), the server 1 returns to the process of step S101. On the other hand, if the subject is in an awake state (step S102; Yes), the server 1 calculates, from the facial image D1, time-series information regarding the degree of eye opening during the interval in which the subject is determined to be in an awake state. The server 1 then calculates eyelid fluctuation feature values ​​based on the time-series information regarding the degree of eye opening (step S103).

[0060] The server 1 estimates the cognitive function of the subject based on the calculated eyelid fluctuation feature amount (step S104). The server 1 then transmits the estimated cognitive function result to the user terminal 2 and displays it (step S105). This allows the subject to check the estimated cognitive function result.

[0061] In the present embodiment, the user terminal 2 is used by the subject, but the present disclosure is not limited to this, and the user terminal 2 may also be used by the subject's family member, a helper, or the like. In this case, the family member or helper operates the user terminal 2 to send the subject's facial image D1 to the server 1, and receive and display the estimated results of the subject's cognitive function. This makes it possible to easily check the subject's cognitive function even when the subject has difficulty operating the user terminal 2 or when the family member or helper has concerns about the subject's cognitive function.

[0062] As described above, the cognitive function estimation system 100 allows the estimation results of the subject's cognitive function to be confirmed using a commonly available device such as a smartphone, without the need for any special equipment. In other words, the subject's cognitive function can be easily measured. Furthermore, the subject's cognitive function can be estimated by extracting eyelid fluctuation features from the facial image D1, without the subject having to visit a medical institution. This allows the subject to easily understand their own cognitive function during their daily lives without any burden. Such measurements during daily life increase opportunities to check for the possibility of mild cognitive impairment or dementia, making it possible to detect decline in cognitive function earlier than before.

[0063] [First Modification] The cognitive function of the subject may be estimated using a cognitive function estimation model, which is a machine learning model. In the first modification, the server 1x references correlation information indicating the correlation between the cognitive function for each purpose and the eyelid fluctuation feature amount, and generates a cognitive function estimation model that has learned the relationship between the eyelid fluctuation feature amount and the evaluation of the cognitive function for each purpose. The server 1x then uses the cognitive function estimation model to estimate the optimized evaluation of the cognitive function for each purpose of the subject and output the estimation result.

[0064] 13 is a block diagram showing the functional configuration of a server 1x that uses a machine learning model. Functionally, the server 1x includes a face image acquisition unit 40, an awakening state determination unit 45, a feature calculation unit 50, a cognitive function estimation unit 60x, a teacher data storage unit 61, a model learning unit 62, a model storage unit 63, and an output unit 70.

[0065] The facial image acquisition unit 40, the arousal state determination unit 45, the feature calculation unit 50, the cognitive function estimation unit 60x, the model learning unit 62, and the output unit 70 are realized by the processor 12 executing a program. The teacher data storage unit 61 and the model storage unit 63 are realized by a predetermined DB connected to the memory 13 or the server 1x. For convenience, a description of functions similar to those in the above-described embodiment will be omitted.

[0066] The teacher data storage unit 61 stores teacher data used to train the cognitive function estimation model. The teacher data is created for eyelid fluctuation feature amounts that are considered to be highly correlated and purpose-specific cognitive functions based on the correlation between purpose-specific cognitive functions and eyelid fluctuation feature amounts as shown in FIG. 11 . Specifically, the input data is one or more values ​​of the eyelid fluctuation feature amounts, i.e., eye closure percentage, movement speed, time fluctuation, laterality, and blink count, and the correct answer data is an evaluation value of the cognitive function that is highly correlated with the eyelid fluctuation feature amount. For example, as shown in FIG. 11 , since it is known that the average eye closure percentage and the MMSE are highly correlated, teacher data is created in which the average eye closure percentage is used as input data and the corresponding MMSE evaluation value (score) is used as correct answer data.

[0067] The training data may also be generated using multiple eyelid fluctuation feature quantities as input data, with the correct answer data being an evaluation value of a specific cognitive function that is highly correlated with these feature quantities. For example, as shown in FIG. 11 , since there is a high correlation between language function in cognitive functions and the average and standard deviation of eyelid fluctuation feature quantities, such as the percentage of eye closure, time fluctuation, and blink count, the training data may be generated using a combination of the average and standard deviation of the percentage of eye closure, time fluctuation, and blink count as input data, with the correct answer data being an evaluation value of the corresponding language function. Furthermore, since there is a high correlation between executive function in cognitive functions and the eyelid fluctuation feature quantities, such as the percentage of eye closure, left-right difference, and the average and standard deviation of the blink count, the training data may be generated using a combination of the percentage of eye closure, left-right difference, and the average and standard deviation of the blink count as input data, with the correct answer data being an evaluation value of the executive function that corresponds to these.

[0068] The model learning unit 62 learns a cognitive function estimation model using the training data prepared in advance as described above. Specifically, the model learning unit 62 uses the training data to train the cognitive function estimation model so as to estimate and output an optimized, objective-specific cognitive function assessment from the input eyelid fluctuation feature amount. The model learning unit 62 then stores the learned cognitive function estimation model in the model storage unit 63.

[0069] The learning algorithm for the cognitive function estimation model may be any machine learning method, such as a neural network, an SVM (Vector Machine), logistic regression, etc. In addition, although one cognitive function estimation model is used in this modification, the present disclosure is not limited to this, and multiple cognitive function estimation models may be generated for each purpose of cognitive function to be measured and stored in the model storage unit 63.

[0070] The cognitive function estimation unit 60x uses a trained cognitive function estimation model stored in the model memory unit 63 to estimate an evaluation of the subject's cognitive function for each purpose based on the eyelid fluctuation features calculated by the feature calculation unit 50.

[0071] In the above configuration, the face image acquisition unit 40, the wakefulness state determination unit 45, and the feature calculation unit 50 of the server 1x are examples of the face image acquisition means, the wakefulness state determination means, and the calculation means of the present disclosure. The feature calculation unit 50 and the cognitive function estimation unit 60x of the server 1x are examples of the cognitive function estimation means of the present disclosure. The server 1x may store correlation information indicating the correlation between the cognitive function for each purpose and the eyelid fluctuation feature amount in advance, or may acquire the information from a predetermined database, etc. This is an example of the correlation information acquisition means of the present disclosure.

[0072] (Learning Process) Next, the learning process performed by the server 1x will be described. Fig. 14 is a flowchart of the learning process for training a machine learning model. This process is realized by the processor 12 shown in Fig. 2 executing a program prepared in advance.

[0073] First, the server 1x acquires training data from the training data storage unit 61 (step S201). Specifically, the server 1x acquires eyelid fluctuation feature amounts as input data and acquires correct data for the objective-specific cognitive function evaluation values ​​corresponding to the eyelid fluctuation feature amounts. Next, the server 1x uses a cognitive function estimation model to estimate the objective-specific cognitive function evaluation values ​​of the subject based on the acquired eyelid fluctuation feature amounts and compares them with the correct data (step S202). The server 1x then updates the cognitive function estimation model so as to reduce the error between the evaluation value estimated by the cognitive function estimation model and the correct data (step S203). The server 1x repeats this process with different training data, thereby training the cognitive function estimation model to improve the estimation accuracy of the cognitive function evaluation.

[0074] (Cognitive Function Estimation Process) Next, the cognitive function estimation process performed by the server 1x will be described. Fig. 15 is a flowchart of the cognitive function estimation process using a machine learning model. This process is realized by the processor 12 shown in Fig. 2 executing a program prepared in advance.

[0075] First, the server 1x acquires a facial image D1 capturing a facial image of the subject (step S301). Next, the server 1x determines whether the subject shown in the facial image D1 is in an awake state (step S302). If the subject is not in an awake state (step S302; No), the server 1x returns to the processing of step S301. On the other hand, if the subject is in an awake state (step S302; Yes), the server 1x calculates, from the facial image D1, time-series information regarding the degree of eye opening during the interval in which the subject is determined to be in an awake state. Then, the server 1x calculates eyelid fluctuation feature values ​​based on the time-series information regarding the degree of eye opening (step S303).

[0076] Next, the server 1x estimates the evaluation of the cognitive function for each purpose of the subject based on the calculated eyelid fluctuation feature amount using the trained cognitive function estimation model (step S304).The server 1x then transmits the evaluation of the cognitive function for each purpose output by the cognitive function estimation model as an estimation result to the user terminal 2 and displays it (step S305).This allows the subject to check the estimation result of the cognitive function for each purpose.

[0077] As described above, according to the server 1x of the first modification, it is possible to estimate the evaluation of the subject's cognitive function for each purpose using a cognitive function estimation model that has learned the correlation between cognitive function for each purpose and eyelid fluctuation feature amount. The trained cognitive function estimation model estimates the evaluation of cognitive function based on the correlation, thereby improving estimation accuracy.

[0078] In this modification, the cognitive function estimation model calculates eyelid fluctuation features based on time-series information regarding the degree of eye opening, and outputs an evaluation of cognitive function for each purpose by inputting one or more calculated features. However, the present disclosure is not limited to this, and time-series information regarding the degree of eye opening may be directly input to output an evaluation of cognitive function for each purpose. In this case, the model learning unit 62 trains the cognitive function estimation model so that it outputs an evaluation of cognitive function for each purpose by inputting time-series information regarding the degree of eye opening.

[0079] (Second Modification) In the above embodiment, the subject uses the user terminal 2, but the present disclosure is not limited to this, and the subject may use a user terminal 2x that has the functions of the server 1. In this case, the user terminal 2x can execute the estimation process that was performed by the server 1, estimate the cognitive function of the subject, and output the estimation result. In other words, the acquisition of the facial image D1 of the subject, the estimation of the cognitive function of the subject, and the output of the estimation result can all be performed by the user terminal 2x alone.

[0080] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0081] (Supplementary Note 1) An information processing device comprising: a facial image acquisition means for acquiring a facial image of a subject; an awake state determination means for determining the awake state of the subject; a calculation means for calculating, from the facial image, information relating to the degree of eye opening in a section determined to be an awake state by the awake state determination means; and a cognitive function estimation means for estimating the cognitive function of the subject based on the information relating to the degree of eye opening.

[0082] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the cognitive function estimation means calculates a feature amount based on information related to the degree of eye opening, and estimates the cognitive function of the subject based on the feature amount.

[0083] (Supplementary Note 3) The information processing device according to Supplementary Note 1, wherein the cognitive function estimation means calculates one or more of an interval mean and a standard deviation of the eye closure percentage as feature quantities based on information related to the degree of eye opening, and estimates a significant decline in cognitive function and dementia of the subject based on one or more of the interval mean and the standard deviation of the eye closure percentage.

[0084] (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the cognitive function estimation means calculates one or more of an interval average and a standard deviation of blinking speed as features based on information related to the degree of eye opening, and estimates a mild decline in cognitive function and mild cognitive impairment of the subject based on one or more of the interval average and the standard deviation of the blinking speed.

[0085] (Supplementary Note 5) The information processing device according to Supplementary Note 1 includes a correlation information acquisition means for acquiring correlation information indicating a correlation between a feature calculated from information relating to the degree of eye opening and a cognitive function, wherein the cognitive function estimation means calculates, based on the information relating to the degree of eye opening, the percentage of eye closure, the blinking speed, the blinking time fluctuation, the difference between the left eye and the right eye, and the interval average and standard deviation of each of the number of blinks as feature amounts, and estimates an evaluation of the cognitive function of the subject based on one or more of the percentage of eye closure, the blinking speed, the time fluctuation, the difference between the left eye and the right eye, and the interval average and standard deviation of each of the number of blinks by referring to the correlation information.

[0086] (Supplementary Note 6) The information processing device according to Supplementary Note 5, wherein the cognitive function estimation means estimates an assessment of language function in the cognitive function of the subject based on the interval average and standard deviation of the eye closure rate, the time fluctuation, and the number of blinks.

[0087] (Supplementary Note 7) The information processing device according to Supplementary Note 5, wherein the cognitive function estimation means estimates an assessment of executive function in the cognitive function of the subject based on the interval average and standard deviation of the eye closure rate, the difference between the left and right eyes, and the number of blinks.

[0088] (Supplementary Note 8) The information processing device according to Supplementary Note 2, wherein the cognitive function estimation means estimates the cognitive function of the subject using a machine learning model that has been trained to output an optimized assessment of cognitive function in response to input of the feature.

[0089] (Supplementary Note 9) An information processing method that acquires a facial image of a subject, determines the wakefulness state of the subject, calculates information relating to the degree of eye opening during a section determined to be the wakefulness state from the facial image, and estimates the cognitive function of the subject based on the information relating to the degree of eye opening.

[0090] (Appendix 10) A recording medium having recorded thereon a program that causes a computer to execute the following processes: acquiring a facial image of a subject; determining the wakefulness state of the subject; calculating information relating to the degree of eye opening during a section determined to be the wakefulness state from the facial image; and estimating the cognitive function of the subject based on the information relating to the degree of eye opening.

[0091] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. In other words, the present disclosure naturally includes various modifications and alterations that would be possible for a person skilled in the art in accordance with the entire disclosure, including the claims, and the technical concept.

[0092] REFERENCE SIGNS LIST 1, 1x, Server 2 User terminal 11, 21 Interface 12, 22 Processor 13, 23 Memory 14, 24 Recording medium 15, 25 Display unit 16, 26 Input unit 27 Imaging unit 40 Facial image acquisition unit 45 Arousal state determination unit 50 Feature amount calculation unit 60 Cognitive function estimation unit 70 Output unit 100 Cognitive function estimation system

Claims

1. An information processing apparatus comprising: a face image acquisition means for acquiring a face image of a subject; a wakefulness state determination means for determining the wakefulness state of the subject; a calculation means for calculating information regarding the eye opening degree in a section determined to be the wakefulness state by the wakefulness state determination means from the face image; and a cognitive function estimation means for estimating the cognitive function of the subject based on the information regarding the eye opening degree.

2. The information processing apparatus according to claim 1, wherein the cognitive function estimation means calculates a feature amount based on the information regarding the eye opening degree and estimates the cognitive function of the subject based on the feature amount.

3. The information processing apparatus according to claim 1, wherein the cognitive function estimation means calculates, as a feature amount, any one or more of the interval average and standard deviation of the closed-eye ratio based on the information regarding the eye opening degree, and estimates a significant decline in the cognitive function and dementia of the subject based on any one or more of the interval average and standard deviation of the closed-eye ratio.

4. The information processing apparatus according to claim 1, wherein the cognitive function estimation means calculates, as a feature amount, any one or more of the interval average and standard deviation of the blink operation speed based on the information regarding the eye opening degree, and estimates a mild decline in the cognitive function and mild cognitive impairment of the subject based on any one or more of the interval average and standard deviation of the blink operation speed.

5. The information processing apparatus according to claim 1, further comprising a correlation information acquisition means for acquiring correlation information indicating a correlation between a feature amount calculated from the information regarding the eye opening degree and the cognitive function, wherein the cognitive function estimation means calculates, as feature amounts, the interval average and standard deviation of each of the closed-eye ratio, blink operation speed, blink time fluctuation, left-right difference which is the difference between the left eye and the right eye, and blink count based on the information regarding the eye opening degree, refers to the correlation information, and estimates an evaluation of the cognitive function of the subject based on one or more of the interval average and standard deviation of each of the closed-eye ratio, operation speed, time fluctuation, left-right difference, and blink count.

6. The information processing apparatus according to claim 5, wherein the cognitive function estimation means estimates an evaluation of the language function in the cognitive function of the subject based on the interval average and standard deviation of each of the closed-eye ratio, time fluctuation, and blink count.

7. An information processing device as described in claim 5, wherein the cognitive function estimation means estimates an assessment of executive function in the cognitive function of the subject based on the interval average and standard deviation of the eye closure percentage, the difference between the left and right sides, and the number of blinks.

8. The information processing device described in claim 2, wherein the cognitive function estimation means estimates the cognitive function of the subject using a machine learning model trained to output an optimized assessment of cognitive function in response to the input of the features.

9. An information processing method for acquiring a facial image of a subject, determining the subject's state of wakefulness, calculating information relating to the degree of eye opening during a section determined to be the wakefulness state from the facial image, and estimating the subject's cognitive function based on the information relating to the degree of eye opening.

10. A recording medium having recorded thereon a program that causes a computer to execute the following processes: acquiring facial images of a subject; determining the subject's state of wakefulness; calculating information relating to the degree of eye opening during sections determined to be in an wakefulness state from the facial images; and estimating the subject's cognitive function based on the information relating to the degree of eye opening.

Images