Cognitive function assessment device, cognitive function assessment system, learning model generation device, cognitive function assessment method, learning model production method, and program
The system uses a learning model to analyze facial expressions to estimate cognitive function values equivalent to MMSE without traditional medical tests, addressing the burden and comparison challenges of existing methods.
Patent Information
- Application Number
- JP2020181215
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-10-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-10-29
AI Technical Summary
Existing technologies for cognitive function assessment, such as the MMSE, are burdensome for patients and do not provide a direct correlation between unique index values and medical cognitive function evaluation scales, making it difficult to compare results easily and requiring in-person medical examinations.
A system that uses a learning model to determine cognitive function by analyzing facial expression scores derived from image data, allowing individuals to express specific facial expressions over defined periods, capturing image data, and correlating these scores with medical cognitive function evaluation scales without requiring a traditional medical test.
Enables the estimation of cognitive function values equivalent to medical tests like MMSE through facial expression analysis, reducing the burden on patients and providing accurate, easily comparable results.
Smart Images

Figure 0007580716000002 
Figure 0007580716000003 
Figure 0007580716000004
Abstract
Description
[Technical field]
[0001] The present invention relates to a cognitive function assessment device and related techniques. [Background technology]
[0002] In the treatment of dementia, medical tests such as the Mini-Mental State Examination (MMSE) are conducted, and the test values obtained from the medical tests are used as objective values indicating the degree of cognitive function (such as the degree of cognitive decline). For example, the MMSE is conducted by having the subject answer a dozen or so questions, and the test value of the subject is obtained as a score out of 30. The test value obtained from the medical test (the value that evaluates cognitive function) is also called the medical cognitive function assessment scale.
[0003] On the other hand, a technique has been proposed that provides information useful for diagnosing dementia and the like by analyzing images of a subject (see Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2018-7792 A Summary of the Invention [Problem to be solved by the invention]
[0005] The technology of Patent Document 1 mentioned above describes a method of determining whether a user belongs to a healthy group or a mild cognitive impairment group based on a learning model trained on vector information and time information, etc., of each part of the face of multiple subjects.
[0006] However, the output result is a judgment result as to whether the user belongs to a healthy group or a mild cognitive impairment group, but does not indicate the degree of cognitive function. Moreover, the output result is a unique index and is not a test value (medical test value indicating the degree of cognitive function) obtained by a medical test performed in medical treatment, etc. In particular, the specific relationship between the unique index value and a predetermined medical test value (medical cognitive function assessment scale) is not necessarily clear.
[0007] Therefore, it is difficult to evaluate the unique index while understanding its correspondence with a specified medical cognitive function assessment scale, or in short, to directly compare the unique index with a specified medical cognitive function assessment scale (at the same level).
[0008] In addition, medical tests such as the MMSE are not necessarily easy to administer, since doctors and other medical professionals ask questions directly to subjects and there are a certain number of questions to be asked. In addition, patients have to go to the hospital to be examined, which can be a burden for them.
[0009] In particular, considering that the number of elderly people and dementia patients is likely to increase as the aging population continues, it is preferable to be able to more easily obtain a medical evaluation of cognitive function.
[0010] Therefore, an objective of the present invention is to provide a technology that makes it possible to obtain a value equivalent to a medical cognitive function assessment scale based on a medical test, without actually conducting a medical test on the person whose cognitive function is to be assessed. [Means for solving the problem]
[0011] In order to solve the above problems, the cognitive function assessment device according to the present invention includes a control unit that estimates a value corresponding to the predetermined medical cognitive function assessment scale of a person based on information about the facial expression score of a person to be assessed for cognitive function, using a learning model that has learned the relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an assessment result of the cognitive function of each of the plurality of subjects, the facial expression score being a value indicating the degree of a specific facial expression and being obtained based on photographed image data of a subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects being a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, in accordance with which the photographed image data is generated, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by:
[0012] The specific facial expression may be a smile, and the facial expression score may be a value indicating the degree of a smile.
[0013] The captured image data may include a plurality of images in a time series, and the information on the facial expression scores may include information on a plurality of facial expression scores obtained respectively from the plurality of images.
[0014] The information regarding the facial expression score may include information regarding a maximum value of two or more facial expression scores acquired in a specific period of the captured image data.
[0015] The information regarding the facial expression score may include information regarding an average value of two or more facial expression scores acquired during a specific period of the captured image data.
[0016] The information on the facial expression scores may include information on a standard deviation of two or more facial expression scores acquired during a specific period of the captured image data.
[0018] Each of the specific facial expressions The duration is determined by calculating a second derivative curve of the time-dependent change curve of the facial expression score extracted from the captured image data. Each instruction for expressing the specific facial expressionAlternatively, the second derivative curve may be calculated as the length of time from a first point in time at which the second derivative curve reaches a maximum value equal to or greater than a predetermined value within a period close to the point in time at which the second derivative curve is applied, to a second point in time after the first point in time at which the second derivative curve again reaches a maximum value equal to or greater than a predetermined value.
[0020] before The information about the facial expression score includes information about a rise time of the specific facial expression, and the rise time is a second derivative curve of a time-varying curve of the facial expression score extracted from the captured image data. The specific facial expression The second derivative curve may be calculated as the length of time from a first point in time at which the second derivative curve reaches a maximum value equal to or greater than a predetermined value within a period surrounding the time at which the instruction to start expression is given, to a second point in time at which the second derivative curve first reaches a minimum value after the first point in time.
[0021] before The information on the facial expression scores includes information on an average value of two or more facial expression scores acquired in the first period, information on an average value of two or more facial expression scores acquired in the second period, information on an average value of two or more facial expression scores acquired in the first period and the second period, information on a standard deviation of two or more facial expression scores acquired in the first period and the first rest period, information on a standard deviation of two or more facial expression scores acquired in the second period and a second rest period after the second period, information on a standard deviation of two or more facial expression scores acquired in the first period, the first rest period, the second period, and the second rest period, and information on a maximum value of two or more facial expression scores acquired in the first period and the second period. ,before A first sustained period is a period during which the specific facial expression continues from the time when the first instruction is given or near the time when the first instruction is given. Take a break Information on the average value of the two or more facial expression scores obtained, and a second duration, which is a period during which the specific facial expression has continued from the time when the second instruction is given or near the time when the second instruction is given, In between At least one of information on an average value of two or more facial expression scores obtained, information on a maximum value of two or more facial expression scores obtained in the first duration, and information on a maximum value of two or more facial expression scores obtained in the second duration. moreover It may include.
[0022] before The information on the facial expression scores includes information on an average value of two or more facial expression scores acquired in the first period, information on an average value of two or more facial expression scores acquired in the second period, information on a standard deviation of two or more facial expression scores acquired in the first period and the first rest period, and information on a standard deviation of two or more facial expression scores acquired in the first period, the first rest period, the second period, and a second rest period after the second period. And, It may include.
[0025] the control unit, upon receiving an instruction to express the specific facial expression, causes a reference face image relating to the specific facial expression to be displayed on a predetermined display unit; The reference face image may be an average face image obtained by averaging a plurality of face images of a plurality of persons having the specific facial expression.
[0026] In order to solve the above problem, the cognitive function assessment system of the present invention is characterized by comprising a cognitive function assessment device having any of the above-mentioned characteristics, and a terminal device that transmits the captured image data of the person to the cognitive function assessment device.
[0027] In order to solve the above problem, the learning model generation device according to the present invention includes a control unit that performs machine learning on a relationship between information on the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects to generate a learning model, the facial expression score being a value indicating the degree of a specific facial expression and being obtained based on photographed image data obtained by photographing a subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects being a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, and the photographed image data is generated accordingly, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by:
[0028] In order to solve the above problem, the cognitive function assessment method according to the present invention includes a step of estimating a value corresponding to the predetermined medical cognitive function assessment scale of a person to be assessed for cognitive function based on information about the facial expression score of the person, using a learning model that has learned the relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an assessment result of the cognitive function of each of the plurality of subjects, wherein the facial expression score is a value indicating the degree of a specific facial expression and is obtained based on photographed image data of the subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, in accordance with which the photographed image data is generated, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by:
[0029] In order to solve the above problem, the learning model creation method according to the present invention includes a step of generating a learning model by machine learning a relationship between information on the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects, the facial expression score being a value indicating the degree of a specific facial expression and being obtained based on photographed image data obtained by photographing a subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects being a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, in accordance with which the photographed image data is generated, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by:
[0031] In order to solve the above problem, the present invention provides a program for causing a computer to execute a step of estimating a value corresponding to the predetermined medical cognitive function assessment scale of a person based on information about the facial expression score of a person to be assessed for cognitive function, using a learning model that has learned the relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects, wherein the facial expression score is a value indicating the degree of a specific facial expression and is obtained based on photographed image data of a subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, in accordance with which the photographed image data is generated, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by:
[0032] In order to solve the above problem, the present invention provides a program for causing a computer to execute a step of performing machine learning on a relationship between information on the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects to generate a learning model, wherein the facial expression score is a value indicating the degree of a specific facial expression and is obtained based on photographed image data obtained by photographing a subject, and the predetermined medical cognitive function assessment scale of each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects. a first instruction is given to the person to be photographed to continuously express the specific facial expression during a first period out of a first period and a second period which are two periods sandwiching a rest period, and a second instruction is given to the person to be photographed to continuously express the specific facial expression during the second period following a first rest period which is a rest period following the first period, and the photographed image data is generated accordingly, and the information on the facial expression score includes information on the duration of the specific facial expression during the first period and information on the duration of the specific facial expression during the second period. It is characterized by: Effect of the Invention
[0033] According to the present invention, it is possible to obtain a value equivalent to a medical cognitive function assessment scale based on a medical test, without actually conducting a medical test on the person whose cognitive function is to be assessed. [Brief description of the drawings]
[0034] [Figure 1] FIG. 1 is a schematic diagram showing a cognitive function assessment system. [Diagram 2] FIG. 2 is a diagram showing functional blocks of the cognitive function assessment device. [Diagram 3] FIG. 2 is a diagram illustrating functional blocks of a terminal device. [Figure 4] FIG. 13 is a conceptual diagram showing the processing in the learning stage. [Diagram 5] FIG. 1 is a conceptual diagram showing the processing at the inference stage. [Figure 6] 13 is a flowchart showing the process of the learning stage. [Figure 7] 13 is a flowchart showing the process of the inference stage. [Figure 8] FIG. 13 is a diagram showing the display screen immediately before the first smile expression period. [Figure 9] FIG. 13 is a diagram showing a display screen relating to a first smile expression instruction, etc.; [Figure 10] FIG. 13 is a diagram showing a display screen during a first smile expression period. [Figure 11] FIG. 13 is a diagram showing a display screen during a first pause period immediately following a first smile expression period. [Figure 12] FIG. 13 is a diagram showing the display screen immediately before the second smile expression period. [Figure 13] FIG. 13 is a diagram showing a display screen relating to a second smile expression instruction, etc.; [Figure 14] 13 is a diagram showing a display screen during a second pause period immediately following a second smile expression period. FIG. [Figure 15] FIG. 13 is a diagram showing a display screen of an inference result. [Figure 16] FIG. 13 is a diagram showing changes in facial expression scores over time for a certain person. [Figure 17] FIG. 13 is a diagram showing an overview of certain learning results, etc. [Figure 18] FIG. 13 is a diagram showing the relationship between actual MMSE test values and MMSE equivalent values (cognition level scores) estimated using a trained model. [Figure 19] FIG. 18 is a diagram showing a table (upper part) indicating the correlation between each of a plurality of parameters and MMSE test values, and a diagram showing the parameters (lower part) that were actually used in the evaluation test of FIG. 17 among these parameters. [Figure 20] FIG. 13 is a diagram showing a time-dependent change curve of facial expression score and its differential curve, etc. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0035] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0036] <1. System Overview> Fig. 1 is a schematic diagram showing a cognitive function assessment system 1. As shown in Fig. 1, the cognitive function assessment system 1 includes a cognitive function assessment device 30 that assesses the level of cognitive function, and a plurality of terminal devices 70 (see also Figs. 2 and 3). Fig. 2 is a diagram showing functional blocks of the cognitive function assessment device 30, and Fig. 3 is a diagram showing functional blocks of the terminal device 70.
[0037] The cognitive function assessment device 30 and each terminal device 70 can communicate with each other via a network (including the Internet, etc.). The cognitive function assessment device 30 is constructed as a server device (such as a cloud server), and the terminal device 70 is constructed as a client device. That is, the cognitive function assessment system 1 is constructed as a client-server system.
[0038] The cognitive function assessment device 30 includes a learning model 410 (see FIG. 2). The learning model 410 after being trained by machine learning is also referred to as a trained model 420. Specifically, the learning parameters of the learning model 410 (learning device) are adjusted using a predetermined machine learning method, and a trained learning model 410 (trained model 420) is generated (see FIG. 2).
[0039] For example, a neural network model composed of multiple layers is used as the learning model 410. Then, weighting coefficients (learning parameters) between multiple layers (input layer, (one or more) intermediate layers, output layer) in the neural network model are adjusted by a predetermined machine learning method. Alternatively, a multiple regression model (regression analysis model) may be used as the learning model 410, and the weighting coefficients of the multiple regression model may be adjusted by using a least squares method related to multiple regression as one machine learning method.
[0040] In this way, the cognitive function assessment device 30 executes the learning stage processing in machine learning. Note that Fig. 4 is a conceptual diagram showing the learning stage processing.
[0041] Specifically, the learning model 410 is a model (regression model) that learns the relationship between information on the facial expression scores of each of the multiple subjects and a predetermined medical cognitive function assessment scale that indicates an evaluation result on the level (degree of decline, etc.) of the cognitive function of each of the multiple subjects. In detail, the learning model 410 is a model that receives information on the facial expression scores (multiple parameters) as input and outputs a value corresponding to a predetermined medical cognitive function assessment scale.
[0042] The predetermined medical cognitive function assessment scale is a medical assessment scale (assessment value indicating the degree of cognitive function) obtained by a predetermined medical test (including the subject's answer to a doctor's question, etc.) for the subject. The predetermined medical cognitive function assessment scale for each of the multiple subjects is obtained by a predetermined medical test for each of the multiple subjects. In the machine learning of the learning model 410, the predetermined medical cognitive function assessment scale for each subject is used as a label (correct answer data). In detail, a combination (dataset) of information on the facial expression score of each subject and the predetermined medical cognitive function assessment scale for each subject is used as teacher data (labeled data) in the machine learning of the learning model 410.
[0043] Here, the MMSE (Mini-Mental State Examination) is exemplified as a predetermined medical cognitive function assessment scale. The full score of the MMSE (also called the MMSE test value) is 30 points, and a lower MMSE test value (score) indicates a decline in cognitive function.
[0044] Moreover, the facial expression score is a value (score) indicating the degree of a specific facial expression, and is a value acquired based on photographed image data 110 (see FIG. 1) of a photograph of a subject. More specifically, the facial expression score is acquired as a value (for example, a value from "0" to "100") that quantifies the degree of a specific facial expression by analyzing the facial image of the subject in the photographed image data 110. Here, a smile is exemplified as a specific facial expression, and a smile score that quantifies the degree of a smile is exemplified as the facial expression score. However, the present invention is not limited to this. For example, the specific facial expression may be surprise, anger, sadness, disgust, etc., and the facial expression score may be a surprise score, an anger score, a sadness score, a disgust score, etc.
[0045] The captured image data 110 has a plurality of images in a time series (for example, a plurality of frame images in video data). An expression score is calculated for each of a plurality of face images extracted from the plurality of images. In this manner, a plurality of expression scores for a certain person are obtained based on a plurality of images of the person. Note that the captured image data 110 is not limited to being configured with a plurality of frame images in video data, and may be configured with a plurality of still images in a time series, etc.
[0046] Based on the plurality of facial expression scores for the person, information 120 on the facial expression scores for the person is generated. As described in detail later, the information 120 includes, for example, information on the average value and / or maximum value of two or more facial expression scores acquired in a specific period of the photographed image data 110. Furthermore, information on the facial expression scores for each of the plurality of people is acquired based on the photographed image data 110 for each of the plurality of people.
[0047] The facial expression score and the like for each person are acquired in response to an instruction to express a specific facial expression. Specifically, the cognitive function assessment device 30 gives an instruction to the person to be photographed (such as a test subject) to express a specific facial expression (here, a smile), and the person to be photographed expresses (creates) the specific facial expression (such as a smile) in response to the expression instruction. For example, in response to an instruction to continue to express a smile for a certain period (15 seconds), the person to be photographed moves the facial muscles of his or her face to try to express (create) a smile. Then, the photographed image data 110 (here, video image data) of the person to be photographed is photographed by the terminal device 70 and transmitted from the terminal device 70 to the cognitive function assessment device 30. The cognitive function assessment device 30 analyzes the photographed image data 110 received from the cognitive function assessment device 30, and acquires the average value, maximum value, and standard deviation of a plurality of facial expression scores in a specific period in the photographed image data 110. The cognitive function assessment device 30 also acquires the duration of a specific facial expression (smile duration), the rise (stand up) time until the specific facial expression, and the like, by performing an analysis process on the captured image data 110.
[0048] Here, when cognitive function declines, it becomes difficult to express a smile (especially a strong smile that is significantly different from a neutral expression). Therefore, it becomes difficult to produce a high smile score (for example, a value as close as possible to the maximum value "100"). As a result, the average and maximum smile scores for a certain period based on a smile expression instruction tend to be lower. In addition, the smile duration tends to be shorter, and the rise time to a smile tends to be longer.
[0049] Based on such findings, in the present application, the degree of cognitive function is determined based on the information 120 related to the facial expression score. Specifically, the learning model 410 is made to learn the relationship between the information 120 related to the facial expression score and the degree of cognitive function. In more detail, the learning model 410 is made to learn the relationship between the information 120 related to the facial expression score and a predetermined medical cognitive function assessment scale, and a trained model 420 is generated.
[0050] More specifically, a learning model 410 is constructed in which information 120 on the facial expression scores of each of the subjects is input, and a value (hereinafter also referred to as the cognitive degree score D) corresponding to a predetermined medical cognitive function assessment scale (such as MMSE) of each of the subjects is output. Then, in the learning model 410, machine learning is performed so as to reduce the error between the output (cognitive degree score D) (estimated value) for each subject and the actual MMSE score (test value obtained by actually performing MMSE) (correct value) of each subject. In other words, the actual MMSE score (test value obtained by actually performing MMSE) is used as correct data to perform machine learning. In this way, in the learning model 410, the relationship between the information 120 on the facial expression scores of each of the subjects and a predetermined medical cognitive function assessment scale (such as MMSE test value) showing the evaluation result of the cognitive function of each of the subjects is machine-learned.
[0051] Furthermore, the cognitive function assessment device 30 executes processing at an inference stage in machine learning. The cognitive function assessment device 30 estimates a recognition degree score D for a person to be assessed (a person different from the multiple subjects used in learning (e.g., a person who has not been tested for MMSE)) based on information 120 on the facial expression score of the person to be assessed, using the trained model 420. Note that FIG. 5 is a conceptual diagram showing processing at the inference stage.
[0052] Specifically, the cognitive function assessment system 1 assesses the cognitive function of a person to be assessed based on a captured image of the person to be assessed for cognitive function. Specifically, the captured image is captured by a camera of a terminal device 70, and is transmitted from the terminal device 70 to a cognitive function assessment device 30 via a network (including the Internet, etc.). The cognitive function assessment device 30 analyzes the captured image and obtains an expression score (e.g., a smile score) that is a value representing the degree of a specific expression (e.g., a smile) for the person in the captured image. Next, information 120 regarding the expression score (in detail, information regarding a plurality of expression scores respectively acquired from a plurality of images in a time series in the captured image) is input to a trained model 420 in the cognitive function assessment device 30. Then, a value (cognitive degree score D) equivalent to a predetermined medical test value (MMSE) is output from the trained model 420. In other words, the cognitive function assessment device 30 uses the trained model 420 to estimate a “cognitive degree score” (a value equivalent to the medical cognitive function assessment scale using the MMSE) for the person to be assessed based on information regarding the facial expression score of the person to be assessed.
[0053] Here, the cognition degree score D is a value obtained by using a trained model 420 that has learned the relationship between information on the facial expression scores of each of the subjects and a predetermined medical cognitive function assessment scale for each of the subjects. The cognition degree score D is a value that has a high correlation with a predetermined medical cognitive function assessment scale (e.g., MMSE test value), and is also expressed as a value equivalent to the predetermined medical cognitive function assessment scale.
[0054] <2. Cognitive function assessment device 30> As shown in FIG. 2, the cognitive function assessment device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.
[0055] The controller 31 is a control device that is built into the cognitive function assessment device 30 and controls the operation of the cognitive function assessment device 30.
[0056] The controller 31 is configured as a computer system including one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 executes, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32, thereby realizing various processes. The program (more specifically, a group of program modules) may be recorded in a portable recording medium such as a USB memory, read from the recording medium, and installed in the cognitive function assessment device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the cognitive function assessment device 30.
[0057] The controller 31 executes processes related to the learning stage of machine learning.
[0058] Specifically, the controller 31 first executes a process of acquiring teacher data. Specifically, the controller 31 acquires information 120 related to the facial expression scores of each of the subjects, and also acquires data indicating a predetermined medical cognitive function assessment scale (MMSE test value) of each of the subjects. That is, a combination of the information 120 related to the facial expression scores of each subject and the medical cognitive function assessment scale of each subject is acquired as teacher data.
[0059] Next, the controller 31 executes a process of optimizing learning parameters related to the learning device (learning model 410) based on the teacher data, and generates a trained model 420. More specifically, machine learning is performed using data indicating a known relationship between the information 120 on the facial expression scores of each of the subjects and a predetermined medical cognitive function assessment scale (MMSE test value) of each of the subjects as teacher data, and the learning parameters are optimized. As a result, the trained model 420 is generated.
[0060] Furthermore, the controller 31 executes processing related to the inference stage of machine learning. Specifically, the inference processing is executed based on the information 120 related to the facial expression score acquired for the person to be determined, using the learning model 410 (trained model 420) in which the learning parameters have been adjusted. In detail, an estimation processing (inference processing) is executed to estimate the recognition degree score D (a value equivalent to the test value of the MMSE). The controller 31 also executes processing to output the inference result (such as display processing of the recognition degree score D).
[0061] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD). The storage unit 32 stores a learning model 410 (including learning parameters and programs related to the learning model) (eventually a trained model 420) and the like. The storage unit 32 also stores a registration database 450 used for authentication processing for the cognitive function assessment device 30. In the registration database 450, various information related to registered users (user ID, name, face image information, etc.) is registered in advance.
[0062] The communication unit 34 is capable of performing network communication via a network. In this network communication, various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) are used. By using the network communication, the cognitive function assessment device 30 can transmit and receive various data to and from a desired counterpart (for example, a terminal device 70).
[0063] The operation unit 35 includes an operation input unit 35a that receives operation input to the cognitive function assessment device 30, and a display unit 35b that displays and outputs various information. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.
[0064] In addition, since this cognitive function assessment device 30 also executes the learning stage processing related to the learning model 410 (generation processing of the learning model 410), it is also called a learning model generation device, etc.
[0065] <3. Terminal device 70> Each terminal device 70 is an information input / output terminal device (information processing device) capable of network communication with the cognitive function assessment device 30. Each terminal device 70 is configured as a smartphone, a tablet terminal, or a personal computer (which may be either a fixed (desktop) type or a portable type), etc. FIG. 1 illustrates the terminal device 70 configured as a tablet terminal.
[0066] The terminal device 70 transmits and receives various types of information to and from the cognitive function assessment device 30. The terminal device 70 generates photographed image data 110 by photographing a person present in the vicinity of the terminal device 70, and transmits the photographed image data 110 to the cognitive function assessment device 30. In addition, the terminal device 70 receives information from the cognitive function assessment device 30 and displays the information on its display unit 75b or the like.
[0067] FIG. 3 is a functional block diagram showing a schematic configuration of the terminal device 70. As shown in FIG.
[0068] As shown in the functional block diagram of FIG. 3, the terminal device 70 includes a controller 71, a memory unit 72, a communication unit 74, an operation unit 75, and an imaging unit 76, and various functions are realized by operating these units in a combined manner.
[0069] The controller (control unit) 71 is a control device that controls the terminal device 70 .
[0070] The controller 71 has the same hardware configuration as the controller 31. The controller 71 executes, in a CPU or the like, a predetermined software program stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 72, thereby realizing various processes. The program (specifically, a group of program modules) may be recorded in a portable recording medium such as a USB memory, read from the recording medium, and installed in the terminal device 70. Alternatively, the program may be downloaded via a communication network or the like and installed in the terminal device 70.
[0071] The controller 71 executes the program and performs the following various processes.
[0072] Specifically, during the learning stage of the learning model 410, the controller 71 performs processes such as photographing facial images, etc. of each subject, and also performs processes to transmit the photographed image data 110 and MMSE test values, etc. of each subject to the cognitive function assessment device 30.
[0073] Moreover, the controller 71 generates captured image data 110 capturing a face image of a person to be assessed for cognitive function in an inference stage using the learning model 410 (more specifically, the trained model 420). The controller 71 also transmits the captured image data 110 of each subject to the cognitive function assessment device 30. Furthermore, the controller 71 acquires an estimation result of MMSE by the cognitive function assessment device 30 using the trained model 420 from the cognitive function assessment device 30, and displays the estimation result of MMSE (a value equivalent to a predetermined medical cognitive function assessment scale) on the display unit 75b.
[0074] The storage unit 72 has the same hardware configuration as the storage unit 32 .
[0075] The communication unit 74 has the same hardware configuration as the communication unit 34. By utilizing network communication by the communication unit 74, the terminal device 70 can transmit and receive various data to and from a desired counterpart (for example, the cognitive function assessment device 30).
[0076] The operation unit 75 includes an operation input unit 75a that receives operation input to the terminal device 70, a display unit 75b that displays and outputs various information, and an audio output unit 75e that outputs audio. A mouse, a keyboard, or the like is used as the operation input unit 75a, and a display (such as a liquid crystal display) is used as the display unit 75b. A touch panel 75c that functions as both a part of the operation input unit 75a and a part of the display unit 75b may also be provided. The audio output unit 75e includes a speaker, or the like.
[0077] The photographing unit 76 is configured with a camera 76c, etc. The photographing unit 76 is capable of generating photographed image data 110 (more specifically, moving image data) by the camera 76c. Specifically, the camera 76c has an RGB image sensor that captures visible light images (color images, etc.), and is capable of photographing color moving images.
[0078] <4. Learning stage processing> Below, the processes in the learning stage of the learning model 410 and the inference stage using the learning model 410 (trained model 420) will be explained in order.
[0079] First, the learning stage processing will be described.
[0080] 6 is a flowchart showing the process in the learning stage. The process shown in FIG. 6 is executed by the controller 31 and the like.
[0081] 6 is also a diagram showing a method for generating a trained model. In this application, generating the trained model 420 means manufacturing (producing) the trained model 420, and the "method for generating a trained model" means the "method for manufacturing a trained model."
[0082] The learning stage processing in FIG. 6 is roughly divided into a preparation stage processing of teacher data (steps S11, S12) and a machine learning processing of the learning model 410 using the teacher data (steps S13, S14).
[0083] First, the process of step S11 will be described. In step S11, the controller 31 acquires photographed image data 110 of each subject (person to be photographed), and acquires information 120 related to the facial expression score of each subject based on the photographed image data 110.
[0084] Specifically, photographed image data 110 of the subject is acquired by the terminal device 70. The subject causes the terminal device 70 to take a photograph including his or her own face at home (or a hospital) or the like.
[0085] More specifically, the subject logs in to the cognitive function assessment device 30 using an application software program (hereinafter, also simply referred to as an application) executed by the terminal device 70. For example, user information (login ID and password) input using the operation unit 75 of the terminal device 70 is transmitted from the terminal device 70 to the cognitive function assessment device 30. Then, the input (transmitted) user information is collated with pre-registered user information in the cognitive function assessment device 30, thereby performing login processing (user authentication processing). Note that, without being limited to this, for example, face recognition processing may be performed as the login processing. Specifically, the subject's current face image (face image (still image) at the time of login) may be photographed by the terminal device 70, and the face recognition processing may be performed by collating the current face image with a pre-registered face image (face image (normal data) of the subject). Then, whether or not to permit login may be determined according to the result of the face recognition processing.
[0086] After logging in, an instruction to express a "smile" is given to the subject by display output and / or audio output from the terminal device 70. The process of giving the expression instruction may be executed by an application executed in the terminal device 70. However, this is not limited to this. The process of giving the expression instruction may be executed by the terminal device 70 according to, for example, an instruction from the cognitive function assessment device 30 (such as a script executed by a web server (cognitive function assessment device 30) or a web browser (terminal device 70)). The same applies to an instruction to stop a specific facial expression (such as a smile) (described later), etc.
[0087] Here, a total of two specific facial expression instructions are given. More specifically, first, a first instruction is given to the subject to continuously express a specific facial expression (here, a smile) in a first period L1 (here, for 15 seconds). Furthermore, after a first pause period R1 (here, for 15 seconds) immediately after the end of the first period L1, a second instruction is given to the subject to continuously express a specific facial expression (smile) in a second period L2 (again, for 15 seconds). Furthermore, when a second pause period R2 (here, for 15 seconds) after the second period L2 ends, the subject is notified that all processing has been completed. Then, an image of the subject is captured during all of these periods (the entire period of the first period L1, the first pause period R1, the second period L2, and the second pause period R2), specifically, during a period including a total of 60 seconds (=15 seconds+15 seconds+15 seconds+15 seconds). Each of the periods L1 and L2 is also called a specific facial expression period (such as a smiling face period).
[0088] Specifically, an overview of the smile training (e.g., smiling for 15 seconds in two sessions with a 15-second break between them) is displayed on a screen (not shown) immediately after logging in. After understanding the overview, the subject presses the smile training start button (displayed on the screen).
[0089] In response to pressing the start button, a screen 210 (211) as shown in FIG. 8 is displayed on the terminal device 70 (more specifically, the display unit 75b). The screen 210 (211) has a display area 221 for a captured image and a display area 223 for a reference face image. The display area 221 displays the captured image of the subject himself, which is updated in real time as needed. Meanwhile, the display area 223 displays a reference smiling face image (also referred to as a reference face image). Here, an "average face image" is displayed as the reference face image. The average face image is a face image obtained in advance by averaging a plurality of face images of a plurality of people having a specific expression (here, a smile). The average face image is obtained, for example, by normalizing the size of a plurality of face images (smiling face images) of a plurality of people and averaging the positions of feature points (more detailed positions of feature points such as eyes, nose, mouth, etc.) in the face images.
[0090] In the upper part of the display area 221 in the screen 211 in FIG. 8, a message is displayed saying, "At the signal of START, we will conduct the first smile training for 15 seconds!! Please smile like the reference face image (average face) on the right." Also, a voice reading out the message is output by the terminal device 70. Also, in the upper part of the display area 221 in the screen 211, the remaining time until the smile expression starts ("30 seconds left" in FIG. 8) is displayed. Note that FIG. 8 to FIG. 11 are diagrams showing display screens related to the first smile expression period L1, etc., and FIG. 12 to FIG. 14 are diagrams showing display screens related to the second smile expression period L2, etc. Also, FIG. 15 is a diagram showing a display screen of an inference result (described later). Among these, FIG. 8 shows a display screen immediately before the first smile expression period L1 (before the start of shooting, etc.). Moreover, FIG. 9 shows a display screen for giving the first smile expression instruction, FIG. 10 shows the display screen during the first smile instruction period L1, and FIG. 11 shows the display screen during the first pause period R1 immediately after the first smile expression period L1.
[0091] Thereafter (after screen 211 in FIG. 8 is displayed), screen 210 (212) as shown in FIG. 9 is displayed on terminal device 70 at an appropriate timing (for example, 5 seconds before the smile training starts). Screen 212 displays "Now, the first smile training will begin in 15 seconds!!", and the remaining time is counted down further. Then, when the start time arrives, an instruction to smile (more specifically, an instruction to start smiling) is given. Specifically, the words "OK, start!" are displayed on screen 212, and the voice "OK, start!" is output.
[0092] In this way, the first expression start instruction P1 (an instruction to start expressing a specific facial expression) (see also FIG. 16) is given. The first expression start instruction P1 is also expressed as a first instruction for continuously expressing a specific facial expression (here, a smile) in the first period L1. The smile expression start instruction P1 is also referred to as a smile instruction P1.
[0093] The terminal device 70 starts photographing the subject (photographing process of the photographed image data 110) several seconds (or several tens of seconds) before the start smiling instruction P1. However, this is not limited to this, and photographing of the subject may be started in response to the start smiling instruction P1. Here, photographing is started 10 seconds before the start smiling instruction P1 (see FIG. 16). FIG. 16 is a diagram showing the change over time of the smile score acquired based on the photographed image data 110 of a certain photographing subject.
[0094] The subject starts expressing (creating) a smile in response to the expression instruction (start smiling instruction P1). At this time, a reference face image (average face image) of a smile is displayed (presented) in the display area 223 of the screen 213 of the display unit 75b, and the subject tries to express (create) a smile close to the reference face image by referring to the reference face image (average face image) displayed on the screen 213.
[0095] In the first period L1, a screen 210 (213) as shown in Fig. 10 is displayed. The remaining time ("3 seconds remaining" in Fig. 10) is counted down and displayed on the screen 213. Note that the first period L1 is also called the smile instruction period (smile instruction section) L1, since it is a period (section) during which a smile is instructed to be displayed.
[0096] When the first period L1 ends, the screen 210 (214) as shown in FIG. 11 is displayed. The screen 214 displays the text "First time, done." At the same time, a voice saying "First time, done" is output. In this manner, the first smile stop instruction Q1 (an instruction to stop (end) the expression of a smile (expression end instruction)) (see FIG. 16) is given.
[0097] Furthermore, on screen 214, the text "Take a 15-second break and then do another round" is further displayed, and the time until the start of the second round (in FIG. 11, "12 seconds left") is counted down.
[0098] After that, at an appropriate timing (for example, 10 seconds before the second smile expression starts), a screen 210 (215) as shown in FIG. 12 is displayed on the terminal device 70. On the screen 215, "The second smile expression training will be performed for 15 seconds at the signal of START!" is displayed, and the remaining time is counted down. After that, the screen 216 of FIG. 13 is displayed, and the remaining time is further counted down. Then, when the start timing arrives, an instruction to express a smile (more specifically, an instruction to start expressing a smile) is given. Specifically, as shown in FIG. 13, the words "OK, START!" are displayed, and the voice "OK, START!" is output. Note that, as described above, FIG. 12 to FIG. 14 show display screens related to the second smile expression period L2 and the like. In detail, FIG. 12 shows a display screen relating to the second smile expression instruction (the display screen immediately before the smile expression instruction), FIG. 13 shows a display screen for giving the second smile expression instruction, etc., and FIG. 14 shows the display screen during the second pause period R2 immediately after the second smile expression period L2.
[0099] In this way, the second start smiling instruction P2 (see FIG. 16) is given. The second start smiling instruction P2 is also expressed as a second instruction for continuing to express a smile in the second period L2 (see FIG. 16).
[0100] The subject starts expressing (creating) a smile in response to the expression instruction (start smiling instruction P2). At this time, the subject, similar to the first period L1, refers to the reference face image (average face image) displayed in the display area 223 and attempts to express (create) a smile close to the reference face image.
[0101] In the second period L2, as in the first period L1, a screen similar to the screen 213 in Fig. 10 is displayed. Note that the second period L2 is also referred to as a smile instruction period (smile instruction period) L2, since it is a period (section) during which a smile is instructed to be displayed.
[0102] When the second period L2 ends, the screen 210 (217) as shown in FIG. 14 is displayed. The screen 217 displays the message "The second time has ended. Please take a 15-second break." At the same time, a voice with the same content is output. In this way, the second smile stop instruction Q2 (an instruction to stop (end) the expression of a smile) (see FIG. 16) is given.
[0103] Then, when 15 seconds have passed since the second smile stop command Q2, the shooting ends. In addition, a screen (not shown) including the words "Thank you for your hard work. All done" is displayed on the display unit 75b, and the subject is notified that shooting (smile training) is finished. In addition, shooting by the terminal device 70 also ends.
[0104] By photographing in this manner, photographed image data 110 including periods L1, R1, L2, and R2 (a total of 60 seconds) is generated. Here, photographed image data 110 is generated over a period of 70 seconds from 10 seconds before the start smiling instruction P1 to the end shooting instruction Q3 (the end of the second pause period R2) (see FIG. 16). That is, photographed image data 110 is generated over a period including the 60 seconds of periods L1, R1, L2, and R2 and the period (10 seconds) immediately before the start smiling instruction P1.
[0105] Furthermore, the terminal device 70 transmits the generated photographed image data 110 to the cognitive function assessment device 30. The cognitive function assessment device 30 stores the photographed image data 110 received from the terminal device 70 in the storage unit 32 (FIG. 2).
[0106] In this manner, the cognitive function assessment device 30 (the controller 31, etc.) acquires photographed image data 110 of each subject (person to be photographed).
[0107] Note that FIG. 15 (described later) is not displayed at the learning stage of the learning model 410, but is displayed at the inference stage (described later, see FIG. 7) using the learning model 410 (trained model 420).
[0108] Next, the controller 31 of the cognitive function assessment device 30 executes an analysis process on the captured image data 110, and extracts information 120 related to the facial expression score. Specifically, first, a calculation process of the facial expression score is performed on all frame images in a time series contained in the captured image data 110 (for a total of 60 seconds), and the facial expression score is calculated for each frame image (see FIG. 16). Note that FIG. 16 is a diagram showing the change over time in the facial expression score for a certain subject. FIG. 16 shows the change over time in the facial expression score during the period from the start of the first period L1 (the time when the start smiling instruction P1 was given) to the time when the end shooting instruction Q3 was given.
[0109] (1) Information regarding facial expression scores (hereinafter also referred to as facial expression score information) 120 includes, for example, information regarding the average value and / or maximum value of two or more facial expression scores acquired in a specific period of the captured image data 110.
[0110] In detail, the facial expression score information 120 includes, for example, information on the average and / or maximum value of two or more facial expression scores (such as multiple smile scores for multiple face images acquired at different times within the first period L1) acquired during a first period L1 (see FIG. 16) of the captured image data 110. Specifically, the facial expression score information 120 includes information on the average and / or maximum value (the maximum value among the 450 smile scores) of 450 smile scores for 450 face images acquired during 15 seconds of the first period L1 at 30 frames / second.
[0111] However, without being limited to this, the facial expression score information 120 may include, for example, information on the average and / or maximum value of two or more facial expression scores (such as multiple smile scores for multiple facial images acquired at different times within the second period L2) acquired during the second period L2 of the captured image data 110.
[0112] The facial expression score information 120 may also include information on the average and / or maximum value of two or more facial expression scores acquired in both the first period L1 and the second period L2 (also referred to as the entire designated period) of the photographed image data 110. In other words, the facial expression score information 120 may include information on the average (and / or maximum) value of multiple smile scores, etc., for multiple face images acquired at different times within the entire designated period. Specifically, the facial expression score information 120 may include information on the average and / or maximum value (the maximum value of the 900 smile scores) of 900 smile scores for 900 face images acquired at 30 frames / second during a total of 30 seconds, consisting of 15 seconds in the first period L1 and 15 seconds in the second period L2.
[0113] The combined period of the first period L1 and the second period L2 (both the first smile instruction period L1 and the second instruction period L2) is the entire period during which a smile is instructed, and is therefore also referred to as the entire instruction period (the entire instruction section or the entire smile instruction period, etc.). The specific period of the photographed image data 110 is not limited to the periods L1 and L2, but may be periods M1 and M2 (also referred to as smile periods or smile sections) during which a smile (e.g., a smile with a predetermined score or higher) is actually expressed (see FIG. 16). For example, the facial expression score information 120 may include information on the average and / or maximum value of two or more facial expression scores acquired during the period M1 (and / or M2) of the photographed image data 110.
[0114] When cognitive function declines, it becomes difficult to intentionally express (produce) a specific facial expression (such as a smile). For example, it becomes difficult to create a "good" smile (to express a smile with a large smile score) and to continue to express a "good" smile. Therefore, for example, the average and / or maximum value of the facial expression score in a specific period of the captured image data 110 correlates with the degree of cognitive function.
[0115] In addition, as cognitive function declines, it becomes difficult to intentionally express (produce) a specific facial expression (such as a smile) twice in a row (in both the first period L1 and the second period L2 (and in both periods M1 and M2)). Therefore, it is preferable to take into account not only the average and maximum facial expression scores in the first specific facial expression period (such as period L1) but also in the second specific facial expression period (such as period L2).
[0116] (2) The facial expression score information 120 may also include information on the standard deviation of two or more facial expression scores acquired from the captured image data 110 during a specific period.
[0117] In detail, the facial expression score information 120 may include information on the standard deviation of two or more facial expression scores acquired in the first period L1 and the first rest period R1 of the captured image data 110. Specifically, the facial expression score information 120 may include information on the standard deviation of 900 smile scores for 900 face images acquired at 30 frames / second in a total of 30 seconds, which is the 15 seconds of the first period L1 and the 15 seconds of the first rest period R1.
[0118] Furthermore, the facial expression score information 120 may include information on the standard deviation of two or more facial expression scores acquired in the second period L2 and the second pause period R2 of the captured image data 110.
[0119] Furthermore, the facial expression score information 120 may include information on the standard deviation of two or more facial expression scores acquired in the first period L1, the first rest period R1, the second period L2, and the second rest period R2 of the captured image data 110. Specifically, the facial expression score information 120 may include information on the standard deviation of 1800 smile scores for 1800 face images acquired at 30 frames / second during a total of 60 seconds for the periods L1, R1, L2, and R2.
[0120] The combined period (L1+R1) of the first period L1 and the first pause period R1 is also called the (first) designated period (designated section) because it is the period during which it is specified whether or not to express a smile. Similarly, the combined period (L2+R2) of the second period L2 and the second pause period R2 is also called the (second) designated period (designated section) because it is the period during which it is specified whether or not to express a smile. Moreover, the combined period of the first period L1, the first pause period R1, the second period L2, and the second pause period R2 is also called the entire designated period (entire designated section) because it is the entire period during which it is specified whether or not to express a smile.
[0121] As described above, when cognitive function declines, it becomes difficult to intentionally express (produce) a specific facial expression (such as a smile). For example, the standard deviation of the facial expression score in a specific period (a designated period including both a period during which a specific facial expression is expressed and a period during which the specific facial expression is not expressed) of the photographed image data 110 also correlates with the degree of cognitive function. Here, the standard deviation of the facial expression score in the total period (first designated period) of the first period L1 and the first rest period R1 can be said to be an index representing the difference between the facial expression score in the first period L1 (period during which a smile is expressed) and the facial expression score in the first rest period R1 (period during which a smile is not expressed). The standard deviation correlates with the degree of cognitive function.
[0122] In addition, as cognitive function declines, it becomes difficult to intentionally express (produce) a specific facial expression (such as a smile) twice in a row (in both the first period L1 and the second period L2). Therefore, it is preferable to take into account not only the standard deviation of the facial expression score in the first designated period (such as the total period of periods L1 and R1) but also in the second designated period (such as the total period of periods L2 and R2).
[0123] (3) Furthermore, the facial expression score information 120 may include information on the duration of a particular facial expression (such as a smile) in the captured image data 110.
[0124] The duration is calculated as the duration during which an expression score equal to or greater than a predetermined score Th1 (see FIG. 16) is actually sustained in response to an instruction to express a specific expression, for example.
[0125] In detail, the time (e.g., 14 seconds) during which an expression score (smile score) equal to or greater than a predetermined score Th1 (e.g., "35") actually continues during the first period L1 of the photographed image data 110 is calculated as the duration of the specific expression (smile) for the first period L1. The duration of the specific expression (smile) for the first period L1 is also expressed as the duration of the first smile period M1 (see FIG. 16). Similarly, the time (e.g., 9 seconds) during which an expression score (smile score) equal to or greater than a predetermined score (e.g., "35") actually continues during the second period L2 of the photographed image data 110 may be calculated as the duration of the specific expression (smile) for the second period L2. The duration is also expressed as the duration of the second smile period M2. The smile periods M1 and M2 are also each expressed as the period during which the smile (specific expression) continues (smile duration (specific expression duration)).
[0126] As described above, when cognitive function declines, it becomes difficult to intentionally produce a specific facial expression (such as a smile). For example, it becomes difficult to continue smiling for a period L1 of 15 seconds, and the smile may disappear due to exhaustion in the middle of the period L1 of 15 seconds. Therefore, the duration of the smile correlates with the level (degree of decline) of cognitive function. In addition, it is preferable to take into consideration not only the duration of the first specific facial expression (such as the duration of the smile) but also the duration of the second specific facial expression.
[0127] (4) Furthermore, facial expression score information 120 may include information on the time length B (the period required to rise) from the time when an instruction to start expressing a specific facial expression is given to the time when the facial expression score reaches its maximum value.
[0128] In detail, the first period L1 of the captured image data 110 may include information on the time B1 (e.g., 2.5 seconds) (see FIG. 16) from the time of the start smiling command P1 to the time when the facial expression score (smile score, etc.) reaches a maximum value (e.g., "62") Similarly, the second period L2 of the captured image data 110 may include information on the time B2 (e.g., 3 seconds) from the time of the start smiling command P2 to the time when the facial expression score (smile score, etc.) reaches a maximum value (e.g., "60").
[0129] As described above, when cognitive function declines, it becomes difficult to intentionally express (produce) a specific facial expression (such as a smile). In particular, it becomes difficult to quickly express a specific facial expression, and it takes a relatively long time to express the facial expression. Therefore, the rise time for expressing a specific facial expression correlates with the level of cognitive function. In addition, it is preferable to take into consideration not only the rise time to the first specific facial expression, but also the rise time to the second specific facial expression.
[0130] Here, it is preferable that the facial expression score information 120 includes a plurality of indexes among the above-mentioned various indexes (average value, maximum value, standard deviation, duration, rise time, etc. of the facial expression score). In other words, it is preferable that a plurality of indexes (input parameters) having a correlation with a predetermined medical cognitive function assessment scale are used for learning the learning model 410.
[0131] Furthermore, it is preferable that the facial expression score information 120 includes information on a plurality of facial expression scores acquired at a plurality of timings within a period of a certain length, rather than a single facial expression score (smile score) based on one still image. It is also preferable that each period length (L1, L2, etc.) for acquiring the facial expression score is longer than a certain length. Specifically, it is not necessarily sufficient for each period length to be, for example, less than about 1 second, but it is preferable that each period length is long enough to make it difficult for a person with dementia (or a person whose cognitive function is declining) to continue smiling (for example, about 10 seconds or more).
[0132] In this manner, facial expression score information 120 for one subject is acquired. The acquisition process of the facial expression score information 120 is executed for each of the multiple subjects. As a result, the facial expression score information 120 for each of the multiple subjects is acquired (step S11). The captured image data 110 of each subject may be acquired by the terminal device 70 of each subject. However, this is not limited to this, and for example, the captured image data 110 of each subject may be acquired using a terminal device 70 (such as a terminal device 70 carried by a doctor) installed in the place where the MMSE is performed (such as a hospital).
[0133] In the next step S12, a process of acquiring a predetermined medical cognitive function assessment scale (here, MMSE) for each of the multiple subjects is executed. Here, it is assumed that a predetermined medical test including a doctor's interview has been performed in advance for each of the multiple subjects, and a predetermined medical cognitive function assessment scale corresponding to the predetermined medical test has been obtained. It is assumed that a predetermined medical cognitive function assessment scale (MMSE test value) for each of the multiple subjects is stored (registered) in advance in the storage unit 32. The medical cognitive function assessment scale for each subject may be transmitted from a terminal device 70 of a doctor or the like to the cognitive function assessment device 30 as data having a predetermined data format, and may be (automatically) registered in the storage unit 32 in the cognitive function assessment device 30. Alternatively, the medical cognitive function assessment scale for each subject may be sent from a doctor or the like to an operator by e-mail or the like, and may be (manually) registered in the storage unit 32 by the operator.
[0134] In step S12, the controller 31 acquires medical cognitive function assessment scales for a plurality of subjects that are registered in advance in the storage unit 32.
[0135] In the next step S13, the controller 31 learns (by machine learning) the relationship between the information 120 regarding the facial expression scores of each of the multiple subjects and a predetermined medical cognitive function assessment scale indicating the evaluation results of the cognitive function of each of the multiple subjects.
[0136] Specifically, machine learning is performed on a learning model 410 that inputs facial expression score information 120 of each of a plurality of subjects and outputs a cognitive degree score D (a value equivalent to a predetermined medical cognitive function assessment scale (such as MMSE)) of each of the plurality of subjects. In detail, machine learning is performed in the learning model 410 so as to reduce an error between the output (cognitive degree score D) of each subject and the actual MMSE score (test value obtained by actually performing MMSE) (correct value) of each subject. Note that, here, an ID (identifier) is assigned to each subject, and each subject is identified by the assigned ID. A combination of facial expression score information 120 and MMSE test value for a person with the same ID is used as one teacher data (labeled data).
[0137] In this way, the relationship between the information 120 regarding the facial expression scores of each of the multiple subjects and the specified medical cognitive function assessment scales of each of the multiple subjects is machine-learned to generate a trained learning model 410 (trained model 420) (step S14).
[0138] According to this, as described below, it is possible to estimate a value equivalent to a predetermined medical cognitive function assessment scale of a person based on information 120 on the facial expression score of the person to be assessed (e.g., a person different from the multiple subjects) using the trained model 420. Furthermore, it is possible to obtain a value equivalent to a medical cognitive function assessment scale based on a medical test without actually conducting a medical test on the person to be assessed for cognitive function.
[0139] <5. Inference stage processing> Next, the inference stage process (MMSE test value estimation stage) using the trained model 420 will be described.
[0140] 7 is a flowchart showing the process of the inference stage. The process shown in FIG.
[0141] Fig. 7 is a flowchart showing the processing of the inference stage. The processing shown in Fig. 7 is executed by the controller 31 and the like. Through the processing of the inference stage, an MMSE test value (more specifically, a value equivalent to the MMSE test value (specifically, the cognitive degree score D)) of the person to be judged is estimated based on the facial expression score information 120 of the person to be judged. That is, a value equivalent to a predetermined medical cognitive function assessment scale is estimated.
[0142] The subject of the assessment is, for example, a person other than the subject whose data was used in the learning stage. However, the assessment is not limited to this, and may be a person (subject) whose data of a specific medical test was used in the learning stage, but whose level of cognitive function at a certain time after the specific medical test is unknown. For the person, inference processing based on the trained model 420 may be used to estimate a specific medical cognitive function assessment scale at that time.
[0143] First, in step S31, facial expression score information 120 of the person to be determined is acquired. In step S31, the same process as that in step S11 is performed for the person to be determined (see also Figs. 8 to 14). Specifically, photographed image data 110 relating to the person to be determined (person to be photographed) is photographed by the terminal device 70 of the person to be determined, and facial expression score information 120 relating to the person to be determined is acquired based on the photographed image data 110. As the facial expression score information 120, an index (multiple indexes) of the same type as the index acquired in step S11 is acquired.
[0144] In the next step S32, when facial expression score information 120 (values of the same type of index as in the learning stage) regarding the person to be determined is input to the trained model 420, an output from the trained model 420, i.e., an estimation result by the trained model 420, is obtained. Specifically, a recognition degree score D (also referred to as an MMSE equivalent value) is output as an estimation result by the trained model 420.
[0145] In step S33, the inference result is displayed. For example, after the screen 217 in FIG. 14, the screen 210 (218) in FIG. 15 is displayed. In the screen 218, for example, an MMSE equivalent value (awareness score D) is displayed, such as "Your awareness score (MMSE equivalent score) is 29." In addition, the controller 31 stores the awareness score D related to the output result in the memory unit 32 as information on the person to be determined.
[0146] <6. Experimental example (an example of learning results)> Next, an example of the learning result of the learning model 410 based on a small number of samples (sample data of 20 people) and an evaluation of the learning model 410 after the learning (trained model 420) will be described.
[0147] However, here, a relationship (also expressed as a type of learning model 410) between a dependent variable and multiple explanatory variables is obtained using a statistical method (multiple regression analysis) based on a very small amount of teacher data (a total of data from 20 people). Obtaining the relationship between multiple explanatory variables (multiple variables in facial expression score information 120) and a dependent variable (MMSE equivalent value) using a statistical method (multiple regression analysis) is substantially equivalent to obtaining a trained model 420. Also, here, an output value for a new input is obtained based on the relationship between the dependent variable (output) and multiple explanatory variables (input) obtained by the statistical method, and the output value is obtained as an estimation result by the trained model 420.
[0148] Specifically, five indexes (variables) shown in the lower part of FIG. 19 are used as the multiple explanatory variables among the multiple indexes (variables) in the facial expression score information 120. In detail, Information on the duration of a specific facial expression for the first period L1 (also referred to as the "first smile duration (specific facial expression duration)"); Information on the average value of two or more facial expression scores (here, smile scores) acquired in the first period L1 (also referred to as the “average value in the first instruction period L1”), Information on the average value of two or more facial expression scores acquired during the second period L2 (also referred to as the "average value during the second instruction period L2"); Information on the standard deviation of two or more facial expression scores acquired in the first period L1 and the first rest period R1 (also referred to as the "standard deviation for the first designated period (instruction period L1 and rest period R1)"), Information on the standard deviation of two or more facial expression scores obtained in the first period L1, the first resting period R1, the second period L2, and the second resting period R2 (also referred to as the "standard deviation for all specified periods"); is adopted. Each piece of information is normalized as a value from 0 to 10 before use. For example, the average facial expression score for each period, "0" to "100", is normalized to correspond to "0" to "10". In addition, the duration of the facial expression score for each period (smile duration) is normalized to correspond to "0" to "15" (seconds).
[0149] Here, the duration of the specific facial expression for the first period L1 is calculated based on the second derivative curve W2 of the facial expression score time-varying curve W0 (FIG. 20) (described later). However, the present invention is not limited to this, and the duration during which two or more facial expression scores having a magnitude equal to or greater than a predetermined value are continuously obtained during the first period L1 may be used as the duration of the specific facial expression for the first period L1. Here, the above five indices are used as an example, but the present invention is not limited to this. For example, instead of the "first smile duration (specific facial expression duration)" and the "standard deviation for the first designated period (instruction period L1 and stop period R1)" among the above five indices, Information on the duration of the specific facial expression for the second period L2 (also referred to as the “second smile duration (specific facial expression duration)”), Information on the standard deviation of two or more facial expression scores acquired during the second period L2 and the second rest period R2 (also referred to as the "standard deviation for the second designated period (instruction period L2 and rest period R2)"); may be used (together with the remaining three indices). The duration of the specific facial expression for the second period L2 may be a duration calculated based on the second derivative curve W2 or the like (described later), or may be a duration during which two or more facial expression scores having a magnitude equal to or greater than a predetermined value are continuously obtained during the second period L2.
[0150] In addition, the MMSE test score is used as the objective variable. The MMSE test score is scaled (affine transformed) so that an MMSE test score of "19" is set to "0" and the maximum MMSE test value of "30" is set to "10." Here, the scale is transformed so that a score of 19 is set to 0, based on the fact that there are no individuals who score less than 19. As mentioned above, the maximum score for the MMSE test score is 30, and a lower MMSE test score (score) indicates a decline in cognitive function.
[0151] An output value E is obtained by inputting the values of each variable related to the person to be determined into a function (multivariate function) obtained as a result of such multiple regression analysis. The output value E is equivalent to the output value E of the trained model 420. The output value E is then inversely scaled (converted to return the scale to the original value) to obtain the recognition degree score D. Specifically, the recognition degree score D is calculated based on the formula D=E×(30−19) / 10+19. The recognition degree score D is calculated as a value expressed on the same scale as the MMSE test value.
[0152] However, without being limited thereto, the cognitive degree score D may be output as a value in a state in which a certain scale conversion has been performed on a predetermined medical cognitive function assessment scale (for example, as the value E). Even if the scale conversion has been performed on the value, it has a high correlation with the predetermined medical cognitive function assessment scale, and can be easily converted into the predetermined medical cognitive function assessment scale by a predetermined scale conversion (inverse scale conversion). In this way, the value corresponding to the predetermined medical cognitive function assessment scale may be a scale conversion value.
[0153] Here, the MMSE is actually performed on 20 subjects, and the test results of the MMSE are obtained in advance. In addition, facial expression score information 120 is also obtained for the 20 subjects.
[0154] In order to evaluate the learning results based on the learning model 410, the 20 subjects U1 to U20 are divided into two groups (first group and second group) of 10 subjects each.
[0155] Then, as shown in the upper part of Fig. 17, a learning model 410 is trained based on data (teacher data) on the 10 subjects U1 to U10 of the first group, and a first trained model 420 (result of multiple regression analysis) is obtained. Next, the 10 subjects (U11 to U20) of the second group are each set as a person to be judged, and the first trained model 420 is used to estimate the recognition degree score D (MMSE equivalent value) of the person to be judged based on the facial expression score information 120 of the person to be judged. In Fig. 18, a plurality of points (shown as black circles) are plotted showing the correspondence between the MMSE equivalent value (recognition degree score D) estimated for the persons U11 to U20 using the first trained model 420 and the actual MMSE test value for the persons U11 to U20.
[0156] Also, the process shown in the lower part of FIG. 17 (but this time the groups are swapped) is executed. Specifically, a learning model 410 is learned based on data (teacher data) on the 10 subjects U11 to U20 of the second group, and a second trained model 420 (multiple regression analysis result) is obtained. Next, the 10 subjects (U1 to U10) of the first group are each set as a person to be judged, and the second trained model 420 is used to estimate the recognition degree score D (MMSE equivalent value) of the person to be judged based on the facial expression score information 120 of the person to be judged. In FIG. 18, a plurality of points (shown as black squares) are plotted showing the correspondence between the MMSE equivalent value (recognition degree score D) estimated for the persons U1 to U10 using the second trained model 420 and the actual MMSE test value for the persons U1 to U10.
[0157] The graph in Fig. 18 is a diagram showing the relationship between the actual MMSE test value and the MMSE equivalent value (awareness degree score D) estimated using the trained model 420. The horizontal axis of the graph in Fig. 18 indicates the actual MMSE test value, and the vertical axis of the graph in Fig. 18 indicates the MMSE equivalent value (awareness degree score D) estimated using the trained model 420. In Fig. 18, data showing the relationship between the estimation result (awareness degree score D) based on the first trained model 420 and the actual MMSE is indicated by black circles. Meanwhile, data showing the relationship between the estimation result (awareness degree score D) based on the second trained model 420 and the actual MMSE is indicated by black squares.
[0158] When the correlation coefficient r was calculated using a total of 20 pieces of data, 10 pieces of data shown by black circles (some of which overlap in the figure) and 10 pieces of data shown by black squares (some of which overlap in the figure), the correlation coefficient r was calculated to be "0.72". The correlation coefficient (also called the correlation value) r is a value indicating the degree of correlation between the MMSE equivalent value (cognitive degree score D) estimated using the first or second trained model 420 and the actual MMSE test value. The correlation coefficient r is calculated based on the following formula (1).
[0159]
number
[0160] Here, x with subscript (i or j) is the actual MMSE test score (diagnosis result) of each person, and x with a bar above is the average value (of 10 people) of the actual MMSE test score (diagnosis result). Also, y with subscript (i or k) is the recognition degree score D of each person, and y with a bar above is the average value (of 10 people) of the recognition degree score D. Also, n is the number of data.
[0161] The larger the absolute value of the correlation coefficient r between two variables (the closer it is to 1), the stronger the correlation between the two variables. In general, when the correlation coefficient between two variables is greater than 0.4, it can be considered that there is a certain degree of correlation between the two variables (a somewhat strong correlation). Also, when the correlation coefficient is greater than 0.7, it can be considered that there is a high correlation (strong correlation) between the two variables. Here, the value of the correlation coefficient r between the two variables, the cognitive degree score D and the MMSE test value, "0.72", is a value greater than 0.7. In other words, it is shown that the two variables (the cognitive degree score D and the MMSE test value) have a high correlation (strong correlation).
[0162] In this way, the cognitive degree score D is a value that has a high correlation with the MMSE test value (a medical cognitive function assessment scale based on a medical test) to a certain degree or more. Therefore, the cognitive degree score D is a value that can be evaluated as being equivalent to the MMSE test value (although it cannot be said to be the same value as the MMSE test value). In other words, according to machine learning (regression) using the above-mentioned five input parameters and one output parameter, it is possible to satisfactorily generate (manufacture) a trained model 420 that can output a value equivalent to the MMSE test value. In addition, by using the trained model 420, it is possible to obtain a value equivalent to a predetermined medical cognitive function assessment scale with high accuracy.
[0163] Here, a statistical method (multiple regression analysis) is used based on a very small amount of data (20 items) to determine the relationship between multiple explanatory variables (multiple variables in the facial expression score information 120) and the objective variable (MMSE equivalent value). As described above, determining the relationship is substantially equivalent to determining the trained model 420. Furthermore, by learning the learning model 410 using a larger amount of data (teacher data) to obtain the trained model 420, it is expected that an even higher correlation can be obtained between the output result (recognition degree score D) of the trained model 420 and the MMSE test value.
[0164] FIG. 19 is a diagram showing the correlation coefficient r (also referred to as r') between each of a plurality (here, 13) of variables (a plurality of input parameters) and the MMSE test value. The 13 variables include a total of 10 variables, including the average value of each smile period (M1, M2) (first and second times, respectively), the maximum value of each smile period (M1, M2), the average value of each instruction period (L1, L2), the standard deviation of each designated period ((L1+R1), (L2+R2)), and each smile duration (M1 period length, M2 period length). The 13 variables also include a total of three variables, including the maximum value of the entire instruction period (L1+L2), the average value of the entire instruction period (L1+L2), and the standard deviation of the entire designated period (L1+R1+L2+R2).
[0165] The correlation coefficient r' is calculated by using the value of a certain variable (input parameter) of each person instead of the recognition degree score D of each person as y with a subscript (i or k) on the right side of the above formula (1). For example, when the first smile duration of each of 10 people is used as y with a subscript (i or k) on the right side of the above formula (1), the correlation coefficient r' between the variable "first smile duration" and the MMSE test value is calculated. Note that the larger the absolute value of the correlation coefficient r' (the closer to 1), the stronger the correlation. In addition, when the value of the correlation coefficient r' for two variables is positive, there is a positive correlation between the two variables, and conversely, when the value of the correlation coefficient r' is negative, there is a negative correlation between the two variables.
[0166] For example, the correlation coefficient between the first smile duration and the MMSE test value is 0.74 (upper right column of the table in FIG. 19). Similarly, the correlation coefficient between the average value in the first instruction period L1 and the MMSE test value is 0.49, and the correlation coefficient between the average value in the second instruction period L2 and the MMSE test value is 0.48. Also, the correlation coefficient between the standard deviation for the entire designated period and the MMSE test value is 0.67. Each of the other values in the table in FIG. 19 indicates the correlation coefficient between each of the other indexes (parameters) and the MMSE test value.
[0167] In this experimental example, taking into consideration the correlation coefficient between each of these variables and the MMSE test value, the above-mentioned five variables (input parameters) (see the black bold box in the upper part of FIG. 19) are used among many types of variables (parameters) in the learning model 410, etc. Note that this is not limited to this, and other variables (input parameters), such as the maximum value of the facial expression score in the entire instruction period (L1+L2), may be used. In addition, any one or more of the 13 variables described in the upper part of FIG. 19 (a single variable or a combination of multiple variables) may be used as information regarding the facial expression score.
[0168] <7. Effects of the embodiment> According to the above embodiment, a learning model 410 (trained model 420) is generated by machine learning the relationship between information 120 about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale for each of the plurality of subjects.
[0169] Then, using the learning model 410 (more specifically, the trained model 420), a value (awareness score D) equivalent to a predetermined medical cognitive function assessment scale (here, MMSE) of the person to be determined is estimated based on the information 120 related to the facial expression score of the person to be determined. The awareness score D is a value based on the output from the trained model 420 trained using the MMSE test value, and is a value that has a high correlation with the MMSE test value. Therefore, the awareness score D can be evaluated as being equivalent to the MMSE test value.
[0170] Therefore, it is possible to obtain a value equivalent to a predetermined medical assessment scale of cognitive function based on a medical test without actually conducting a medical test on the person whose cognitive function is to be assessed. Since it is not necessary to conduct a medical test involving a medical interview by a doctor, it is possible to obtain a value equivalent to a predetermined medical assessment scale of cognitive function relatively easily.
[0171] In addition, the actual test time for obtaining a value equivalent to a predetermined medical cognitive function assessment scale is a relatively short time (about 1 to 2 minutes) for taking a relatively short image of the person to be evaluated. In other words, the actual test time is shorter than the time required for a medical test (for example, 30 minutes).
[0172] Furthermore, photographed image data 110 of the person to be assessed is generated by the terminal device 70, and the photographed image data 110 is transmitted from the terminal device 70 to the cognitive function assessment device 30 via a network. Then, the cognitive function assessment device 30 assesses the degree of cognitive function. In detail, a value equivalent to a predetermined medical cognitive function assessment scale (a value equivalent to the MMSE) is estimated. Therefore, the person to be assessed can know a value equivalent to a predetermined medical cognitive function assessment scale (a value equivalent to the MMSE) while remaining at home or the like (without the need to go to a hospital or the like).
[0173] In the above embodiment, the cognitive function assessment device 30 issues an instruction to the subject to be photographed to express a specific facial expression, and the subject expresses the specific facial expression in response to the instruction. In particular, in consideration of the characteristic that it becomes difficult to intentionally express (create) a specific facial expression (such as a smile) when cognitive function declines, the score related to the specific facial expression expressed in response to the instruction is used as the facial expression score of the specific facial expression. By constructing the learning model 410 using such a score, it is possible to obtain a recognition degree score D that has a high correlation with the MMSE test value.
[0174] In the above embodiment, in both the learning stage and the inference stage, a reference face image of a specific facial expression (such as a smile) is displayed on the screen 210 (see FIG. 10, etc.) when an instruction to express a specific facial expression is given. Therefore, the subject can easily understand the facial expression that he or she should express (what kind of facial expression he or she should express).
[0175] <8. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.
[0176] <8-1. Duration and onset time of specific facial expressions, etc.> For example, in the above-described embodiment, the duration during which a facial expression score equal to or greater than a predetermined score Th1 (see FIG. 16) is continued is exemplified as the duration of a specific facial expression (such as a smiling face duration), but this is not limiting.
[0177] Specifically, the duration (length) etc. may be determined based on the maximum point etc. of the second derivative curve W2 (see FIG. 20) of the time-varying curve W0 of the facial expression score extracted from the captured image data 110.
[0178] In addition, the top of FIG. 20 shows a time-dependent change curve W0 of the facial expression score (smile score, etc.) for the first instruction period L1, the middle shows a first derivative curve W1 of the time-dependent change curve W0 of the facial expression score, and the bottom shows a second derivative curve W2 of the time-dependent change curve W0 of the facial expression score. The time-dependent change curve W0 is a curve formed by connecting (linking) a plurality of continuous time-series facial expression scores (a plurality of facial expression scores after noise removal) extracted from the captured image data 110. The time-dependent change curve W0 is obtained by smoothing (by a smoothing filter, etc.) the connecting line of the continuous plurality of facial expression scores. In the following, the facial expression scores for the first instruction period L1 will be mainly illustrated, but the same applies to the facial expression scores for the second instruction period L2, etc.
[0179] In detail, as shown in Fig. 20, time C1 is determined to be the beginning (starting point) of the duration (sustaining period) of the specific facial expression. Time C1 is the time when the second derivative curve W2 reaches a maximum value equal to or greater than a predetermined value TH11 near the time when the command P1 to start expressing the specific facial expression (e.g., smiling) is given. Time C9 is determined to be the end (ending point) of the duration (sustaining period) of the specific facial expression. Time C9 is the time when the second derivative curve W2 again reaches a maximum value equal to or greater than the predetermined value TH11 after time C1. The period (and its length) from time C1 to time C9 is then calculated as the duration (and its duration) of the specific facial expression (e.g., smiling).
[0180] The second derivative curve W2 reaches a maximum value equal to or greater than the threshold value TH11 at time C1, and immediately thereafter reaches a minimum value equal to or less than the threshold value TH12 (time C3). Thereafter, during a period during which the specific facial expression is stably expressed (stable expression period), the extreme values (minimum and maximum values) of the second derivative curve W2 are greater than the threshold value TH12 and less than the threshold value TH11. Between two times C3 and C7 (described below), the second derivative curve W2 has maximum and minimum values whose magnitudes (absolute values) are less than a predetermined value. Then, when the specific facial expression (smile) ends (in response to the expression end instruction Q1 or before the expression end instruction Q1), the second derivative curve W2 reaches a minimum value smaller than the threshold value TH12 (time C7), and immediately thereafter reaches a maximum value larger than the threshold value TH11 (time C9). In consideration of such fluctuations of the second derivative curve W2, the length of the period from time C1 to time C9 may be regarded as the duration of the specific facial expression (smile, etc.), as described above.
[0181] According to this embodiment, the period during which the value of the second derivative curve W2 falls within a predetermined range (between thresholds TH11 and TH12) is determined to be the stable duration of the specific facial expression (the period during which the specific facial expression is stably continued). Also, the time point C1 corresponding to the maximum value immediately before the stable duration is determined to be the start of the duration of the specific facial expression, and the time point C9 corresponding to the maximum value immediately after the stable duration is determined to be the end of the duration of the specific facial expression. Therefore, it is possible to detect the duration of the specific facial expression without depending on the magnitude of each person's facial expression score itself (without depending on the magnitude relationship between each person's facial expression score and threshold Th1, etc.).
[0182] In addition, in the above-described embodiments, the rise time of a specific facial expression is exemplified as the length of time (period length) from the time when the start of expression is instructed to the time when the facial expression score reaches its maximum value, but this is not limited to this.
[0183] 20, the length of the period from time C1 to time C3 is calculated as the rise time of the specific facial expression (such as a smile). Here, time C3 is the time when the second derivative curve W2 first reaches a minimum value (particularly a minimum value equal to or less than a predetermined value TH12) after time C1.
[0184] In this manner, the rise time of a particular facial expression may be determined.
[0185] Furthermore, the expression of a specific facial expression is originally started after the expression start instruction P1 (in response to the expression start instruction P1). As shown in FIG. 20, the expression start instruction P1 is originally a time point prior to time point C1. However, the expression of a specific facial expression may start slightly earlier than the expression start instruction P1. For example, the person being photographed (subject, etc.) may anticipate (sense) the start timing and attempt to start the specific facial expression slightly earlier than the original start time. In such a case, as shown by P1 (also written as P1') in parentheses in FIG. 20, the position of the expression start instruction P1 on the time axis may be later (to the right) than time point C1.
[0186] In consideration of such a situation, a time point (C1) before the expression start instruction P1 (P1') may be determined to be the start time of the "duration of the specific facial expression" and the "rise time of the specific facial expression". Specifically, the time point at which the second derivative curve W2 reaches a maximum value equal to or greater than a predetermined value within a period (a period from a few seconds before the expression start instruction P1' to a few seconds after the expression start instruction P1') in the vicinity of the time point at which the expression start instruction P1' is given may be determined to be the time point C1. If the "duration of the specific facial expression" calculated in this manner exceeds a theoretical maximum value (such as the value of the first period L1 (15 seconds)), the value exceeding the maximum value may be changed to the maximum value. In other words, an upper limit may be set for the "duration of the specific facial expression". For example, if the duration of the specific facial expression is calculated to be "18 seconds", "18 seconds" may be changed to "15 seconds".
[0187] <8-2.Other> In addition, in the above embodiment, the cognitive function assessment device 30 executes both the learning process and the inference process, but the present invention is not limited to this, and the learning process and the inference process may be realized by separate devices (30A, 30B). In this case, the device 30B that executes the inference process may obtain, for example, information (learned learning parameters, etc.) related to the trained model 420 generated by the device 30A that executed the learning process from the device 30A, and use the trained model 420.
[0188] In the above-mentioned embodiment, the MMSE (Mini-Mental State Examination) is exemplified as the predetermined medical cognitive function evaluation scale, but the present invention is not limited thereto. As the predetermined medical cognitive function evaluation scale, other evaluation scales such as the UPDRS (Unified Prkinson's Disease Rating Scale) may be used. For example, an evaluation value (evaluation scale) related to cognitive function in the UPDRS may be used. Specifically, a five-level evaluation value from 0 to 4 points (indicating a decline in cognitive function as the score increases) indicating the degree of cognitive function in the UPDRS may be used as the predetermined medical cognitive function evaluation scale.
[0189] In the above embodiment, the instruction to make an expression such as a smile and an instruction to pause are given using both the display output and the audio output from the terminal device 70, but this is not limiting. For example, the instruction to make an expression such as a smile and an instruction to pause may be given using only one of the display output and the audio output from the terminal device 70.
[0190] In the above embodiment, the subject is photographed by the terminal device 70, but the present invention is not limited to this. For example, the cognitive function assessment device 30 may photograph the subject and generate the photographed image data 110. In this case, the screen 210 (including the display area 221 of the photographed image and the display area 223 of the reference face image, etc.) may be displayed on the display unit 35b of the cognitive function assessment device 30. In other words, the smile command, the photographed image, the reference face image, etc. may be displayed on the display unit 35b of the cognitive function assessment device 30. Similarly, a sound (such as a smile command) may be output from a sound output unit (not shown) of the cognitive function assessment device 30. [Explanation of symbols]
[0191] 1. Cognitive function assessment system 30 Cognitive function assessment device 70 Terminal Equipment 75c Touch Panel 76c Camera 110 Photographed image data 120 Facial Expression Score Information 410 Learning Model 420 trained models 450 registration database D. Awareness score E Output Value L1, L2 smile instruction period P1, P2 Smile start instruction (expression start instruction) Q1, Q2 Smile stop instruction Q3 End shooting instruction R1, R2 break period r correlation coefficient
Claims
1. a control unit that estimates a value corresponding to the predetermined medical cognitive function assessment scale of a person based on information about the facial expression scores of the person to be assessed with respect to cognitive function, using a learning model that has learned a relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects; Equipped with The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A cognitive function assessment device comprising:
2. The specific facial expression is a smile, The cognitive function assessment device according to claim 1 , wherein the facial expression score is a value indicating a degree of smiling.
3. The photographed image data includes a plurality of images in a time series, The cognitive function assessment device according to claim 1 or 2, wherein the information on the facial expression scores includes information on a plurality of facial expression scores respectively obtained from the plurality of images.
4. The cognitive function assessment device according to claim 3 , wherein the information regarding the facial expression score includes information regarding a maximum value of two or more facial expression scores acquired in a specific period of the photographed image data.
5. The cognitive function assessment device according to claim 3 or 4, wherein the information on the facial expression score includes information on an average value of two or more facial expression scores acquired in a specific period of the photographed image data.
6. The cognitive function assessment device according to claim 3 , wherein the information regarding the facial expression score includes information regarding a standard deviation of two or more facial expression scores obtained during a specific period of the photographed image data.
7. A cognitive function assessment device as described in any of claims 3 to 6, characterized in that the duration of each of the specific facial expressions is calculated as the time length from a first point in time at which a second derivative curve of the time-course curve of the facial expression score extracted from the captured image data reaches a maximum value equal to or greater than a predetermined value within a period adjacent to the point in time at which each instruction for expressing the specific facial expression is given, to a second point in time at which the second derivative curve again reaches a maximum value equal to or greater than a predetermined value after the first point in time.
8. The information regarding the facial expression score includes information regarding a rise time of the specific facial expression, 8. The cognitive function assessment device according to claim 3, wherein the rise time is calculated as a time length from a first time point at which a second derivative curve of a time-course curve of the facial expression score extracted from the captured image data reaches a maximum value equal to or greater than a predetermined value within a period near a time point at which an instruction to start expressing the specific facial expression is given, to a second time point at which the second derivative curve first reaches a minimum value after the first time point.
9. The information regarding the facial expression score is Information on an average value of two or more facial expression scores acquired during the first period; and Information on the average value of two or more facial expression scores acquired during the second period; and Information on an average value of two or more facial expression scores acquired in the first period and the second period; and Information on the standard deviation of two or more facial expression scores acquired during the first period and the first pause period; and Information on the standard deviation of two or more facial expression scores acquired during the second period and a second pause period following the second period; and Information on the standard deviation of two or more facial expression scores acquired in the first period, the first rest period, the second period, and the second rest period; Information on a maximum value of two or more facial expression scores acquired in the first period and the second period; and Information on an average value of two or more facial expression scores acquired during a first duration, which is a period during which the specific facial expression continues from a time point at which the first instruction is given or close to the time point at which the first instruction is given; and information on an average value of two or more facial expression scores acquired during a second duration, which is a period during which the specific facial expression continues from a time point at which the second instruction is given or close to the time point at which the second instruction is given; and Information on a maximum value of two or more facial expression scores acquired during the first duration; and Information on a maximum value of two or more facial expression scores acquired during the second duration; and The cognitive function assessment device according to claim 3 , further comprising at least one of the following:
10. The information regarding the facial expression score is Information on an average value of two or more facial expression scores acquired during the first period; and Information on the average value of two or more facial expression scores acquired during the second period; and Information on the standard deviation of two or more facial expression scores acquired during the first period and the first pause period; and Information on the standard deviation of two or more facial expression scores acquired in the first period, the first rest period, the second period, and a second rest period after the second period; The cognitive function assessment device according to claim 3 , further comprising:
11. The control unit, upon receiving an instruction to express the specific facial expression, causes a reference face image relating to the specific facial expression to be displayed on a predetermined display unit, The cognitive function assessment device according to claim 1 , wherein the reference face image is an average face image obtained by averaging a plurality of face images of a plurality of persons having the specific facial expression.
12. A cognitive function assessment device according to any one of claims 1 to 11, a terminal device that transmits the photographed image data of the person to the cognitive function assessment device; A cognitive function assessment system comprising:
13. a control unit that performs machine learning on a relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an assessment result of the cognitive function of each of the plurality of subjects to generate a learning model; Equipped with The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A learning model generating device comprising:
14. A step of estimating a value corresponding to the predetermined medical cognitive function assessment scale of a person to be assessed for cognitive function based on information about the facial expression scores of the person, using a learning model that has learned a relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects; Equipped with The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A method for assessing cognitive function, comprising:
15. A step of performing machine learning on a relationship between information on the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an assessment result of the cognitive function of each of the plurality of subjects to generate a learning model; Equipped with The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A learning model producing method comprising:
16. A step of estimating a value corresponding to the predetermined medical cognitive function assessment scale of a person to be assessed for cognitive function based on information about the facial expression scores of the person, using a learning model that has learned a relationship between information about the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an evaluation result of the cognitive function of each of the plurality of subjects; A program for causing a computer to execute the above, The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A program comprising:
17. A step of performing machine learning on a relationship between information on the facial expression scores of each of a plurality of subjects and a predetermined medical cognitive function assessment scale indicating an assessment result of the cognitive function of each of the plurality of subjects to generate a learning model; A program for causing a computer to execute the above, The facial expression score is a value indicating a degree of a specific facial expression and is obtained based on photographed image data of a subject, The predetermined medical cognitive function assessment scale for each of the plurality of subjects is a value obtained by a predetermined medical test for each of the plurality of subjects; the captured image data is generated by giving a first instruction to the subject to continuously display the specific facial expression during a first period out of two periods including a first period and a second period including a rest period therebetween, and giving a second instruction to the subject to continuously display the specific facial expression during a second period following a first rest period which is a rest period following the first period; The information regarding the facial expression score includes: information about the duration of the particular facial expression with respect to the first time period; information on the duration of the particular facial expression with respect to the second time period; and A program comprising:
Citation Information
Patent Citations
Dementia care supporting system, dementia care supporting server and dementia care support program
JP2016071897A
Health management assist device and health management assist method
JP2016147006A
Emotion estimation system and emotion estimation method
JP2016149063A
Expression recognition diagnosis support device
JP2018007792A
Diagnosis support information providing device and diagnosis support information providing method
JP2019055064A
