Estimation method, estimation program, and estimation device

Abstract

To provide a novel facial expression-based estimation technique that can be used for assessing frailty. [Solution] An estimation method comprising: an image acquisition step of acquiring facial image data including changes in facial expression of a subject; and an estimation step of calculating estimated values ​​of multiple frailty indicators relating to a subject from the facial image data of the subject, based on an estimation model for estimating frailty indicators from facial image data including changes in facial expression.

Description

Technical Field

[0004] , , , , , , ,

[0006] , , , ,

[0005] , , , , , , ,

[0001] The present invention relates to a method for estimating frailty indices, an estimation program, and an estimation device.

Background Art

[0002] In recent years, technologies for monitoring the health status of the elderly and maintaining and improving the quality of life have been emphasized. In particular, early detection of frailty is important in the health management of the elderly, and technologies for evaluating frailty indices are in demand. Conventionally, direct medical evaluations such as measuring walking speed, muscle strength, or performing cognitive function tests have been required for estimating frailty indices. However, these require specialized examination equipment and the involvement of medical staff, and establishing a method that can be easily implemented has been an issue.

[0003] Under such circumstances, in recent years, non-contact health status estimation technologies that utilize image processing and artificial intelligence technologies have been under development. Patent Document 1 discloses a technology related to the evaluation of cognitive function. This technology learns the relationship between the facial expression score of a subject and a cognitive function evaluation scale (such as the MMSE test value) using a learning model, and by using this model, estimates the scale value of cognitive function without performing an actual medical examination.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, while a multi-faceted analysis is required for the evaluation of frailty, the technology described in Patent Document 1 can only perform the evaluation of cognitive function.

[0006] In view of the above circumstances, the object of the present invention is to provide a novel estimation technique based on facial expressions that can be used for evaluating frailty. [Means for solving the problem]

[0007] [1] An estimation method comprising: an image acquisition step of acquiring facial image data including changes in facial expression of a subject; and an estimation step of calculating estimated values ​​of a plurality of frailty indicators relating to the subject from the facial image data of the subject based on an estimation model for estimating frailty indicators from facial image data including changes in facial expression.

[0008] This configuration makes it possible to comprehensively evaluate the subject's condition using multiple frailty indicators based on images that include changes in facial expressions.

[0009] [2] The estimation method described in [1], wherein the frailty index includes two or more of the following: a physical index relating to physical condition, a cognitive index relating to cognitive function, a psychological index relating to psychological state, a social index relating to social state, and a functional index relating to daily living function.

[0010] [3] The estimation method according to [2], wherein in the estimation step, estimated values ​​of the physical indicator, the cognitive indicator, the psychological indicator, the social indicator, and the life function indicator are calculated based on the estimation model.

[0011] This configuration allows for a more detailed assessment of the frailty state.

[0012] [4] The estimation method according to [2] or [3], wherein the physical indicators include any of the following: an indicator relating to physical function, an indicator relating to body composition, and an indicator based on a medical interview regarding physical condition.

[0013] [5] The cognitive index is an estimation method described in any of [2] to [4], which includes an index relating to cognitive function.

[0014] [6] The estimation method described in any of [2] to [5], wherein the social indicators include any of the following: an indicator relating to cohabitants, an indicator relating to social interaction, an indicator relating to going out, an indicator relating to hobby activities, and an indicator relating to feelings of loneliness.

[0015] [7] The estimation model is generated based on facial image data including changes in the subject's facial expression and the measured value of the frailty index at the time the facial image data of the subject was acquired, according to any one of the estimation methods in [1] to [6].

[0016] This configuration allows for the estimation of frailty indicators based on the relationship between facial image data and frailty indicators, which is statistically derived from the subjects' data.

[0017] [8] The estimation method according to [7] generates the estimation model using as training data an index relating to the amount of movement of facial muscles obtained based on the facial image data and the measured value of the frailty index when the facial image data of the subject was obtained.

[0018] [9] The estimation method according to any one of [1] to [8], wherein the estimation model is generated using facial image data obtained by giving a subject a task that causes a specific facial expression to be expressed, and the image acquisition step includes giving the subject the task and acquiring facial image data.

[0019]

[10] The estimation method according to [9], wherein the estimation model is generated using as training data an evaluation value based on a comparison of facial expressions recognized from the subject's facial image data and the specific facial expression, and the measured value of the frailty index when the subject's facial image data was acquired.

[0020]

[11] The estimation method according to [9] or

[10] , wherein the task is given by instructing the subjects and subjects to make a specific facial expression.

[0021]

[12] The instruction prompts the subject and the target person to present and imitate a specific expression. The image acquisition step includes instructing the target person to present and imitate a specific expression and acquiring face image data, which is the estimation method described in

[11] .

[0022]

[13] The task is given by instructing the subject and the target person to imagine a virtual situation and express an expression while feeling the emotion, which is the estimation method described in [9] or

[10] .

[0023]

[14] The estimation model is a model that estimates frailty indices from information related to the movement of the face. The estimation step includes acquiring information related to the movement of the face in the expression change from the face image data and estimating a plurality of frailty indices based on the information related to the movement of the face, which is the estimation method described in any one of [1] to

[13] .

[0024]

[15] The estimation step includes extracting, as the information related to the movement of the face, a feature quantity related to the movement of feature points of the face in the expression change from the face image data, which is the estimation method described in

[14] .

[0025]

[16] The feature quantity includes at least one of the following (A) to (F), which is the estimation method described in

[15] : (A) The amount of movement of the feature points included in the face; (B) The ratio of the amounts of movement of a plurality of feature points included in the face; (C) The directionality of the movement of the feature points included in the face; (D) The movement speed of the feature points included in the face or its maximum value; (E) The response time to the instruction in the movement of the feature points included in the face; (F) The integral value of the movement speed in the response time.

[0026]

[17] The feature quantity includes (E) the response time to the instruction in the movement of the feature points included in the face, or (F) the integral value of the movement speed in the response time, and the response time includes at least one of the following (a) to (c): The estimation method according to

[16] : (a) The reaction time indicating the time from the instruction until the movement speed of the feature points becomes maximum. (b) The expression adjustment time from when the movement speed of the feature points becomes maximum until the movement speed of the feature points becomes sufficiently small. (c) The sum of the reaction time and the expression adjustment time.

[0027]

[18] The feature quantity includes a quantity based on the comparison of the left and right of the face regarding the movement of the feature points. The estimation method according to any one of

[15] to

[17] .

[0028]

[19] The feature points include one or more points indicating parts of the face. The estimation method according to any one of

[15] to

[18] .

[0029]

[20] An image acquisition step of acquiring face image data including the expression change of a subject, and an estimation step of calculating an estimated value of a plurality of frailty indices regarding the subject from the face image data of the subject based on an estimation model for estimating a frailty index from the face image data including the expression change, which is executed by a computer. An estimation program.

[0030]

[21] An estimation device including an image acquisition unit that acquires face image data including the expression change of a subject, and an estimation unit that calculates an estimated value of a plurality of frailty indices regarding the subject from the face image data of the subject based on an estimation model for estimating a frailty index from the face image data including the expression change.

[0031]

[22] The psychological index includes any one of an index regarding depression, an index regarding anxiety, and an index regarding loneliness. The estimation method according to any one of [2] to [3].

[0032]

[23] The life function index includes any one of an index regarding the height of life function and an index regarding self - reliance in life. The estimation method according to any one of [2] to [3].

[0033]

[24] The estimation method according to any one of [1] to

[15] ,

[18] ,

[19] ,

[22] , or

[23] , wherein the facial image data includes facial image data when an expression is made and facial image data when no expression is made.

[0034]

[25] The estimation method according to

[24] , which calculates an estimated value of the frailty index by calculating one or more of the difference, the absolute value of the difference, or the product based on the features extracted from the facial expression image data and the features extracted from the expressionless image data.

[0035]

[26] The estimation model includes an auxiliary branch that performs an auxiliary task that takes the expressionless image data as input, The estimation method according to

[24] , wherein one or more layers of the estimation model that perform the main task are learned including the loss function in the auxiliary task.

[0036]

[27] Using the features extracted from the facial expression image data and the features extracted from the expressionless image data, features related to facial expression are combined and generated. The estimation method described in

[24] is characterized in that the estimation model is trained to minimize the dot product of the features extracted from the expressionless image data and the combined generated features.

[0037]

[28] In the estimation step, the estimation model further calculates an estimated value of the frailty index based on the subject data of the subject, The estimation methods described in [1]~

[15] ,

[18] ,

[19] ,

[22] ~

[27] .

[0038]

[29] In the estimation process, the scale and shift are determined based on the background conditions of the input data (one or more such as task type, facial expression type, gender, etc.), The estimation method according to

[27] , wherein the combined facial expression features are adjusted and optimized based on the scale and shift to calculate an estimated value of the frailty index.

[0039]

[30] The estimation model is equipped with an auxiliary classifier that performs the task of identifying a person based on the features obtained from the facial image data, The estimation method according to

[24] , wherein one or more layers performing the main task of the estimation model are updated by inverting and returning the gradient obtained from the auxiliary classifier.

[0040]

[31] The estimation method described in

[30] wherein the auxiliary classifier does not learn features to identify an individual from the features obtained from the expressionless image data.

[0041]

[32] The estimation method described in [1] to

[15] ,

[18] ,

[19] ,

[22] to

[31] is a model that has been trained on data extracted by sampling that increases the probability of extracting a small number of classes. [Effects of the Invention]

[0042] According to the present invention, a novel estimation technique based on facial expressions can be provided that can be used for evaluating frailty. [Brief explanation of the drawing]

[0043] [Figure 1] A flowchart illustrating the outline of this embodiment. [Figure 2] A flowchart illustrating the procedure for generating the estimation model of this embodiment. [Figure 3] A flowchart illustrating the procedure for capturing facial image data according to this embodiment. [Figure 4] An example of a method for indicating facial expressions in this embodiment. [Figure 5] An example of a method for indicating facial expressions in this embodiment. [Figure 6] A flowchart illustrating the procedure for generating the estimation model of this embodiment. [Figure 7] A flowchart relating to the estimation method of this embodiment. [Figure 8] A flowchart illustrating an example of the estimation process in this embodiment. [Figure 9]A diagram showing an example of the estimation model of this embodiment. [Modes for carrying out the invention]

[0044] <Embodiment 1> This invention relates to a method for estimating a frailty index for a subject, using facial image data including changes in the subject's facial expression. The method for estimating a frailty index according to the present invention will be described below with reference to the drawings. The embodiments shown below are examples of the present invention and the present invention is not limited to these embodiments.

[0045] For example, this embodiment describes a method for estimating frailty indicators, but systems, devices, computer programs, etc., that perform such methods can achieve similar effects. Furthermore, the program may be stored on a computer-readable, non-transient recording medium, or it may be provided for download from an external server. For example, by installing the program on a computer, the method according to the present invention can be performed by the computer.

[0046] In the following embodiments, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and information processing of software that can be specifically realized by these hardware resources. In this embodiment, "information" can be represented, for example, by the physical value of a signal value representing voltage or current, the high or low value of a signal value as a set of binary bits composed of 0s or 1s, or by a quantum superposition (so-called qubit), and communication and calculations can be performed on a circuit in a broad sense.

[0047] In a broad sense, a circuit is a set of circuits (Circuitry) that are realized by appropriately combining circuits, processors, and memory. For example, it includes circuits that contain any of the following: CPU (Central Processing Unit), GPU (Graphics Processing Unit), LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0048] <1.Definition> First, the definitions of the main terms used in this embodiment will be explained. In this invention, a face image refers to an image showing the process of facial expression changes in a subject. Here, a face image is not a single image of the face at the moment the expression is expressed, but rather an image that includes the process of this facial expression change, and may include, for example, multiple still or moving images showing the before and after of the change. Face image data is data of such a face image.

[0049] Furthermore, a frailty index refers to an indicator related to frailty (the decline of mind and body due to aging). Generally, frailty is often observed from three perspectives: "physical condition," "cognitive and psychological condition," and "social condition." In this embodiment, the frailty index is defined as three types: a physical index related to physical condition, a cognitive index related to cognitive and psychological condition, and a social index related to social condition. However, the frailty index of the present invention is not limited to these, and any index capable of representing frailty may be used as a frailty index.

[0050] The following describes the frailty index estimation method, frailty index estimation program, and frailty index estimation device according to the present invention, illustrating their specific forms. In the following, "subject" refers to a person who is subject to facial image data capture and frailty index measurement when generating an estimation model, while "target person" refers to a person who is subject to frailty index estimation from facial image data using the generated estimation model.

[0051] <2. Overview of estimation method> Figure 1 is a flowchart showing the general procedure for estimating frailty indicators using facial image data. As shown, an estimation model is first generated in step S11. As will be described in detail later, facial image data including changes in facial expressions and frailty indicators are obtained from multiple subjects, and an estimation model is created based on this information.

[0052] In this embodiment, an estimation model is created that estimates physical indicators, cognitive indicators, and social indicators as frailty indicators. Note that an estimation model may be created for each indicator to be estimated, or a single model may be created to estimate multiple indicators.

[0053] Then, in step S12, the frailty index is actually estimated using the estimation model created in step S11. Facial image data, including changes in facial expressions, is acquired for the subject for whom the frailty index is to be estimated, and the frailty index of the subject is estimated from the facial image data based on the estimation model. The details of the procedures in steps S11 and S12 are described below.

[0054] <3. Estimation Model Generation> Figure 2 is a flowchart showing an example of the procedure for generating an estimation model in this embodiment. Note that the procedure shown in Figure 2 is just an example, and the order may be changed within a range that does not affect the results. For example, the order of steps S21 and S22 may be reversed, and although it is preferable that they be performed as close together as possible, a certain amount of time may pass between them. Specifically, for example, a period of several days to several months may pass between the acquisition of facial image data and the measurement of the frailty index, as long as the elements that may affect the frailty index, such as physical function, way of thinking, social environment, and habits, do not change. Note that in the generation of the estimation model, it is preferable that the order and interval of steps S21 and S22 are unified for multiple subjects, but the order and interval of steps S21 and S22 may differ for each subject as long as it does not affect the frailty index.

[0055] <3-1. Frailty Indicators> First, in step S21, multiple subjects are measured for multiple types of frailty indicators. In this embodiment, the measurement of frailty indicators involves obtaining frailty indicators for subjects, as exemplified below. In this embodiment, various frailty indicators are measured for multiple subjects in the generation of the estimation model. The frailty indicators in this embodiment are classified into three types: physical indicators, cognitive indicators, and social indicators.

[0056] A physical indicator is an indicator relating to the physical state. In this embodiment, the physical indicator is particularly an indicator relating to the state of the body excluding the head, and more more specifically, an indicator relating to the state of the body excluding the head and neck. Preferably, physical indicators include indicators showing physical ability, indicators showing body composition, and indicators based on questionnaires about the body.

[0057] As indicators of physical ability, for example, indicators obtained through physical ability tests such as muscle strength and walking speed can be used. Indicators of body composition can include, for example, skeletal muscle index (SMI), muscle mass index (MMI), muscle quality (Phase Angle), fat mass index (FMI), body fat percentage (BF%), body mass index (BMI), height, weight, and other indices obtained by measuring with a device.

[0058] While any questions and evaluation methods may be used for indicators based on physical questionnaires, it is conceivable that the questionnaire items would include information on physical pain, exercise habits, diet, walking ability (use of a cane, falls, etc.), sleep, driving, excretion, vision, hearing, illnesses currently being treated or previously diagnosed, medication use, etc. Indicators based on the answers to these questions can be obtained.

[0059] Cognitive indicators are measures related to cognitive and psychological states. Cognitive indicators can be further divided into indicators related to cognitive function and indicators related to psychological states.

[0060] As indicators of cognitive function, for example, indicators obtained from existing assessment methods such as the MMSE concerning orientation, memory, attention and calculation, language, and constructional ability can be used. Of course, other indicators of cognitive function can be used as needed, for example, assessments of working memory (work function) tasks or indicators related to NCGG-FAT (memory function, executive function, performance function, and processing ability) may be used.

[0061] As indicators related to psychological state, for example, the results of depression assessments based on responses to questionnaires can be used. Other indicators, such as happiness levels, may also be used. Here, happiness refers to an overall measure of well-being that includes factors such as physical and mental health and social environment, rather than just temporary emotions. This concept of happiness also includes elements such as health achievement, purpose in life, and a sense of fulfillment.

[0062] Social indicators are indicators related to social conditions. These indicators are obtained, for example, through questionnaires given to the individual and those around them. More specifically, indicators based on factors such as whether or not one lives with others, social interaction, feelings of loneliness, habits of going out, and the extent of hobbies can be used as social indicators.

[0063] <3-2. Acquisition of facial image data> Once the measurement of frailty indicators is complete, the process proceeds to step S22, where facial image data is acquired for multiple subjects. Figure 3 is a flowchart showing the procedure for acquiring facial image data in step S22. Facial image data acquisition may be performed by computer. Furthermore, subjects may be asked to practice an expression task before facial image data acquisition. In step S22, multiple facial image data is acquired for each of the multiple subjects by performing the acquisition procedure shown in Figure 3. For facial image acquisition, after starting the acquisition in step S31, the process proceeds to step S32, where subjects are given a task that elicits specific facial expressions.

[0064] Task assignment methods include, for example, instructing participants to make specific facial expressions, or presenting them with a particular scenario and asking them to express the emotions in that scenario through their facial expressions.

[0065] The former method involves giving verbal instructions, such as "Please give me your biggest smile," or showing an example of an expression and instructing the subject to imitate it. In this embodiment, the subject is instructed to express an expression by using an image that serves as an example of an expression.

[0066] Figure 4 shows an example of facial expression instruction. In this embodiment, facial expression instruction is given by displaying the facial expression of a model as an example on a screen placed in front of the subject, as shown in Figure 4(a), and instructing the subject to imitate the model's facial expression. Here, in order to film the process of facial expression change, the subject is instructed to change their facial expression at predetermined intervals, for example, every 3 seconds, as shown in Figure 4(b).

[0067] Specifically, the image on the screen (hereinafter referred to as the instruction image) is changed at predetermined intervals, and the subject is instructed to imitate the changes. In this embodiment, multiple still images, each displaying a different facial expression and switching at predetermined intervals, are used as the instruction images. Alternatively, a moving image showing the model changing facial expressions may be used as the instruction image. Examples of facial expressions that can be used here include joy, anger, sadness, surprise, smile, satisfaction, laughter, happiness, disgust, and a neutral expression. However, the facial expressions are not limited to these, and the estimation model and frailty index can be estimated using any facial expression.

[0068] As for the latter approach, for example, when requesting a disappointed expression, one might instruct the participant to "show the expression you normally make when you feel an emotion in the following situation," and present a scenario such as "You are taking a walk and step on some gum. You feel disappointed." Figure 5 shows an example of a screen displayed to a participant when instructing them to express an emotion through a scenario.

[0069] In this embodiment, first, a screen is placed in front of the subject, displaying instructions to express emotions as they arise, as shown in Figure 5(a). Then, as shown in Figure 5(b), a specific situation is displayed to give the subject the task of expressing emotions. In this way, as an example, facial image data can be obtained by capturing changes in facial expressions that are consciously expressed in response to instructions, rather than expressions that are naturally expressed based on emotions.

[0070] Alternatively, as shown in Figure 5(c), subjects may be instructed to "express your emotions as you feel them while watching the video," and then given a facial expression task by playing a video that evokes a certain emotion. Still images may be used instead of video. In this way, as an example, facial image data may be obtained by capturing changes in facial expressions expressed based on naturally arising emotions. In this case, it is preferable to have the subjects respond after the video has been played, describing their emotions upon watching the video.

[0071] As instructed, the subject changes their facial expression accordingly (Figure 3, step S33). After the change in facial expression is complete, the recording is terminated in step S34. This allows for the acquisition of facial image data that includes the process of the subject's facial expression change as they follow the instructions.

[0072] <3-3. Estimation Model Generation> After measuring frailty indicators and capturing facial image data for a sufficient number of subjects, the process proceeds to step S23 to generate an estimation model. In this invention, an estimation model refers to a model of any structure for estimating frailty indicators from information on facial movements, and includes, for example, regression models and machine learning models. The procedure for generating an estimation model is shown in detail in Figure 6. The estimation model can be created by regression analysis using facial image data and frailty indicator measurements for multiple subjects, or by machine learning.

[0073] Figure 6(a) is a flowchart showing the method for generating an estimation model using regression analysis. In this embodiment, first, in step S41, a set of facial image data and frailty index data for multiple subjects is obtained. Next, in step S42, features are extracted from the facial image data.

[0074] When generating an estimation model using regression analysis, features are obtained from facial image data, and the model is generated by performing regression analysis on these features and the frailty index. For example, if facial image data is obtained by providing a task that elicits a specific facial expression, the facial expression accuracy can be used as a feature.

[0075] Facial expression accuracy is an indicator of the accuracy with which a specified facial expression is expressed. For example, if a task is given to imitate a model facial expression, the facial expression recognition system can recognize the emotion between the model and the facial image, and the accuracy can be evaluated based on the degree of match. Alternatively, as shown in Figure 5(c), if a task is given to express the emotion felt through facial expression, the facial expression accuracy can be evaluated based on the degree of match between the emotion the subject responds to and the emotion recognized from the facial image by the facial expression recognition system.

[0076] Alternatively, an index relating to the amount of movement of facial muscles may be used as a feature. As an index relating to the amount of movement of facial muscles, for example, a video showing the process of facial expression changes can be used as facial image data, and the average amount of movement of facial muscles per second during the process of facial expression change can be used.

[0077] In addition, information about facial movements may be used as features. This information about facial movements can be any index that changes in response to changes in facial expression. For example, it could be a value relating to the entire facial expression, such as the amount or rate of change in a score indicating the degree of expression, or it could be a value relating to the movement of a specific part of the face.

[0078] For example, as information about facial movement, feature quantities relating to the movement of feature points located at arbitrary positions on the face may be used. In this embodiment, information about facial movement may be obtained from facial image data or from sensors installed on the face.

[0079] Feature points are points placed at arbitrary locations on the face, and preferably move in response to changes in facial expression. They may be one or more points indicating parts of the face, such as the cheeks, eyebrows, eyes, mouth, and chin. Feature points may be identified by placing markers at predetermined locations, or they may be identified by image analysis without the use of markers. Alternatively, feature extraction may be performed using the optical flow method, with arbitrary points on the face used as feature points.

[0080] Furthermore, the following can be used as features: the amount of movement of a feature point, the ratio of the amounts of movement of multiple feature points, the direction of movement of a feature point, the speed of movement of a feature point, the maximum value of the speed of movement of a feature point, the response time to the facial expression instruction during the movement of a feature point, and the integral value of the speed of movement of a feature point during the response time. In addition, an index related to the position of the feature point may be normalized, and the difference or ratio of the amount of movement between the feature point in the instruction image and the feature point of the subject may be used as a feature. Here, the amount of movement is the difference between the position of the feature point when there is no facial expression and the position of the feature point when a facial expression is expressed (when the expression of facial expression has stabilized). The speed of movement refers to the amount of movement per unit time. For example, it can be expressed as the difference between the position of the feature point in each frame of the video and the position of the feature point in the previous frame, i.e., the amount of movement in each frame of the face image data.

[0081] Response time refers to the time it takes for the subject or participant to respond to an instruction regarding facial expression. Response time can be calculated from the amount of movement of feature points in facial image data and the timing of task presentation. Alternatively, facial expression scores may be used instead of the amount of feature point movement.

[0082] In this embodiment, the response time is a concept that includes reaction time, which is the time from when an instruction is given until the rate of movement of feature points, the rate of change of facial expression score per unit time, etc., reaches its maximum during the process of facial expression change; facial expression adjustment time, which is the time from when the rate of movement of feature points, the rate of change of facial expression score per unit time, etc., reaches its maximum until the rate of change of feature points, the rate of change of movement of feature points, the rate of change of facial expression score per unit time, etc., becomes sufficiently small; and the sum of reaction time and facial expression adjustment time.

[0083] Reaction time can be defined as the time it takes from the instruction to the change in facial expression, while facial expression adjustment time can be defined as the time it takes from the maximization of the facial expression until its expression stabilizes. Therefore, when the movement speed of feature points is used to determine the response time, the integral value of the movement speed of feature points in the response time represents the cumulative value of the movement of feature points during the various response times described above.

[0084] Furthermore, other features may also be used, such as quantities based on a comparison of the left and right sides of the face regarding the movement of feature points. For example, feature points can be placed at corresponding positions on the left and right sides of the face, and quantities like those exemplified above can be calculated for each of these feature points to determine the left-right difference, left-right ratio, or left-right similarity, which can then be used as features.

[0085] To calculate quantities based on a comparison of the left and right sides of the face, vectors relating to the movement of feature points on the left and right sides can be used. In this embodiment, first, the facial region is divided into multiple areas on both the left and right sides, and for multiple feature points within each region, the difference vector of the position of the feature points is calculated when there is no expression (before expression change) and when an expression is expressed (after expression change). Next, the average of the difference vectors for multiple feature points in the same region is calculated and this is used as the average movement vector for each region. Using the average movement vectors for the corresponding left and right regions, the similarity of the amount of movement on the left and right sides and the similarity of the direction of movement on the left and right sides can be calculated for each region.

[0086] The similarity of the left-right movement amounts can be calculated, for example, using the magnitude of the average movement vectors of the corresponding left and right regions, as shown in Equation 1. This is the logarithm of the reciprocal of the absolute difference between the magnitude of the average movement vector of the left region and the magnitude of the average movement vector of the right region.

number

[0087] Furthermore, the similarity in the direction of left-right movement can be calculated, for example, using the magnitude of the average movement vector of the corresponding left and right regions, as shown in Equation 2. Note that the similarity in the amount of left-right movement and the similarity in the direction of left-right movement may also be calculated using the difference vector of the positions of the corresponding feature points on the left and right sides when expressionless and when facial expression is displayed, instead of the average movement vector.

number

[0088] After calculating the feature quantities exemplified above for each of the facial image data of multiple subjects, in step S43, a regression analysis is performed using the data from multiple subjects to examine the relationship between the feature quantities and the frailty index, with the feature quantities as explanatory variables and the various frailty indices as dependent variables. This generates an estimation model that estimates the frailty index from the feature quantities.

[0089] Any known regression analysis method may be used; for example, principal component multiple regression, principal component polynomial regression, PLS multiple regression, PLS polynomial regression, Lasso regression, Ridge regression, ElasticNet regression, Gaussian kernel support vector regression, random forest regression, etc. are available. It is preferable to evaluate the analysis results using correlation coefficients or correlation tests, etc., and select a method that can more appropriately represent the relationship between features and frailty indicators.

[0090] Next, with reference to Figure 6(b), we will explain the generation of an estimation model using machine learning. When generating an estimation model using machine learning, the model may be trained by using pairs of input information and output information as training data.

[0091] In this embodiment, first, in step S51, a set of facial image data and frailty index data for multiple subjects is obtained, similar to the case of regression analysis. Then, in step S52, the obtained information is provided to the model as training data, and the model is trained to perform the task of estimating the frailty index from facial image data. This makes it possible to generate a trained model as an estimation model that directly estimates the frailty index from facial image data.

[0092] In this embodiment, the facial image data itself is used as input. However, for example, features similar to those extracted in step S42 of Figure 6(a) may be extracted, and one or more features may be used as input for learning. In this case, an estimation model for estimating the frailty index can be generated using the features obtainable from the facial image data. Furthermore, in this embodiment, it is assumed that the facial image data is directly input into the estimation model generated by machine learning, but it is also possible to input facial image data that has undergone arbitrary preprocessing.

[0093] As described above, the present invention makes it possible to generate an estimation model for estimating the frailty index from facial image data using a set of facial image data and frailty index data from multiple subjects.

[0094] <4. Estimation of Frailty Indicators> Next, the procedure for estimating the frailty index using the estimation model created as described above will be explained in detail. Herein, the method for estimating the frailty index described below can also be executed by a computer. For example, the frailty index estimation program can be stored in a server device, and facial image data of a subject can be received from a terminal device connected to the server device via a network such as the Internet, and the estimation method of the present invention can be executed. Any server device that stores the estimation program can function as an estimation device according to the present invention. However, the present invention is not limited to the following embodiments, and each process performed by the server device and terminal device may be executed by a single computer, or multiple computers may cooperate to function as an estimation device according to the present invention.

[0095] As estimation devices, general-purpose computer equipment such as server devices equipped with computing units such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units), main memory such as RAM (Random Access Memory), auxiliary storage devices such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), and flash memory, and various input / output devices including means for connecting to a network can be used.

[0096] Furthermore, any computer device, such as a PC (Personal Computer), equipped with various input / output devices including a camera, processing unit, storage device, and means for connecting to a network, can be used as a terminal device. In addition, smartphones and tablet devices may be used as terminal devices. Dedicated applications for inputting and transmitting various information to and from the server device, and browser applications for accessing dedicated web pages are stored in the storage device, and the processing unit executes various processes, so that any computer device functions as a terminal device of the present invention. For example, when a terminal device captures facial image data of a subject and transmits it to an estimation device, the estimation method according to the present invention is executed in the estimation device.

[0097] The terminal device of this embodiment captures facial image data, including changes in the subject's facial expression, using the same procedure as shown in Figures 3 and 4 above, and transmits it to the estimation device. When using a device in which the camera and display device are located in the same direction as the terminal device, the above-mentioned instruction image may also be presented by the display device of the terminal device. Here, when acquiring facial image data, it is preferable to give the subject the same task that was presented to the subject when creating the estimation model.

[0098] The estimation device comprises an image acquisition unit, an estimation unit, and a storage unit, and is configured to communicate with a terminal device. These functional components do not necessarily need to be housed within a single device; for example, multiple computers may collaborate to perform the functions of each component.

[0099] The image acquisition unit receives and acquires facial image data, including changes in the subject's facial expression, from a terminal device via the network, and then passes it on to the estimation unit.

[0100] The estimation unit calculates an estimated value of the frailty index of the subject from the facial image data acquired by the image acquisition unit, based on an estimation model stored in the memory unit beforehand. Specifically, the estimation unit inputs information based on the facial image data acquired by the image acquisition unit into the estimation model generated by the method described above, and calculates an estimated value of the frailty index. In this embodiment, multiple types of frailty indices are estimated, but different estimation models may be used for each frailty index.

[0101] Information based on facial image data refers to information predetermined in the estimation model stored in the memory unit. For example, if the estimation model stored in the memory unit is one that estimates a frailty index based on features related to the movement of facial feature points obtained from facial image data, then those features are input as information based on facial image data. Alternatively, if the estimation model stored in the memory unit is one that has learned the task of outputting an estimated value of the frailty index using facial image data as input, then the facial image data can be input as is or after appropriate preprocessing.

[0102] <5. Estimation method> Hereinafter, an embodiment of the estimation method according to the present invention will be described with reference to Figure 7. First, in the image acquisition step S61, the image acquisition unit of the estimation device receives face image data, including changes in the subject's facial expression, from the terminal device. Here, when acquiring face image data, it is preferable to give the subject the same task that was presented to the subject when creating the estimation model. The image acquisition unit then passes the face image data to the estimation unit.

[0103] Next, in step S62, the estimation unit estimates various frailty indicators based on the estimation model. Figure 8 is a flowchart showing a detailed example of the estimation process (step S62) in this embodiment. As described above, the procedure of the estimation process differs depending on the estimation model used.

[0104] Figure 8(a) is a flowchart for estimating a frailty index using feature quantities as input. In this case, first, in step S71, the estimation unit calculates feature quantities from the facial image data acquired by the image acquisition unit. For the feature quantities, the same type of information used to generate the estimation model stored in the memory unit is extracted.

[0105] Then, in step S72, the estimation unit inputs the feature quantities calculated from the facial image data into the estimation model, thereby obtaining an estimated value of the frailty index output by the estimation model. This makes it possible to estimate the frailty index of a subject from facial image data.

[0106] On the other hand, Figure 8(b) is a flowchart for estimating the frailty index using image data as input. In this case, the estimation unit inputs the image data acquired by the image acquisition unit into the estimation model in step S81. If preprocessing of the image data is performed when generating the estimation model, the same preprocessing is also performed before inputting to the model during estimation in step S81. The estimation model then outputs an estimated value of the frailty index, making it possible to estimate the frailty index of the subject from the facial image data.

[0107] As described above, according to this embodiment, by instructing the subject to make facial expressions and capturing the changes in those expressions, the frailty index can be estimated from facial image data. This eliminates the need for the subject to answer questionnaires or other forms of questionnaires to evaluate the frailty index, and also allows the evaluator to estimate the frailty index simply by inputting the facial image data.

[0108] In particular, in this embodiment, the frailty status of the subject can be evaluated from multiple perspectives by estimating physical indicators, cognitive indicators, and social indicators, respectively.

[0109] <Examples> The following describes an example conducted by the inventors of the present invention to investigate the relationship between feature quantities obtained from facial image data including facial expression changes and a frailty index. While the present invention is based on the following experimental results, the feature quantities, frailty index, and other values ​​used in the present invention are not limited to the following forms.

[0110] [1] Subject The study involved approximately 1,750 Japanese men and women aged 70 and over as subjects.

[0111] [2] Measurement of frailty indicators As physical indicators, walking speed and grip strength were measured. As indicators of cognitive function (cognitive indicators), we measured cognitive function (MMSE score) and MCI (Mild Cognitive Impairment) assessment (assessment based on MMSE score). As an indicator of psychological state, a depression score (GDS-15 score) was measured. As a social indicator, a loneliness score (UCLA score) was measured.

[0112] [3] Acquisition of facial image data [2] Facial image data was taken for each subject no later than one month after the frailty index measurement. A display was placed in front of the subject's face, and the subject was instructed to express facial expressions by presenting tasks on the display, and the process of the subject's facial expression changes was captured as video. The content of the tasks is as described in <3-2. Acquisition of facial image data> of the above embodiment. Specifically, the subject was given a task to imitate a model of facial expression (facial expression instruction image), a task to express a specific scenario and emotion through facial expression, and a task to express emotions as they arise from a video, and facial image data including changes in facial expression was acquired.

[0113] [4] Feature extraction In this embodiment, the facial expression accuracy, which indicates the degree of matching between the estimated emotion of the facial image data including the model (facial expression training image) and facial expression changes by the facial expression recognition system, and the amount of facial muscle movement were used as features in a task in which a model (facial expression training image) is presented. The facial analysis API provided by Amazon® Recognition® was used as the facial expression recognition system (reference URL: https: / / docs.aws.amazon.com / ja_jp / rekognition / latest / dg / faces.html).

[0114] [5] Analysis Based on the facial image data, including changes in facial expressions, obtained as described above, and the frailty index, an analysis was conducted using information from multiple subjects, and the following relationships were observed.

[0115] When comparing facial expression accuracy between a group with a fast walking speed and a group with a slow walking speed, the fast walking speed group showed significantly higher facial expression accuracy compared to the slow walking speed group. Furthermore, the group with a faster walking speed tended to exhibit greater facial muscle movement compared to the group with a slower walking speed.

[0116] A tendency was observed where higher grip strength was associated with higher facial expression accuracy. Furthermore, there was a tendency for the range of motion of facial muscles to be greater with higher grip strength.

[0117] When facial expression accuracy was compared between a group with high cognitive function and a group with low cognitive function, the group with high cognitive function showed significantly higher facial expression accuracy compared to the group with low cognitive function.

[0118] A comparison of facial expression accuracy between a group with normal cognitive function and a group with mild cognitive impairment (MCI) showed that the group with normal cognitive function tended to have higher facial expression accuracy than the MCI group. Furthermore, a comparison of facial expression accuracy between the normal group and the group suspected of having dementia revealed that the normal group had significantly higher facial expression accuracy than the group suspected of having dementia. Furthermore, a comparison of facial expression accuracy between the MCI group and the group suspected of having dementia showed that the MCI group tended to have higher facial expression accuracy than the group suspected of having dementia.

[0119] A comparison of facial muscle movement between a group with normal cognitive function and a group with mild cognitive impairment (MCI) revealed that the normal cognitive function group tended to exhibit greater facial muscle movement than the MCI group. Furthermore, a comparison of facial muscle movement between the group with no issues in the MCI assessment and the group suspected of having dementia revealed that the group with no issues tended to have greater facial muscle movement than the group suspected of having dementia. Furthermore, a comparison of facial muscle movement between the MCI group and the group suspected of having dementia revealed that the MCI group tended to exhibit greater facial muscle movement than the group suspected of having dementia.

[0120] When comparing facial expression accuracy between groups with high and low levels of loneliness based on their loneliness scores, the low-loneliness group tended to have higher facial expression accuracy than the high-loneliness group. Furthermore, the group with low levels of loneliness tended to have less facial muscle movement compared to the group with high levels of loneliness.

[0121] As described above, it was suggested that there is a relationship between physical indicators, cognitive indicators, and social indicators and features derived from facial image data.

[0122] <6. Details of the estimation model using machine learning> The following describes the configuration when the estimation model is composed of a pre-trained multi-layered neural network using machine learning.

[0123] In this embodiment, maximum pooling is used as the pooling process in the neural network, but other known pooling methods, such as average pooling, may also be used.

[0124] Furthermore, in the layers constituting the estimation model in this embodiment, activation functions such as ReLU (Rectified Linear Unit) are applied to stabilize learning. In addition, methods such as regularization may be used to stabilize learning and suppress overfitting.

[0125] In this embodiment, continuous variables, ordered labels, and classification labels are used as training data for the output data (frailty index).

[0126] Furthermore, in this embodiment, in addition to the frailty index, evaluation metrics for evaluating the estimation model are output from the estimation model. In this embodiment, the evaluation metrics output include QWK (Quadratic Weighted Kappa) for evaluating the classification output and MAE (Mean Absolute Error) and RMSE (Root Mean Square Error) for evaluating the regression output, but the output metrics are not limited to these, as long as they are metrics that can evaluate the model. These evaluation metrics may be used during model training, for example, by adjusting the parameters of the layers constituting the neural network to optimize these evaluation metrics.

[0127] Furthermore, during the training of the estimation model, there is a possibility that information that becomes noise when identifying frailty, such as an individual's facial features rather than facial movements, may be extracted. Therefore, the estimation model in this embodiment includes an auxiliary classifier capable of identifying who the target person is based on features acquired from facial image data, and the estimation model is trained using gradient inversion based on the gradient obtained from the auxiliary classifier. Gradient inversion is a method of updating the model by inverting and propagating the gradient obtained from the output layer (fully connected layer in this embodiment) during backpropagation. Note that the estimation model only needs to include an auxiliary classifier at least during the training stage. The auxiliary classifier is installed alongside one or more layers that perform the main task (calculation of an estimated value of the frailty index) in the estimation model, and is a classifier that performs the auxiliary task of estimating a person based on features extracted from facial image data by the estimation model. In training the estimation model using gradient inversion, the sign of the gradient (error) obtained from this auxiliary classifier is inverted and returned, and one or more layers (main task execution layers) that perform the main task (calculation of an estimated value of the frailty index) in the estimation model are updated. By adopting this configuration, instead of updating and learning the model in the direction of identifying individuals as in a typical backpropagation method, it is possible to construct an estimation model that learns in the direction of not identifying individuals (by not learning features that identify individuals), and outputs features that are more important for extracting frailty rather than features for identifying individuals.

[0128] Furthermore, the auxiliary classifier calculates similarity based on the features extracted from the facial image data in the person identification task, determining which person's features the facial image data features are most similar to. In this embodiment, a temperature parameter is used in the similarity calculation by dividing the similarity score. By adopting this configuration, it is possible to soften overly sharp separations that depend on individual facial features and shape the embedding into one that is less sensitive to individual identification. In this embodiment, by combining gradient inversion and temperature-based similarity calculation, it is possible to generate an estimation model that cancels out components related to individual identification while retaining components that are effective for changes in facial expression. Note that a small temperature parameter results in sharper identification, so it is preferable that the temperature parameter be a relatively high value.

[0129] In the medical and aging fields, the weighting of severe and mild cases tends to be skewed, but if the majority of cases are heavily weighted, the likelihood of misjudging minority cases (such as severe cases) increases. Therefore, in this embodiment, a sampler such as a Weighted Sampler is used to extract data from a set of data so that minority class data is extracted with a high probability and majority class data is extracted with a low probability. The extracted data is then used as training data to train an estimation model. At this time, the sampler extracts data with a probability corresponding to each data class in the set of data prepared for use as training data. Note that the data class may be one that has been previously labeled on the data, or it may be one that has been estimated based on the features contained in the data. Furthermore, the sampler that extracts data to be used as training data may identify data at class boundaries (for example, data of patients on the border between severe and mild cases) based on data about the target subjects, such as frailty indices, and increase the frequency of data extraction at these class boundaries.

[0130] Furthermore, when using heavy data such as high-resolution image data, computational load and memory may be strained, and if the cost of training and inference is high, it may not be possible to implement it in the field. In this embodiment, AMP (Automatic Mixed Precision) is adopted in the training of the estimation model, and some calculations such as gradients in the training process are calculated using mixed precision such as 16-bit. AMP is a technique that increases the training speed in deep learning and reduces memory usage. It is a technique in which the format (16-bit or 32-bit) to be used for each calculation is decided for each calculation, such as using half-precision floating-point format (16-bit format) for some of the model calculations and single-precision floating-point format (32-bit format) for the numbers used in the remaining calculations, and calculations are performed in the corresponding format. Note that existing software packages such as modules in PyTorch may be used for training the estimation model using AMP.

[0131] <Embodiment 2> In the embodiments described above, the frailty index relating to cognitive and psychological state was observed as one perspective and defined as a cognitive index relating to cognitive and psychological state. However, cognitive function and psychological state may be observed as separate perspectives. In this case, the frailty index includes a cognitive index relating to cognitive function and a psychological index relating to psychological state. In this embodiment, estimated values ​​are calculated for two or more indicators from among physical indicators relating to physical state, cognitive function relating to cognitive function, psychological indicators relating to psychological state, social indicators relating to social state, and life function indicators relating to life function. In this embodiment, frailty is evaluated using indicators relating to cognitive function (cognitive indicators), indicators relating to psychological state (psychological indicators), various physical indicators, and social indicators, as well as life function indicators. Frailty indicators related to cognition are estimated for overall cognitive function, memory, attention, executive function, processing speed, and working memory. Frailty indicators related to physical function are calculated for physical function (walking speed), muscle strength (grip strength), and skeletal muscle mass (skeletal muscle index). Frailty indicators related to psychology are calculated for depression, anxiety, and loneliness. Furthermore, frailty indicators related to social frailty and daily living function are calculated. As a daily living function indicator, frailty indicators related to the level of function and independence in daily life are calculated.

[0132] In this embodiment, the time required to answer the TMT (Trail Making Test) and the number of correct answers within a predetermined time in the SDST (Symbol Digit Substitution Test) are used as indicators of frailty related to cognition. However, any indicator obtained from any test that measures cognitive function may be used as an indicator of frailty related to cognition. In addition, in this embodiment, the Geriatric Depression Scale (GDS), HADS-A (Hospital Anxiety and Depression Scale Anxiety subscale), and UCLA LS are used as indicators of frailty related to psychology. Furthermore, in this embodiment, Instrumental Activities of Daily Living (IADL) is used as an indicator of frailty related to daily living function. While specific examples of frailty indicators in this embodiment are described here, the frailty indicators are not limited to the examples described above.

[0133] <Embodiment 3> <Configuration of the estimation model> Next, we will describe the detailed configuration of the estimation model constructed using machine learning (in this embodiment, particularly deep learning) in the process described above. While Embodiment 1 described an example of an estimation model constructed using regression models and machine learning models, this embodiment will describe an example in which a neural network model, such as a CNN (Convolutional Neural Network), which is particularly strong in image processing, is adopted as the estimation model. The estimation model in this embodiment processes data acquired step by step through the layers that make up the neural network and generates an output. In the layers that make up the estimation model, one or more of the following processes are performed on the input data: convolution, pooling, or any other operation, and the output is passed to the next layer. Furthermore, the number of layers in the estimation model is not limited.

[0134] Figure 9 will be used to explain the structure of the estimation model in this embodiment. Note that if only one image is used, information about the original facial features will be mixed in, making it difficult to extract information about facial features related to frailty. Therefore, in the example described later, we will explain an example in which processing is performed using a combination of facial image data, which is facial image data of the subject when they are making an expression, and expressionless image data, which is when they are expressionless, as facial image data that includes changes in the subject's facial expression.

[0135] As shown in Figure 9, the estimation model in this embodiment includes a route consisting of one or more layers that process facial image data and a route consisting of one or more layers that process neutral image data. The estimation model combines the features (individual features in this embodiment) independently extracted in each route and processes them in a fully connected layer to calculate an estimated value of the frailty index. In this embodiment, since the layers constituting each route are configured independently of the layers of other routes, the weights of the layers constituting each route in the estimation model are learned independently. In this embodiment, each route functions as an encoder that extracts features from facial image data and is configured using ConvNeXt, EfficientNet, ViT (Vision Transformer), etc., to extract features from images, but each route may be configured using any image recognition technology.

[0136] In the layer that combines feature maps, features extracted from different routes are combined to generate facial expression features. Facial expression features are features related to the facial expressions of the subject, and in this embodiment, they are features used in processing in the fully connected layer, generated based on features obtained from neutral image data and features obtained from facial expression image data. In this embodiment, the difference between features extracted from different routes, the absolute value of the difference, and the product of the features are calculated as derived features and used in processing in the fully connected layer. At this time, the features used in processing in the fully connected layer (facial expression features) may include one or more of the following: a vector obtained by combining feature vectors based on features obtained from facial expression image data and neutral image data, the difference between the feature vector based on neutral image data and the feature vector based on facial expression image data, the absolute value of the difference, and the product of the feature vector based on neutral image data and the feature vector based on facial expression image data. Furthermore, in this embodiment, since features are obtained by performing similar processing (convolution, pooling, etc.) using the same encoder on both the neutral image data and the facial image data, the feature vector obtained from the neutral image data and the feature vector obtained from the facial image data have the same number of dimensions, semantic axes, and other similar structures.

[0137] Furthermore, the way the face moves changes depending on the situation (for example, background conditions of the input data such as the type of task, emotions, and the gender of the subject). Therefore, in this embodiment, Feature-wise Linear Modulation (FiLM) is used to adjust the features based on background conditions such as the type of task, emotions, and types of facial expressions. FiLM is a technique that transforms features by performing a linear transformation on each feature x as shown in Equation 3 below. In this embodiment, the scale γ and shift β used for feature transformation are generated based on background conditions including the content of the task, emotions, and the gender of the subject. The background conditions are the conditions for capturing the subject's facial image data, which is input into the estimation model and used to calculate the frailty index, and include the type of task instructed to the subject, emotions, types of facial expressions, and the gender of the subject. Other data, such as age included in the subject data, may also be used as background conditions. In this embodiment, feature adjustment using FiLM is performed on each individual feature included in the feature vector, but it may be performed similarly on the feature vector, which is a set of features. In this case, a vector for transforming the feature vector is generated based on the situational data, and the feature transformation may be performed using this vector.

[0138]

number

[0139] Furthermore, in this embodiment, important features are weighted, for example, if the mouth is a distinctive feature, weights are assigned to the feature corresponding to the mouth. By adopting such a configuration, the estimation model can be processed while considering the strong influence of key features. In this embodiment, gates are provided in the layer that combines feature maps and in the fully connected layer, and important features (in this embodiment, the difference between the features of each root) are weighted. At this time, the gate g for weighting is generated based on the features obtained from the neutral image data, the features obtained from the facial image data, and the absolute value of the difference between these feature vectors. Gate g is a vector that indicates the importance of each feature included in the feature vector. In this embodiment, important features (in this embodiment, especially the difference) are weighted by multiplying the obtained features by gate g. Note that the processing related to weighting of this feature may be performed in the layer that combines feature maps or in the fully connected layer. In addition, in this embodiment, weighting using gates is performed on the difference vector, but weighting using gates may also be performed on feature vectors other than the difference, such as the product. Furthermore, in this embodiment, the neural network may be equipped with a Neural Attention-based Adaptive Fusion (NAAF) module to adaptively extract and enhance difference features. This module takes the difference between each feature map obtained from neutral image data and facial image data as input, dynamically calculates the contribution of each feature using a gating mechanism, and learns to emphasize the difference components with high importance. This allows for the efficient extraction of effective features derived from facial expression changes, thereby improving the accuracy of estimating multiple frailty indices.

[0140] In this embodiment, the estimation model processes the acquired facial image data by extracting features from facial image data, generating derived features such as differences from the extracted features, assigning weights to the features using gates, and correcting them with FiLM. However, the structure of the estimation model and the order of processing are not limited to this.

[0141] Furthermore, in the example shown in Figure 9, the estimation model consists of two routes that process different data, but there may be more than two routes. In this case, the estimation model may further include a route consisting of a layer that processes difference image data (difference image data) generated based on data from multiple face images. Difference image data is difference data generated based on multiple face image data, and may be data obtained by taking the difference between images at the pixel level, or data obtained by taking the difference in features.

[0142] Furthermore, if we only track data related to facial movement, such as the difference between expressionless image data and feature data obtained from expression image data, learning may become unstable, and it may become difficult to estimate people whose facial expressions do not change. Therefore, in this embodiment, the estimation model has one or more layers that perform the main task (main task execution layer), as well as auxiliary branches that perform an auxiliary task using only expressionless image data as input. In the learning of the estimation model in this embodiment, the neural network that performs the main task (calculation of an estimated value of the frailty index) is trained based on the loss function of the auxiliary task and the loss function of the main task. In this embodiment, the loss function of the auxiliary branches is adjusted by a predetermined weight λ_aux and used in the learning of the estimation model. The main task execution layer is a layer that contributes to the calculation of an estimated value of the frailty index, which is the main task, and in this embodiment it is composed of one or more fully connected layers, but it may also include a layer for extracting features.

[0143] If the features derived from expressionless image data and the features in the fully connected layer containing difference data contain the same information twice, training resources will be wasted, potentially leading to overfitting or noise amplification. Therefore, in training the estimation model in this embodiment, a penalty is imposed to minimize the dot product between the features derived from expressionless image data and the features in the fully connected layer.

[0144] Furthermore, the following describes an example in which, in addition to the facial image data mentioned above, data about the target subject (subject data) is used in the processing related to the estimation of the frailty index. In this case, the estimation model is a multimodal model that accepts inputs containing different types of features and generates an output. The subject data is data about the target subject obtained using questionnaires, etc., and includes data on the subject's attributes such as gender and age, and data on questionnaire responses (such as whether the subject holds a driver's license, educational history, whether they live with others and information about those living with others). This subject data is combined with features extracted from facial image data in the layer that connects the feature maps of the estimation model and used in processing in the fully connected layer to calculate the estimated value of the frailty index. In this embodiment, 20 features with a high contribution to the prediction are input to the estimation model and used to calculate the estimated value of the frailty index, but there is no limit to the number of features that can be input to the estimation model.

Claims

1. Image acquisition process to acquire facial image data including changes in the subject's facial expression, An estimation method comprising: an estimation step of calculating estimated values ​​of a plurality of frailty indicators relating to a subject from the subject's facial image data, based on an estimation model for estimating frailty indicators from facial image data including changes in facial expression.

2. The estimation method according to claim 1, wherein the frailty index includes two or more of the following: a physical index relating to physical condition, a cognitive index relating to cognitive function, a psychological index relating to psychological state, a social index relating to social state, and a functional index relating to daily living function.

3. The estimation method according to claim 2, wherein in the estimation step, estimated values ​​of the physical indicator, the cognitive indicator, the psychological indicator, the social indicator, and the life function indicator are calculated based on the estimation model.

4. The estimation method according to any one of claims 2 to 3, wherein the physical indicators include any of the following: an indicator relating to physical function, an indicator relating to body composition, and an indicator based on a medical interview regarding physical condition.

5. The estimation method according to any one of claims 2 to 3, wherein the cognitive index includes an index relating to cognitive function.

6. The estimation method according to any one of claims 2 to 3, wherein the social indicators include any of the following: an indicator relating to cohabitants, an indicator relating to social interaction, an indicator relating to going out habits, an indicator relating to hobby activities, and an indicator relating to feelings of loneliness.

7. The estimation method according to any one of claims 1 to 3, wherein the estimation model is generated based on facial image data including changes in the subject's facial expression and the measured value of the frailty index at the time the facial image data of the subject was acquired.

8. The estimation method according to claim 7, wherein the estimation model is generated using an index relating to the amount of movement of facial muscles obtained based on the facial image data and the measured value of the frailty index at the time the facial image data of the subject was obtained as training data.

9. The aforementioned estimation model was generated using facial image data obtained by giving subjects tasks that elicited specific facial expressions. The estimation method according to any one of claims 1 to 3, wherein the image acquisition step includes giving the subject the task and acquiring facial image data.

10. The estimation method according to claim 9, wherein the estimation model is generated using as training data an evaluation value based on a comparison of an expression recognized from the subject's facial image data and the specific expression, and the measured value of the frailty index when the subject's facial image data was acquired.

11. The estimation method according to claim 9, wherein the task is given by instructing the subject and target to make a specific facial expression.

12. The aforementioned instructions involve presenting specific facial expressions to the subjects and individuals and encouraging them to imitate them. The estimation method according to claim 11, wherein the image acquisition step includes presenting a specific facial expression to a subject and instructing them to imitate it, thereby acquiring facial image data.

13. The estimation method according to claim 9, wherein the task is given by instructing the subject and the target to imagine a hypothetical situation and to express emotions as they feel them.

14. The estimation model is a model that estimates frailty indicators from information about facial movements, The estimation method according to claim 1, wherein the estimation step includes obtaining information regarding facial movements in the facial expression change from the facial image data and estimating a plurality of frailty indices based on the information regarding facial movements.

15. The estimation method according to claim 14, wherein the estimation step includes extracting feature quantities relating to the movement of facial feature points in facial expression changes from the facial image data as information relating to the movement of the face.

16. The estimation method according to claim 15, wherein the aforementioned feature includes at least one of the following (A) to (F): (A) Amount of movement of feature points included in the face (B) Ratio of the amount of movement of multiple feature points included in the face (C) Direction of movement of feature points included in the face (D) The movement speed of the feature points included in the face or its maximum value. (E) Response time to instructions in the movement of feature points included in the face (F) The integral value of the moving velocity during the response time.

17. The feature quantity includes (E) the response time to the instruction in the movement of the feature points included in the face, or (F) the integral value of the movement speed during the response time. The estimation method according to claim 16, wherein the response time includes at least one of the following (a) to (c): (a) Reaction time, which indicates the time from the instruction until the movement speed of the feature point reaches its maximum. (b) Facial expression adjustment time from when the movement speed of the feature point reaches its maximum until the movement speed of the feature point becomes sufficiently small. (c) The sum of the reaction time and facial expression adjustment time.

18. The estimation method according to claim 15, wherein the feature quantity includes a quantity based on a comparison of the left and right sides of the face with respect to the movement of the feature point.

19. The estimation method according to any one of claims 15 to 18, wherein the feature points include one or more points indicating a part of the face.

20. Image acquisition process to acquire facial image data including changes in the subject's facial expression, An estimation program that causes a computer to perform an estimation step of calculating estimated values ​​of multiple frailty indicators relating to a subject from the subject's facial image data, based on an estimation model for estimating frailty indicators from facial image data including changes in facial expression.

21. An image acquisition unit that acquires facial image data including changes in the subject's facial expression, An estimation device comprising: an estimation unit that calculates estimated values ​​of a plurality of frailty indicators relating to a subject from the subject's facial image data, based on an estimation model for estimating frailty indicators from facial image data including changes in facial expression; and an estimation unit that calculates estimated values ​​of a plurality of frailty indicators relating to the subject from the subject's facial image data.

22. The aforementioned psychological indicators include any one of the following: an indicator related to depression, an indicator related to anxiety, or an indicator related to loneliness. The estimation method according to any one of claims 2 to 3.

23. The aforementioned functional ability indicators include either an indicator relating to the level of functional ability or an indicator relating to independence in daily life. The estimation method according to any one of claims 2 to 3.

24. The aforementioned facial image data includes facial image data when an expression is made and facial image data when no expression is made. The estimation method according to claim 1.

25. Based on the features extracted from the facial expression image data and the features extracted from the neutral expression image data, one or more of the following are calculated: the difference, the absolute value of the difference, and the product, to calculate an estimated value of the frailty index. The estimation method according to claim 24.

26. The estimation model includes an auxiliary branch that performs an auxiliary task using the expressionless image data as input, One or more layers of the estimation model that perform the main task are trained, including the loss function in the auxiliary task. The estimation method according to claim 24.

27. Using the features extracted from the aforementioned facial expression image data and the features extracted from the aforementioned expressionless image data, features related to facial expression are combined and generated. The estimation model is trained to minimize the dot product of the features extracted from the expressionless image data and the combined and generated features. The estimation method according to claim 24.

28. In the estimation step, the estimation model further calculates an estimated value of the frailty index based on the subject data of the subject. The estimation method according to claim 1.

29. In the estimation process described above, the scale and shift are determined based on background conditions of the input data (one or more such as task type, facial expression type, gender, etc.). Based on the scale and shift, the combined facial expression features are adjusted and optimized to calculate an estimated value of the frailty index. The estimation method according to claim 27.

30. The estimation model is equipped with an auxiliary classifier that performs the task of identifying a person based on the features obtained from the facial image data. One or more layers performing the main task of the estimation model are updated by inverting and returning the gradient obtained from the auxiliary classifier. The estimation method according to claim 24.

31. The auxiliary classifier is configured not to learn features for identifying individuals from the features obtained from the expressionless image data. The estimation method according to claim 30.

32. The aforementioned estimation model is a model that has been trained on data extracted through sampling that increases the probability of extracting a small number of classes. The estimation method according to claim 1.

Images