Cognitive function evaluation system

Abstract

To provide a cognitive function evaluation system capable of accurately evaluating a cognitive function of a person to be evaluated.SOLUTION: A system 100 includes a motion detection part 20, a first motion feature extraction part 2a, a second motion feature extraction part 2b, and an output part 4. The motion detection part 20 images a person SJ to be evaluated who performs an exercise task and a dual task and generates a plurality of frames indicating three-dimensional coordinates of a plurality of joints of the person SJ to be evaluated obtained from a result of imaging along a time series. The first motion feature extraction part 2a extracts a first motion features hs indicating features of a spatial physical relation of the plurality of imaged joints and a feature of temporal fluctuation of the plurality of imaged joints on the basis of a frame group corresponding to the exercise task. Likewise, the second motion feature extraction part 2b extracts a second motion feature hd on the basis of a frame group corresponding to the dual task. The output part 4 outputs evaluation data indicating the cognitive function of the person SJ to be evaluated on the basis of the first motion feature hs and the second motion feature hd.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a cognitive function assessment system. [Background technology]

[0002] Early detection of cognitive decline is important for delaying the progression of dementia. For example, Non-Patent Document 1 discloses a system for early detection of cognitive decline using dual tasks. Specifically, the system of Non-Patent Document 1 calculates 12 types of dementia-related features from data collected by having subjects perform single tasks and dual tasks, and estimates MMSE (Mini-Mental State Examination) scores using machine learning.

[0003] More specifically, in the system of Non-Patent Document 1, the single task is a motor task in which the subject is tasked with stamping their feet. The dual task is a task in which the subject is tasked with simultaneously performing a motor task and a cognitive task. The motor task of the dual task, like the single task, involves the subject stamping their feet. The cognitive task of the dual task involves the subject answering a calculation problem or a game of "rock-paper-scissors." The system of Non-Patent Document 1 calculates the following six types of features (12 types in total) from each of the single task data and dual task data: (1) Average stepping speed (2) Standard deviation of stepping speed (3) Average knee angle (4) Standard deviation of knee angle (5) Correct answer rate for cognitive tasks (6) Average number of responses to cognitive tasks [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Taku Matsuura et al., "Estimation of cognitive function scores in elderly people based on dual-task gait analysis," Institute of Electronics, Information and Communication Engineers, Technical Research Report, Kawasaki, Vol. 119, No. HCS2019-99, pp. 83-88, Mar. 2020 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the system in Non-Patent Document 1 estimates cognitive function scores based on limited features, resulting in low accuracy. Therefore, there is room for improvement.

[0006] The present invention has been made in consideration of the above problems, and its purpose is to provide a cognitive function assessment system that can more accurately assess the cognitive function of a subject. [Means for solving the problem]

[0007] The cognitive function evaluation system according to the present invention evaluates the cognitive function of an evaluation subject. The cognitive function evaluation system includes a motion detection unit and an evaluation unit. The motion detection unit captures an image of the evaluation subject performing a motor task and a dual task, and generates a plurality of frames in a time series, which indicate three-dimensional coordinates of a plurality of joints of the evaluation subject obtained from the image capture result. The evaluation unit evaluates the cognitive function of the evaluation subject based on the plurality of frames. The evaluation unit has a machine learning model that constructs a neural network. The neural network includes a first motion feature extraction unit, a second motion feature extraction unit, and an output unit. The first motion feature extraction unit extracts a first motion feature indicating a feature of a spatial positional relationship of the plurality of joints and a feature of a temporal variation of the plurality of joints based on a motor task frame group, which is a plurality of frames corresponding to the motor task, among the plurality of frames. The second motion feature extraction unit extracts a second motion feature indicating a feature of a spatial positional relationship of the plurality of joints and a feature of a temporal variation of the plurality of joints based on a dual task frame group, which is a plurality of frames corresponding to the dual task, among the plurality of frames. The output unit outputs evaluation data indicating a cognitive function of the subject based on the first and second action features. The exercise task indicates a task of assigning a first exercise to the subject. The dual task indicates a task of assigning a second exercise and an answer to an intelligence question (cognitive question) simultaneously to the subject.

[0008] In one embodiment, the output section outputs data classifying the evaluation subject as positive or negative as the evaluation data.

[0009] In one embodiment, the output unit outputs, as the evaluation data, data classifying the subject into a dementia class or a class including mild cognitive impairment and non-dementia, or outputs data classifying the subject into a class including dementia and mild cognitive impairment, or a non-dementia class.

[0010] In one embodiment, the parameter values ​​of the neural network are determined based on a first loss function that optimizes the sum of sensitivity, which indicates the rate at which positives can be correctly recognized, and specificity, which indicates the rate at which negatives can be correctly recognized.

[0011] In one embodiment, the output unit includes an acquisition unit that acquires a feature difference indicating a difference between the first action feature and the second action feature, and outputs the evaluation data based on the first action feature, the second action feature, and the feature difference.

[0012] In one embodiment, the output unit includes an acquisition unit that acquires a feature difference indicating a difference between the first action feature and the second action feature. The output unit outputs the evaluation data based on the first action feature, the second action feature, and the feature difference. Parameter values ​​of the neural network are determined based on the first loss function and a second loss function that maximizes the feature difference.

[0013] In one embodiment, the first motion feature extraction unit generates a spatial graph showing the spatial relationship of the joints for each frame, convolves the spatial graph, generates a time graph showing the fluctuation of the same joint between adjacent frames, and convolves the time graph, thereby extracting the first motion feature. The second motion feature extraction unit generates a spatial graph showing the spatial relationship of the joints for each frame, convolves the spatial graph, generates a time graph showing the fluctuation of the same joint between adjacent frames, and convolves the time graph, thereby extracting the second motion feature.

[0014] In one embodiment, the first motion and the second motion include the same motion as each other. Effect of the Invention

[0015] According to the cognitive function assessment system of the present invention, the cognitive function of the subject can be assessed with greater accuracy. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing a cognitive function evaluation system according to a first embodiment of the present invention. [Diagram 2] 1(a) to 1(d) are diagrams showing examples of tasks presented to an evaluation subject by a task presenter included in the cognitive function evaluation system according to the first embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram showing another example of an intelligence question presented to an evaluation subject by a task presenter included in the cognitive function assessment system according to the first embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram showing the configuration of a cognitive function assessment system according to a first embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing a first neural network constructed by a machine learning model included in the cognitive function assessment system according to the first embodiment of the present invention. [Figure 6] FIG. 1 illustrates a model generation system that generates a machine learning model. [Figure 7] FIG. 2 is a block diagram showing a part of the configuration of a model generation system. [Figure 8] FIG. 2 is a block diagram showing the configuration of an evaluation unit during training of a first neural network included in the cognitive function evaluation system according to the first embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing a second neural network constructed by a machine learning model included in the cognitive function assessment system according to embodiment 2 of the present invention. [Figure 10] FIG. 11 is a block diagram showing the configuration of an evaluation unit during training of a second neural network included in a cognitive function evaluation system according to embodiment 3 of the present invention. [Figure 11] FIG. 11 is a diagram showing a cognitive function evaluation system according to a fourth embodiment of the present invention. [Figure 12] FIG. 11 is a block diagram showing the configuration of a cognitive function assessment system according to a fourth embodiment of the present invention. [Figure 13]FIG. 11 is a diagram showing a third neural network constructed by a machine learning model included in the cognitive function assessment system according to embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the cognitive function evaluation system of the present invention will be described with reference to the drawings (FIGS. 1 to 13). However, the present invention is not limited to the following embodiments, and can be implemented in various aspects without departing from the gist of the present invention. Note that, where the explanation is repeated, the explanation may be omitted as appropriate. In addition, in the drawings, the same or corresponding parts are denoted by the same reference symbols, and the explanation will not be repeated.

[0018] [Embodiment 1] First, a cognitive function assessment system 100 of this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the cognitive function assessment system 100 of this embodiment. The cognitive function assessment system 100 of this embodiment assesses the cognitive function of the subject SJ. More specifically, the cognitive function assessment system 100 of this embodiment classifies the subject SJ into a dementia (Cognitive Impairment) class, or a class including Mild Cognitive Impairment (MCI) and non-dementia. Alternatively, the cognitive function assessment system 100 of this embodiment classifies the subject SJ into a class including dementia and mild cognitive impairment, or a non-dementia class.

[0019] Mild cognitive impairment is a pre-dementia stage, and refers to an intermediate state between a normal state and a dementia state. Specifically, mild cognitive impairment refers to a state in which cognitive functions such as memory and attention are impaired, but not to the extent that they interfere with daily life.

[0020] As shown in FIG. 1, the cognitive function assessment system 100 of this embodiment includes a task presenting unit 10. The task presenting unit 10 presents a task to be performed by the assessment subject SJ. In this embodiment, the task presenting unit 10 has a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The task presenting unit 10 displays on the display a screen showing the task to be performed by the assessment subject SJ. The display is installed, for example, in front of the assessment subject SJ.

[0021] The tasks to be performed by the evaluation subject SJ include an exercise task (single task) and a dual task. In this embodiment, the evaluation subject SJ performs the exercise task and the dual task consecutively in this order.

[0022] The motor task refers to a task that assigns a first movement to the person to be evaluated SJ. The dual task refers to a task that has the person to be evaluated SJ simultaneously perform a motor task and an intellectual task (or cognitive task). Specifically, the dual task refers to a task that assigns the person to be evaluated SJ simultaneously to perform a second movement and answer an intellectual question (or cognitive question). The first and second movements include, for example, stepping on the spot, walking, running, or skipping. The first and second movements may be the same movement as each other, or may be different movements from each other. The intellectual question (or cognitive question) includes, for example, a calculation problem, a position memory problem, or a rock-paper-scissors problem.

[0023] More specifically, the subject SJ performs an exercise task (single task) for a predetermined time (e.g., 20 seconds). Thereafter, the subject SJ performs a dual task for a predetermined time (e.g., 30 seconds). Hereinafter, the time during which the subject SJ performs an exercise task (single task) may be referred to as "exercise task performance time." Additionally, the time during which the subject SJ performs a dual task may be referred to as "dual task performance time." Note that the lengths of the exercise task performance time and the dual task performance time are arbitrary.

[0024] In this embodiment, the exercise (first exercise) imposed on the evaluation subject SJ by the exercise task (single task) and the exercise (second exercise) imposed on the evaluation subject SJ by the dual task exercise task include the same exercise.

[0025] Next, the cognitive function assessment system 100 of this embodiment will be further described with reference to Fig. 1. As shown in Fig. 1, the cognitive function assessment system 100 further includes a motion detection unit 20 and an evaluation unit 40.

[0026] The motion detection unit 20 captures an image of the subject SJ performing the motor task and the dual task at least once, and generates a plurality of frames in chronological order indicating the three-dimensional coordinates of a plurality of joints of the subject SJ obtained from the image capture results. In particular, the motion detection unit 20 has an imaging unit 21 and a motion capture unit 22. In this embodiment, the subject SJ performs the motor task and the dual task once. Hereinafter, a task that causes the subject SJ to perform the motor task and the dual task once may be referred to as a "task set."

[0027] The imaging unit 21 captures an image of the evaluation subject SJ to generate a captured image. Specifically, the imaging unit 21 captures an image of the evaluation subject SJ performing a task set to generate moving image data (or video data). The imaging unit 21 may have a sensor such as a CCD image sensor, a CMOS image sensor, or a range sensor. The imaging unit 21 is disposed, for example, in front of the evaluation subject SJ. By disposing the imaging unit 21 in front of the evaluation subject SJ, all of the joints of the evaluation subject SJ can be captured.

[0028] The motion capture unit 22 generates multiple frames in chronological order that indicate the three-dimensional coordinates of each of the imaged joints of the evaluation subject SJ, among all the joints, based on the imaging results (images) of the imaging unit 21. In more detail, the motion capture unit 22 generates a human skeletal model based on the imaged images (video data). Each frame includes a human skeletal model of the evaluation subject SJ. The human skeletal model includes each of the imaged joints of the evaluation subject SJ (all joints included in the imaged images). Hereinafter, the multiple frames generated by the motion detection unit 20 may be referred to as "consecutive frames."

[0029] The motion capture unit 22 includes, for example, a processor and a storage unit. The motion capture unit 22 may include, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) as the processor. The storage unit stores a computer program executed by the processor. The computer program includes a computer program for generating successive frames from the output of the imaging unit 21. The storage unit includes, for example, a semiconductor memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0030] The evaluation unit 40 evaluates the cognitive function of the evaluation subject SJ based on the consecutive frames. In this embodiment, the evaluation unit 40 classifies the evaluation subject SJ into a dementia class, or a class including mild cognitive impairment and non-dementia. Alternatively, the evaluation unit 40 classifies the evaluation subject SJ into a class including dementia and mild cognitive impairment, or a non-dementia class. In detail, the evaluation unit 40 has a machine learning model ML. The evaluation unit 40 inputs the consecutive frames into the machine learning model ML. The machine learning model ML classifies the evaluation subject SJ into a dementia class, or a class including mild cognitive impairment and non-dementia, based on the consecutive frames. Alternatively, the evaluation unit 40 classifies the evaluation subject SJ into a class including dementia and mild cognitive impairment, or a non-dementia class, based on the consecutive frames.

[0031] Next, an example of a task presented by the task presenter 10 to the subject SJ will be described with reference to Fig. 2(a) to Fig. 2(d). Fig. 2(a) to Fig. 2(d) are diagrams showing an example of a task presented to the subject SJ by the task presenter 10 included in the cognitive function assessment system 100 of this embodiment. In the example shown in Fig. 2(a) to Fig. 2(d), the exercise (first and second exercises) assigned to the subject SJ is "tapping", and the intelligence problem (or cognitive problem) assigned to the subject SJ is "calculation problem".

[0032] As shown in Fig. 2(a), the task presenter 10 first displays a first notification screen 11 that notifies the evaluation subject SJ of the start of the task. Next, as shown in Fig. 2(b), the task presenter 10 presents an exercise task (single task) to the evaluation subject SJ. Specifically, the task presenter 10 displays an exercise presentation screen 12 that presents the exercise (tapping) to be assigned to the evaluation subject SJ.

[0033] In this embodiment, the exercise (second exercise) assigned to the evaluation subject SJ in the dual task is the same as the exercise (first exercise) assigned to the evaluation subject SJ in the exercise task (single task). Therefore, the task presenter 10 displays a screen presenting an intellectual task (cognitive task) to the evaluation subject SJ after a predetermined exercise task performance time has elapsed. The evaluation subject SJ performs the intellectual task (cognitive task) presented by the task presenter 10 while performing the exercise task (tapping).

[0034] 2(c), the task presenter 10 displays a problem presenting screen 13a presenting a calculation problem. For example, the task presenter 10 causes a subtraction problem to be displayed on the display as the calculation problem.

[0035] The task presenter 10 ends the presentation of the calculation problem at a predetermined timing. Specifically, the task presenter 10 erases the problem presentation screen 13a from the display. After erasing the problem presentation screen 13a from the display, the task presenter 10 displays an answer candidate presentation screen 13b showing two answer candidates.

[0036] The cognitive function assessment system 100 of this embodiment further includes a left hand answer switch (not shown) and a right hand answer switch (not shown). The assessment subject SJ answers the intelligence question (or cognitive question) presented by the task presenter 10 by pressing the right hand answer switch or the left hand answer switch. Specifically, the assessment subject SJ presses one of the two answer switches to select one of two answer candidates.

[0037] The answer switch for the left hand may be held in the left hand of the person to be evaluated SJ, or may be fixed to a handrail installed on the left side of the person to be evaluated SJ. Similarly, the answer switch for the right hand may be held in the right hand of the person to be evaluated SJ, or may be fixed to a handrail installed on the right side of the person to be evaluated SJ.

[0038] When the evaluation subject SJ selects an answer, a calculation problem different from the one just answered is presented as the next problem by the task presenter 10. Thereafter, the presentation of the calculation problem is repeated until a predetermined dual task performance time has elapsed.

[0039] As shown in FIG. 2(d), after a predetermined dual task performance time has elapsed, the task presenter 10 displays on the display a second notification screen 14 to inform the subject SJ of the completion of the task.

[0040] In the example shown in FIG. 2(c), the calculation problem is a subtraction problem, but the calculation problem is not limited to a subtraction problem. The calculation problem may be an addition problem. Alternatively, the calculation problem may include a subtraction problem and an addition problem.

[0041] Furthermore, the intellectual problem (or cognitive problem) presented by the task presenter 10 is not limited to a calculation problem. For example, the intellectual problem (or cognitive problem) may be a location memory problem or a rock-paper-scissors problem.

[0042] Fig. 3 is a diagram showing another example of an intelligence problem (or cognitive problem) presented to the assessment subject SJ by the task presenter 10 included in the cognitive function assessment system 100 of this embodiment. In the example shown in Fig. 3, the intelligence problem (or cognitive problem) is a "position memory problem." As shown in Fig. 3, when the intelligence problem (or cognitive problem) is a position memory problem, the task presenter 10 displays a problem presenting screen 13a in which a figure is placed in one of four areas in which figures can be placed.

[0043] After erasing the question presentation screen 13a from the display, the task presenter 10 displays the answer candidate presentation screen 13b in which a shape is placed in one of the four areas in which shapes can be placed. The answer candidate presentation screen 13b displays the question statement along with two answer candidates, "Yes" and "No." The question statement indicates a question that can be answered with "Yes" or "No." Here, the subject SJ is asked whether the position in which the shape was placed is the same between the question presentation screen 13a and the answer candidate presentation screen 13b.

[0044] If the evaluation subject SJ determines that the position where the figure was placed is the same between the question presentation screen 13a and the answer candidate presentation screen 13b, he / she presses the answer switch for the left hand to select "Yes." Alternatively, if the evaluation subject SJ determines that the position where the figure was placed is different between the question presentation screen 13a and the answer candidate presentation screen 13b, he / she presses the answer switch for the right hand to select "No."

[0045] Next, the "rock-paper-scissors problem" will be described. When the intelligence problem (or cognitive problem) is a "rock-paper-scissors problem," for example, the task presenter 10 displays one of "rock," "scissors," and "paper" on the problem presenting screen 13a. Then, one of "rock," "scissors," and "paper," a problem statement, and "yes" and "no" are displayed on the answer candidate presenting screen 13b. On the answer candidate presenting screen 13b, for example, the person to be evaluated SJ is asked whether the finger pose displayed on the answer candidate presenting screen 13b can beat the finger pose displayed on the problem presenting screen 13a.

[0046] If the evaluation subject SJ judges that the finger pose displayed on the answer candidate presentation screen 13b is better than the finger pose displayed on the question presentation screen 13a, he / she presses the answer switch for the left hand to select "Yes." Alternatively, if the evaluation subject SJ judges that the finger pose displayed on the answer candidate presentation screen 13b is worse than the finger pose displayed on the question presentation screen 13a, he / she presses the answer switch for the right hand to select "No."

[0047] Next, the cognitive function assessment system 100 of this embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the cognitive function assessment system 100 of this embodiment. In detail, Fig. 4 shows the configuration of the assessment unit 40. As shown in Fig. 4, the cognitive function assessment system 100 further includes an identification information acquisition unit 50. In addition, the assessment unit 40 includes a memory unit 41 and a processing unit 42.

[0048] Each evaluation subject SJ who uses the cognitive function evaluation system 100 is assigned unique identification information in advance. The identification information acquisition unit 50 acquires the identification information assigned to each evaluation subject SJ. The identification information acquisition unit 50 includes, for example, a card reader, a keyboard, or a touch panel. If the identification information acquisition unit 50 is a card reader, each evaluation subject SJ has the card reader read the identification information carried on the card. If the identification information acquisition unit 50 is a keyboard or a touch panel, each evaluation subject SJ operates the keyboard or touch panel to input the identification information assigned to him or her.

[0049] The storage unit 41 stores various computer programs and various data. Specifically, the storage unit 41 has a storage device. The storage device includes, for example, a semiconductor memory (semiconductor storage device). The storage unit 41 may include, for example, a RAM and a ROM as the semiconductor memory. The storage unit 41 may further include, for example, a VRAM (Video RAM). The storage unit 41 may also have a storage device. The storage unit 41 may include, for example, at least one of an HDD (Hard Disk Drive) and an SSD (Solid State Drive) as the storage device.

[0050] Specifically, the storage unit 41 stores a machine learning model ML. The machine learning model ML is a computer program for evaluating the cognitive function of the evaluation subject SJ based on a plurality of frames (continuous frames) output from the motion detection unit 20.

[0051] The processing unit 42 executes various computer programs stored in the storage unit 41 to perform various processes such as numerical calculations, information processing, and device control. The processing unit 42 has a processor. Alternatively, the processing unit 42 may have a quantum computer. The processor includes, for example, a CPU or an MPU. The processor may include a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit).

[0052] Specifically, when the identification information is acquired by the identification information acquisition unit 50, the processing unit 42 causes the task presentation unit 10 to display various screens as described with reference to Figures 2(a) to 2(d) and 3. In addition, the processing unit 42 inputs a plurality of frames (consecutive frames) output from the action detection unit 20 to the machine learning model ML.

[0053] More specifically, the processing unit 42 extracts consecutive frames acquired during the performance of a motor task (single task) and consecutive frames acquired during the performance of a dual task from the consecutive frames output from the motion detection unit 20, and inputs them to the machine learning model ML. The number of consecutive frames corresponding to the motor task (single task) is, for example, 160 frames. The number of consecutive frames corresponding to the dual task is, for example, 260 frames. Hereinafter, among the consecutive frames (multiple frames) output from the motion detection unit 20, consecutive frames (multiple frames) corresponding to the motor task (single task) may be referred to as a "motor task frame group." Also, among the consecutive frames (multiple frames) output from the motion detection unit 20, consecutive frames (multiple frames) corresponding to a dual task may be referred to as a "dual task frame group."

[0054] The processing unit 42 acquires an evaluation result of the cognitive function of the subject SJ based on the output of the machine learning model ML. In this embodiment, the evaluation result includes a classification result of the cognitive function. The classification result indicates a class into which the cognitive function of the subject SJ is classified. Specifically, the class into which the cognitive function of the subject SJ is classified includes two classes. One of the two classes indicates a dementia class. The other of the two classes indicates a class including mild cognitive impairment and non-dementia. Alternatively, one of the two classes indicates a class including dementia and mild cognitive impairment. The other of the two classes indicates a non-dementia class.

[0055] When the processing unit 42 acquires the evaluation result of the cognitive function of the evaluation subject SJ, the processing unit 42 stores the evaluation result of the cognitive function of the evaluation subject SJ in association with the identification information assigned to the evaluation subject SJ in the memory unit 41. As a result, the memory unit 41 stores the history of the evaluation result of the cognitive function of the evaluation subject SJ in association with the identification information.

[0056] The cognitive function assessment system 100 may include a result presentation unit that presents the assessment result of the cognitive function or the history of the assessment result of the cognitive function to the assessment subject SJ. The result presentation unit may include, for example, a printer. Alternatively, the result presentation unit may include a communication device.

[0057] When the cognitive function evaluation system 100 includes a printer, the processing unit 42 controls the printer to output a sheet on which the evaluation results of the cognitive function are printed from the printer. Alternatively, the processing unit 42 creates a table or graph showing the history of the evaluation results of the cognitive function, and outputs a sheet on which the table or graph showing the history of the evaluation results of the cognitive function is printed from the printer. When the cognitive function evaluation system 100 includes a communication device, the processing unit 42 may control the communication device to send an email showing the evaluation results of the cognitive function of the evaluation subject SJ to an email address registered in advance by the evaluation subject SJ, for example. Alternatively, the processing unit 42 may send an email showing a table or graph showing the history of the evaluation results of the cognitive function.

[0058] Next, the machine learning model ML will be described with reference to Fig. 5. Fig. 5 is a diagram showing a first neural network NW1 constructed by the machine learning model ML included in the cognitive function assessment system 100 of this embodiment. The first neural network NW1 is, for example, a neural network that performs deep learning.

[0059] 5, the machine learning model ML constructs a first neural network NW1. Therefore, the evaluation unit 40 includes the first neural network NW1. The first neural network NW1 includes a first motion feature extraction unit 2a, a second motion feature extraction unit 2b, a first conversion unit 3a, a second conversion unit 3b, and an output unit 4.

[0060] As already explained, the evaluation subject SJ performs the task set once. The movement detection unit 20 captures the evaluation subject SJ while the evaluation subject SJ performs the task set once. As a result, consecutive frames showing the evaluation subject SJ performing the task set once are acquired. The processing unit 42 extracts a motor task frame group and a dual task frame group from the consecutive frames. Then, the processing unit 42 inputs the motor task frame group to the first movement characteristic extraction unit 2a and inputs the dual task frame group to the second movement characteristic extraction unit 2b.

[0061] The first movement feature extraction unit 2a extracts a first movement feature based on a plurality of frames (movement task frame group) corresponding to a movement task among a plurality of frames (consecutive frames) generated by the movement detection unit 20. The first movement feature indicates a movement feature of the evaluation subject SJ performing the movement task. In detail, the first movement feature indicates a feature of the spatial positional relationship of each joint (all joints of the human skeletal model) of the imaged evaluation subject SJ and a feature of the temporal fluctuation of each joint (all joints of the human skeletal model) of the imaged evaluation subject SJ.

[0062] The second motion feature extraction unit 2b extracts a second motion feature based on a plurality of frames (dual task frame group) corresponding to a dual task among a plurality of frames (consecutive frames) generated by the motion detection unit 20. The second motion feature indicates a feature of the motion of the evaluation subject SJ performing a dual task. In detail, the second motion feature indicates a feature of the spatial positional relationship of each joint (all joints of the human skeletal model) of the imaged evaluation subject SJ and a feature of the temporal fluctuation of each joint (all joints of the human skeletal model) of the imaged evaluation subject SJ.

[0063] Specifically, the first motion feature extraction unit 2a captures the human body skeleton model as a graph structure. More specifically, the first motion feature extraction unit 2a captures two graph structures from a group of motion task frames. In detail, the first motion feature extraction unit 2a generates a spatial graph showing the spatial (three-dimensional) positional relationship of each joint (all joints of the human body skeleton model) of the imaged evaluation subject SJ for each frame, and generates a time graph showing the fluctuation of the same joint between adjacent frames (i.e., the temporal fluctuation of the joint). Then, the first motion feature extraction unit 2a extracts the first motion feature by convolving the spatial graph and the time graph. That is, the first motion feature extraction unit 2a extracts the feature of the spatial positional relationship of all joints of the human body skeleton model by convolving the spatial graph. Also, the first motion feature extraction unit 2a extracts the feature of the temporal fluctuation of all joints of the human body skeleton model by convolving the time graph. The first motion feature extraction unit 2a includes, for example, a graph convolutional neural network such as a spatio-temporal graph convolutional neural network (ST-GCN).

[0064] The second motion feature extraction unit 2b also captures two graph structures from the dual task frame group, similarly to the first motion feature extraction unit 2a. In detail, the second motion feature extraction unit 2b generates a spatial graph showing the spatial (three-dimensional) positional relationship of each joint (all joints of the human skeletal model) of the imaged evaluation subject SJ for each frame, and generates a time graph showing the fluctuation of the same joint between adjacent frames (i.e., the temporal fluctuation of the joint). Then, the second motion feature extraction unit 2b extracts the second motion feature by convolving the spatial graph and the time graph. That is, the second motion feature extraction unit 2b extracts the feature of the spatial positional relationship of all joints of the human skeletal model by convolving the spatial graph, and extracts the feature of the temporal fluctuation of all joints of the human skeletal model by convolving the time graph. The second motion feature extraction unit 2b includes, for example, a graph convolution neural network such as a space-time graph convolution neural network (ST-GCN).

[0065] For example, when the exercise (first and second exercises) imposed on the subject SJ is stepping (stepping in place), the first motion feature extraction unit 2a extracts spatiotemporal features (first motion feature) of the stepping (exercise) by the subject SJ performing the exercise task from the spatial (three-dimensional) positional relationship and temporal fluctuation of all joints included in the three-dimensional human skeletal model. Similarly, the second motion feature extraction unit 2b extracts spatiotemporal features (second motion feature) of the stepping (exercise) by the subject SJ performing the dual task from the spatial (three-dimensional) positional relationship and temporal fluctuation of all joints included in the three-dimensional human skeletal model.

[0066] The movement features (first movement feature and second movement feature) extracted by the first movement feature extraction unit 2a and the second movement feature extraction unit 2b are expressed in a high-dimensional vector space. The first conversion unit 3a converts the first movement feature (high-dimensional vector space) into a low-dimensional vector space. Similarly, the second conversion unit 3b converts the second movement feature (high-dimensional vector space) into a low-dimensional vector space. That is, the first conversion unit 3a and the second conversion unit 3b perform so-called embedding. The first conversion unit 3a and the second conversion unit 3b include, for example, a fully connected layer (FC). Hereinafter, the first movement feature converted into a low-dimensional vector may be referred to as a "first embedding feature hs". Similarly, the second movement feature converted into a low-dimensional vector may be referred to as a "second embedding feature hd".

[0067] In detail, ST-GCN includes a GCN (Graph Convolutional Network) and a TCN (Temporal Convolutional Network). The GCN extracts spatial features, and the TCN extracts temporal features. For example, the output of the GCN for the t-th frame can be expressed by the following equation (1).

number

[0068] In formula (1), Ak denotes an adjacency matrix, Dk denotes a diagonal matrix or a degree matrix of the adjacency matrix Ak, and x t in denotes the feature of the t-th frame, and w k denotes a learnable weight matrix, and Mk denotes a learnable attention region of the adjacency matrix Ak.

[0069] The TCN performs two-dimensional convolution. The kernel size of the TCN is, for example, "75". The first conversion unit 3a linearly converts, for example, the first motion feature (high-dimensional vector space) into a 64-dimensional vector space. Similarly, the second conversion unit 3b linearly converts the second motion feature (high-dimensional vector space) into a 64-dimensional vector space.

[0070] The output unit 4 outputs evaluation data indicating the cognitive function of the evaluation subject SJ based on the first and second action features. In this embodiment, the output unit 4 outputs data classifying the evaluation subject SJ as positive or negative as the evaluation data. Specifically, the output unit 4 outputs data classifying the evaluation subject SJ into a dementia class (positive) or a class including mild cognitive impairment and non-dementia (negative) as the evaluation data. Alternatively, the output unit 4 outputs data classifying the evaluation subject SJ into a class including dementia and mild cognitive impairment (positive) or a non-dementia class (negative) as the evaluation data. In detail, the output unit 4 of the first neural network NW1 includes a combining unit 411 and a classifying unit 412.

[0071] The combining unit 411 combines the first embedded feature hs and the second embedded feature hd to output a scalar value s. The combining unit 411 includes, for example, a fully connected layer (FC). The classification unit 412 classifies the cognitive function of the subject SJ into positive or negative based on the scalar value s. For example, the scalar value s can be expressed by the following formula (2).

number

[0072] In equation (2), b denotes the bias term, and W denotes the weight matrix of the fully connected layer (FC).

[0073] In detail, the scalar value s includes a positive scalar value s1 and a negative scalar value s2. The positive scalar value s1 indicates the probability that a positive is correctly recognized. The negative scalar value s2 indicates the probability that a negative is correctly recognized. When the positive scalar value s1 is greater than the negative scalar value s2, the classification unit 412 outputs evaluation data indicating a positive result. On the other hand, when the negative scalar value s2 is greater than the positive scalar value s1, the classification unit 412 outputs evaluation data indicating a negative result.

[0074] 1 to 5, the cognitive function of the subject SJ can be evaluated based on the characteristics of the spatial positional relationships and the characteristics of the temporal fluctuations of the joints of the subject SJ obtained from the imaging results. Therefore, the cognitive function of the subject SJ can be evaluated more accurately than a system that evaluates the cognitive function of the subject SJ based on limited characteristics.

[0075] Next, a training method (learning method) for the first neural network NW1 will be described with reference to Figures 6 to 8. Figure 6 is a diagram showing a model generation system 200 that generates a machine learning model ML.

[0076] The model generation system 200 is used for training the first neural network NW1. Specifically, the model generation system 200 creates training data (a data set for learning) and makes the first neural network NW1 learn the training data (the data set for learning). To create the training data, the model generation system 200 collects various types of data.

[0077] In detail, the model generation system 200 collects multiple continuous data (explanatory variables) from multiple subjects who perform a task set. The model generation system 200 also collects label data (objective variables) from each subject. The model generation system 200 then combines the multiple continuous data (explanatory variables) and the multiple label data (objective variables) to create training data (a learning dataset).

[0078] As shown in FIG. 6, the model generation system 200 includes a task presenter 10, a motion detector 20, an evaluator 40, an identification information acquirer 50, and a data collector 70.

[0079] The task presenting unit 10 presents an exercise task and a dual task to the subject, as described with reference to Figures 1 to 5. The motion detecting unit 20 outputs consecutive frames (an exercise task frame group and a dual task frame group), as described with reference to Figures 1 to 5. The identification information acquiring unit 50 acquires the identification information of the subject, as described with reference to Figures 1 to 5.

[0080] The data collection unit 70 collects data obtained by the subject performing the task sets described with reference to Figures 1 to 5. Specifically, the data collection unit 70 collects consecutive frames (a group of motor task frames and a group of dual task frames) from the output of the motion detection unit 20. Note that the data collected by the data collection unit 70 may include multiple pieces of data obtained from the same subject.

[0081] Next, the model generation system 200 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing a part of the configuration of the model generation system 200. In detail, Fig. 7 shows the configuration of the data collection unit 70. As shown in Fig. 7, the data collection unit 70 includes an input unit 71, a storage unit 72, and a processing unit 73. The data collection unit 70 is, for example, a server.

[0082] The input unit 71 is operated by an operator to input various pieces of information to the processing unit 73. The input unit 71 includes input devices such as a keyboard, a mouse, and a touch panel. For example, the operator operates the input unit 71 to input label data (objective variables) used in supervised learning or semi-supervised learning. As a result, a plurality of pieces of label data are collected by the data collection unit 70.

[0083] Specifically, when generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a dementia class or a class including mild cognitive impairment and non-dementia, the label data indicates whether the subject is in the dementia class (positive) or a class including mild cognitive impairment and non-dementia (negative). Similarly, when generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a class including dementia and mild cognitive impairment, or a non-dementia class, the label data indicates whether the subject is in the class including dementia and mild cognitive impairment (positive) or a non-dementia class (negative).

[0084] For example, the worker may input multiple label data by referring to the result of the definitive diagnosis of each subject by a doctor. Specifically, when generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a dementia class or a class including mild cognitive impairment and non-dementia, label data indicating positive is input for subjects whose definitive diagnosis results are dementia, and label data indicating negative is input for subjects whose definitive diagnosis results are mild cognitive impairment or non-dementia. The same applies to generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a class including dementia and mild cognitive impairment, or a non-dementia class.

[0085] The storage unit 72 stores various computer programs and various data. Specifically, the storage unit 72 has a storage device. The storage device includes, for example, a semiconductor memory (semiconductor storage device). The storage unit 72 may include, for example, a RAM and a ROM as the semiconductor memory. The storage unit 72 may further include a VRAM as the semiconductor memory. The storage unit 72 may also have a storage device. The storage unit 72 may include, for example, at least one of an HDD and an SSD as the storage device.

[0086] In this embodiment, the storage unit 72 stores training data (learning data set). Specifically, the storage unit 72 stores consecutive frames (motor task frame group and dual task frame group) acquired from each subject in association with identification information assigned to each subject. The storage unit 72 further stores label data of each subject input from the input unit 71 in association with the identification information of each subject. The storage unit 72 may store, for example, a database of training data. Note that multiple data obtained from the same subject are managed as different data.

[0087] The processing unit 73 performs various processes such as numerical calculations, information processing, and device control by executing various computer programs stored in the storage unit 72. The processing unit 73 has a processor. The processor includes, for example, a CPU or an MPU.

[0088] Specifically, the processing unit 73 associates the label data input from the input unit 71 with the identification information of the subject and stores it in the storage unit 72. Furthermore, the processing unit 73 causes the task presenting unit 10 to display various screens as described with reference to Figs. 2 and 3, acquires consecutive frames (a group of motor task frames and a group of dual task frames) from the output of the motion detecting unit 20, and stores the acquired consecutive frames in the storage unit 72 in association with the identification information of the subject. As a result, the training data is stored in the storage unit 72.

[0089] The processing unit 73 outputs the training data stored in the storage unit 72 to the evaluation unit 40. In this embodiment, the evaluation unit 40 performs mini-batch learning. Therefore, the processing unit 73 generates training data for mini-batch learning and outputs it to the evaluation unit 40. The batch size is set to, for example, "8".

[0090] Next, the model generation system 200 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the evaluation unit 40 during training of the first neural network NW1 included in the cognitive function evaluation system 100 of this embodiment. As shown in Fig. 8, the evaluation unit 40 further includes an input unit 43.

[0091] The input unit 43 is operated by an operator to input various information to the processing unit 42. The input unit 43 includes input devices such as a keyboard, a mouse, and a touch panel. For example, the operator operates the input unit 43 to input a first loss function L1 and a learning rate. For example, when the number of epochs (number of learning times) is set to "50", the initial learning rate may be set to "0.01" and the learning rate may be multiplied by 0.1 at each of epochs 13, 25, and 37.

[0092] During training (learning), the storage unit 41 stores a learning program TP. The learning program TP is a program for executing an algorithm for finding certain rules from training data and generating a model (machine learning model ML) that expresses the rules.

[0093] After providing the first loss function L1 and the learning rate to the learning program TP, the processing unit 42 executes the learning program TP using training data (training data for mini-batch learning) acquired from the data collection unit 70. The training data includes a motor task frame group, a dual task frame group, and label data for each subject. As a result, the neural network constructed by the learning program TP is trained, and a machine learning model ML is generated. Specifically, the processing unit 42 determines the values ​​of multiple parameters (parameter values) included in the first neural network NW1 to values ​​that minimize the first loss function L1. For example, the processing unit 42 may determine the parameter values ​​of the first neural network NW1 based on an algorithm of the backpropagation learning method.

[0094] Here, the first loss function L1 will be described. The first loss function L1 is a loss function that optimizes the sum (P1+P2) of the sensitivity P1, which indicates the rate at which positives can be correctly recognized, and the specificity P2, which indicates the rate at which negatives can be correctly recognized. The processing unit 42 determines the parameter values ​​of the first neural network NW1 based on the first loss function L1. In other words, the parameter values ​​of the first neural network NW1 are determined based on the first loss function L1. The sensitivity P1 can be expressed by the following formula (3). The specificity P2 can be expressed by the following formula (4). Sensitivity P1=N TP / N P =N TP / (N TP +N FN ) · · · (3) Specificity P2=N TN / N N =N TN / (N FP +N TN ) · · · (4)

[0095] In the above formulas (3) and (4), N P indicates the number of positive samples, N N indicates the number of negative samples, N TP denotes the number of correctly classified positive samples, and N TNdenotes the number of correctly classified negative samples, and N FP indicates the number of false positive samples, and N FN indicates the number of false negative samples.

[0096] Specifically, the processing unit 42 searches for the minimum value of the first loss function L1 by the gradient method. That is, the processing unit 42 searches for the minimum value of the first loss function L1 by partially differentiating the first loss function L1. In this embodiment, the processing unit 42 calculates the gradient based on the following formulas (5) to (8). Formula (5) represents the first loss function L1, and formula (6) is an equation obtained by partially differentiating formula (5). Formula (7) represents a "weight."

number

number

number

number

[0097] In formulas (5) to (7), N P denotes the number of positive samples in the mini-batch, and N N indicates the number of negative samples in the mini-batch. Therefore, the weight w(N P , N N ) denotes the harmonic mean of the number of positive and negative samples included in the mini-batch. Also, S P and S N respectively denote the positive and negative indices contained in the mini-batch. As shown in equation (8), f(s i ) is a sigmoid function with a scalar value s as a variable. i In equations (5) and (6), “f(s i ) indicates the probability that the i-th sample is positive, and 1-f(s i ) indicates the probability that the i-th sample is negative. i) is assigned a positive scalar value s1. i ) is assigned the negative scalar value s2.

[0098] In addition, equation (5) is a weight w(N P , N N ) is added.

number

[0099] Equation (9) includes the following equations (10) and (11). Equation (10) shows an equation for relaxing sensitivity P1 using a sigmoid function with scalar value s as a variable. Equation (11) shows an equation for relaxing specificity P2 using a sigmoid function with scalar value s as a variable.

number

number

[0100] Therefore, the above formula (5) indicates a loss function including the sum (P1+P2) of the sensitivity P1 and the specificity P2, and by searching for the minimum value of the loss function shown in the above formula (5) based on the above formula (6), it is possible to determine the parameter value that optimizes the sum (P1+P2) of the sensitivity P1 and the specificity P2. Furthermore, according to the above formula (5), since the gradient does not disappear, learning can be performed using the backpropagation learning method. In addition, by introducing the weight shown in the above formula (7), even if the number of positive samples and the number of negative samples are imbalanced, the influence can be mitigated.

[0101] As described above with reference to Figs. 6 to 8, the training method (learning method) of the first neural network NW1 includes the steps of inputting training data including label data indicating whether the subject is positive or negative, a group of motor task frames, and a group of dual task frames to the first neural network NW1, and adjusting the parameter values ​​of the first neural network NW1 based on the first loss function L1. According to this training method, the first neural network NW1 is trained so that the sum (P1+P2) of the sensitivity P1 and the specificity P2 is optimized. As a result, the first neural network NW1 can evaluate the cognitive function of the subject SJ with even greater accuracy.

[0102] In this embodiment, the cognitive function assessment system 100 has the assessment subject SJ perform the task set once, but the cognitive function assessment system 100 may have the assessment subject SJ perform the task set two or more times in succession.

[0103] In addition, in this embodiment, the cognitive function assessment system 100 has the assessment subject SJ perform an exercise task (single task) and a dual task in this order, but the order in which the assessment subject SJ performs the exercise task (single task) and the dual task can be reversed.

[0104] In addition, in this embodiment, the cognitive function assessment system 100 has the assessment subject SJ perform an exercise task (single task) and a dual task, but the cognitive function assessment system 100 may have the assessment subject SJ perform a task set including an intellectual task (single task), an exercise task (single task), and a dual task. The intellectual task (single task) may be a task that imposes the same intellectual problem (or cognitive problem) as the intellectual task of the dual task on the assessment subject SJ, or may be a task that imposes an intellectual problem (or cognitive problem) different from the intellectual task of the dual task on the assessment subject SJ. The order in which the assessment subject SJ performs the intellectual task (single task), the exercise task (single task), and the dual task is arbitrary.

[0105] Further, the cognitive function assessment system 100 of the present embodiment classifies the cognitive function of the subject SJ, but the cognitive function assessment system 100 may determine the cognitive function score of the subject SJ. The cognitive function score is, for example, a general intelligence assessment scale such as the Mini Mental State Examination (MMSE) score or the Hasegawa Brief Intelligence Scale. In a diagnosis using the MMSE score, a subject is judged to be suspected of having dementia when the MMSE score is 23 points or less, and is judged to be suspected of having mild cognitive impairment when the MMSE score is higher than 23 points and lower than 27 points. Also, a subject is judged to be non-dementia when the MMSE score is higher than 27 points. In this case, the data collection unit 70 collects the cognitive function score of each subject as label data. The cognitive function score can be obtained by having each subject take a paper test (for example, a paper test of the MMSE) in advance.

[0106] In the present embodiment, label data indicating positive and label data indicating negative are input based on the result of the definitive diagnosis by the doctor, but label data indicating positive may be input for subjects whose cognitive function scores are equal to or less than a threshold, and label data indicating negative may be input for subjects whose cognitive function scores are greater than the threshold. Specifically, when generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a dementia class or a class including mild cognitive impairment and non-dementia, label data indicating positive may be input for subjects whose MMSE score is 23 points or less, and label data indicating negative may be input for subjects whose MMSE score is greater than 23 points. When generating a machine learning model ML (first neural network NW1) that classifies the subject SJ into a class including dementia and mild cognitive impairment or a non-dementia class, label data indicating positive may be input for subjects whose MMSE score is 27 points or less, and label data indicating negative may be input for subjects whose MMSE score is greater than 27 points.

[0107] In this embodiment, the parameter values ​​of the first neural network NW1 are determined using the loss function (first loss function L1) described with reference to formulas (5) to (11), but the loss function is not limited to the first loss function L1. The loss function may be a known loss function such as mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE), mean square logarithmic error (MSLE), Huber loss, Poisson loss, hinge loss, and Kullback-Leibler divergence (KLD).

[0108] [Embodiment 2] Next, a second embodiment of the present invention will be described with reference to Fig. 9. However, only differences from the first embodiment will be described, and explanations of the same matters as in the first embodiment will be omitted. The second embodiment differs from the first embodiment in that the machine learning model ML constructs a second neural network NW2.

[0109] 9 is a diagram showing the second neural network NW2 constructed by the machine learning model ML included in the cognitive function assessment system 100 of this embodiment. The second neural network NW2 is, for example, a neural network that performs deep learning.

[0110] As shown in Fig. 9, the machine learning model ML constructs a second neural network NW2. Therefore, the evaluation unit 40 includes the second neural network NW2. The second neural network NW2 includes a first motion feature extraction unit 2a, a second motion feature extraction unit 2b, a first conversion unit 3a, a second conversion unit 3b, and an output unit 4. The output unit 4 of the second neural network NW2 includes a combination unit 411, a classification unit 412, and an acquisition unit 413.

[0111] The acquisition unit 413 calculates and acquires a feature difference indicating the difference between the first movement feature extracted by the first movement feature extraction unit 2a and the second movement feature extracted by the second movement feature extraction unit 2b. The output unit 4 outputs evaluation data based on the first movement feature, the second movement feature, and the feature difference. The difference (feature difference) between the first movement feature and the second movement feature indicates the difference between the movement feature of the evaluation subject SJ when performing a dual task and the movement feature of the evaluation subject SJ when performing a motor task.

[0112] Specifically, the acquiring unit 413 calculates the difference hc between the first embedded feature hs and the second embedded feature hd as the feature difference. Hereinafter, the difference hc between the first embedded feature hs and the second embedded feature hd may be referred to as a "dual task cost hc." For example, the acquiring unit 413 may calculate the dual task cost hc based on the following formula (12). hc = second embedded feature hd - first embedded feature hs (12)

[0113] The combining unit 411 combines the first embedded feature hs, the second embedded feature hd, and the dual task cost hc to output a scalar value s. The classifying unit 412 classifies the cognitive function of the subject SJ into positive or negative based on the scalar value s. For example, the scalar value s can be expressed by the following formula (13).

number

[0114] In equation (13), b denotes the bias term, and W denotes the weight matrix of the fully connected layer (FC).

[0115] The second embodiment of the present invention has been described above with reference to Fig. 9. According to the second embodiment, as in the first embodiment, the cognitive function of the subject SJ can be evaluated more accurately than a system that evaluates the cognitive function of the subject SJ based on limited features.

[0116] Furthermore, according to the second embodiment, the cognitive function assessment system 100 can assess the cognitive function of the subject SJ by using the difference (dual task cost hc) between the first movement feature (first embedded feature hs) and the second movement feature (second embedded feature hd). In other words, the cognitive function of the subject SJ can be assessed by utilizing the difference between the movement feature (first movement feature) of the subject SJ when performing a motor task and the movement feature (second movement feature) of the subject SJ when performing a dual task as a discrimination element. As a result, the cognitive function of the subject SJ can be assessed with even greater accuracy.

[0117] Specifically, the lower the cognitive function of the subject SJ, the larger the characteristic difference (dual task cost hc). This is because the burden on the brain during dual task execution is greater than during motor task execution. For example, if the exercise (first and second exercises) imposed on the subject SJ in the motor task (single task) and dual task is stepping, the lower the cognitive function of the subject SJ, the slower the stepping speed and the more the body balance is lost during dual task execution compared to when the motor task is executed. Therefore, by utilizing the difference between the movement characteristic (first movement characteristic) of the subject SJ during motor task execution and the movement characteristic (second movement characteristic) of the subject SJ during dual task execution as a discrimination factor, the cognitive function of the subject SJ can be evaluated with even greater accuracy.

[0118] In addition, by utilizing the difference between the movement characteristics of the subject SJ when performing a motor task (first movement characteristics) and the movement characteristics of the subject SJ when performing a dual task (second movement characteristics) as a discriminant, it is possible to evaluate the cognitive function of the subject SJ while suppressing the influence of differences in the physical abilities of each subject SJ (individual differences).

[0119] [Embodiment 3] Next, a third embodiment of the present invention will be described with reference to Figures 9 and 10. However, differences from the first and second embodiments will be described, and a description of the same aspects as the first and second embodiments will be omitted. The third embodiment differs from the first and second embodiments in the parameter values ​​of the second neural network NW2. Specifically, the third embodiment differs from the first and second embodiments in the method of determining the parameter values ​​of the second neural network NW2. That is, the third embodiment differs from the first and second embodiments in the training method (learning method) of the second neural network NW2.

[0120] FIG. 10 is a block diagram showing the configuration of the evaluation unit 40 during training of the second neural network NW2 included in the cognitive function evaluation system 100 of this embodiment. As shown in FIG. 10, in the third embodiment, a first loss function L1 and a second loss function L2 are used as loss functions. Therefore, the parameter values ​​of the second neural network NW2 are determined based on the first loss function L1 and the second loss function L2. More specifically, a third loss function Lt that combines the first loss function L1 and the second loss function L2 is used as the loss function. The second loss function L2 indicates a loss function that maximizes the difference (feature difference) between the first movement feature extracted by the first movement feature extraction unit 2a and the second movement feature extracted by the second movement feature extraction unit 2b. In this embodiment, the second loss function L2 maximizes the dual task cost hc.

[0121] In detail, the processing unit 42 executes the learning program TP using the training data (training data for mini-batch learning) acquired from the data collection unit 70 after providing the third loss function Lt and the learning rate to the learning program TP. The training data includes a motor task frame group, a dual task frame group, and label data of each subject. As a result, the neural network constructed by the learning program TP is trained, and a machine learning model ML is generated. Specifically, the processing unit 42 determines the values ​​of a plurality of parameters (parameter values) included in the second neural network NW2 to values ​​that minimize the first loss function L1 and the second loss function L2. For example, the processing unit 42 may determine the parameter values ​​of the second neural network NW2 based on an algorithm of the backpropagation learning method.

[0122] Here, the second loss function L2 will be described. The second loss function L2 maximizes the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task. In this embodiment, the second loss function L2 maximizes the dual task cost hc. The second loss function L2 is expressed, for example, by the following formula (14).

number

[0123] In formula (14), the rows of the transposed determinant Hs represent the first embedded feature hs, and the rows of the determinant Hd represent the second embedded feature hd. As shown in formula (14), when determining the parameter values ​​of the second neural network NW2, the second loss function L2 may impose an orthogonalization constraint on the first embedded feature hs (vector) and the second embedded feature hd (vector). In other words, the second loss function L2 may impose a constraint on the first embedded feature hs (vector) and the second embedded feature hd (vector) such that their inner product is "0". As a result, the second loss function L2 determines the parameter values ​​that maximize the dual task cost hc.

[0124] Next, a description will be given of the third loss function Lt. The third loss function Lt can be expressed, for example, by the following formula (15). Lt = L1 + αL2 (15)

[0125] In equation (15), α is a hyperparameter, which is a coefficient for harmonizing the first loss function L1 and the second loss function L2. By setting the coefficient α to an appropriate value, the machine learning model ML (the second neural network NW2) can be optimized.

[0126] As described above with reference to Fig. 9 and Fig. 10, the training method (learning method) of the second neural network NW2 includes a step of inputting training data including label data indicating whether the subject is positive or negative, a motor task frame group, and a dual task frame group to the second neural network NW2, and adjusting the parameter values ​​of the second neural network NW2 based on the first loss function L1 and the second loss function L2. According to this training method, the sum (P1+P2) of the sensitivity P1 and the specificity P2 is optimized, and the second neural network NW2 is trained so as to maximize the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task. As a result, the second neural network NW2 can evaluate the cognitive function of the evaluation subject SJ with even greater accuracy.

[0127] Specifically, the second loss function L2 maximizes the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task. In other words, the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task is emphasized. As already explained, by utilizing the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task as a discrimination element, the evaluation accuracy of the cognitive function can be improved. Therefore, by emphasizing the difference between the movement feature (first movement feature) of the evaluation subject SJ when performing a motor task and the movement feature (second movement feature) of the evaluation subject SJ when performing a dual task, the evaluation accuracy of the cognitive function can be further improved.

[0128] [Embodiment 4] Next, a fourth embodiment of the present invention will be described with reference to Figures 11 to 13. However, differences from the first to third embodiments will be described, and explanations of the same matters as the first to third embodiments will be omitted. The fourth embodiment differs from the first to third embodiments in that the characteristics of the answers of the subject SJ to the intelligence questions (or cognitive questions) are used to evaluate the cognitive function of the subject SJ.

[0129] Fig. 11 is a diagram showing a cognitive function assessment system 100 of this embodiment. Fig. 12 is a block diagram showing the configuration of the cognitive function assessment system 100 of this embodiment. In detail, Fig. 12 shows the configuration of the evaluation unit 40. As shown in Figs. 11 and 12, the cognitive function assessment system 100 of this embodiment further includes an answer detection unit 30.

[0130] The answer detection unit 30 detects answers to the intelligence questions (or cognitive questions) by the subject SJ who is performing the task set. The answer detection unit 30 includes, for example, an answer switch for the left hand (not shown) and an answer switch for the right hand (not shown).

[0131] In this embodiment, the answer detection unit 30 detects answers to intelligence questions (or cognitive questions) by the subject SJ who is performing a dual task. The processing unit 42 inputs a plurality of frames (consecutive frames) output from the movement detection unit 20 to the machine learning model ML. Furthermore, the processing unit 42 extracts features from the answers detected by the answer detection unit 30, and inputs the extracted features of the answers to the machine learning model ML. In this embodiment, the machine learning model ML evaluates the cognitive function of the subject SJ based on the plurality of frames (consecutive frames) output from the movement detection unit 20 and the answers detected by the answer detection unit 30.

[0132] In detail, the processing unit 42 calculates the answer speed and the correct answer rate as characteristics of the answer based on the answers detected by the answer detection unit 30. The answer speed indicates the number of times the evaluation subject SJ answers per unit time. The unit time is, for example, one second. The correct answer rate indicates the ratio between the number of answers and the number of correct answers. The number of answers indicates the number of times the evaluation subject SJ answers. The number of correct answers indicates the number of correct answers.

[0133] The answering speed is calculated with respect to the dual task performance time. That is, the answering speed is calculated by dividing the number of answers given by the subject SJ when performing the dual task by the dual task performance time. Similarly, the correct answer rate indicates the ratio of the number of answers given by the subject SJ when performing the dual task to the number of correct answers given by the subject SJ when performing the dual task.

[0134] The answer features are not limited to the answer speed and the correct answer rate. For example, the answer features may be only one of the answer speed and the correct answer rate. Alternatively, the answer features may include at least one of the number of answers, the number of correct answers, the average answer time interval, and the standard deviation of the answer time interval, instead of or in addition to the answer speed and the correct answer rate. Here, the answer time interval indicates the time interval required from when the answer candidate presentation screen 13b described with reference to FIG. 2(c) and FIG. 3 is presented by the task presenter 10 until the evaluation subject SJ presses the answer switch. Hereinafter, the answer features acquired by the evaluation subject SJ performing the task set may be referred to as "answer feature data".

[0135] Next, the machine learning model ML will be described with reference to Fig. 13. Fig. 13 is a diagram showing a third neural network NW3 constructed by the machine learning model ML included in the cognitive function assessment system 100 of this embodiment. The third neural network NW3 is, for example, a neural network that performs deep learning.

[0136] 13, the machine learning model ML constructs a third neural network NW3. Therefore, the evaluation unit 40 includes the third neural network NW3. The third neural network NW3 includes a first motion feature extraction unit 2a, a second motion feature extraction unit 2b, a first conversion unit 3a, a second conversion unit 3b, an output unit 4, and a convolution unit 5.

[0137] The convolution unit 5 convolves the answer feature data. The convolution unit 5 includes, for example, a CNN (Convolutional Neural Network). The output unit 4 outputs evaluation data based on the first action feature, the second action feature, and the feature of the answer of the evaluation subject SJ to the intelligence question (or the cognitive question). In this embodiment, the output unit 4 includes a combination unit 411, a classification unit 412, and an acquisition unit 413. The combination unit 411 combines the first embedded feature hs, the second embedded feature hd, the dual task cost hc, and the convolved answer feature data, and outputs a scalar value s.

[0138] When training the third neural network NW3, the data collection unit 70 described with reference to Fig. 6 and Fig. 7 extracts and collects answer features from the output of the answer detection unit 30. Therefore, the training data further includes answer feature data.

[0139] Specifically, the processing unit 73 of the data collection unit 70 associates the label data input from the input unit 71, the consecutive frames (the motor task frame group and the dual task frame group) obtained from the output of the motion detection unit 20, and the answer feature data obtained from the output of the answer detection unit 30 with the subject's identification information, and stores them in the storage unit 72. As a result, the training data is stored in the storage unit 72.

[0140] Above, the fourth embodiment of the present invention has been described with reference to Figs. 11 to 13. According to the fourth embodiment, as in the first to third embodiments, the cognitive function of the subject SJ can be evaluated more accurately than a system that evaluates the cognitive function of the subject SJ based on limited features. Furthermore, according to the fourth embodiment, the cognitive function of the subject SJ can be evaluated using the features of the responses of the subject SJ to the intelligence questions (or cognitive questions) in addition to the features of the movements of the subject SJ. Therefore, the accuracy of the evaluation of the cognitive function can be further improved.

[0141] In addition, when the task set includes an intellectual task (single task), the answer detection unit 30 may further detect the answer of the subject SJ who is performing the intellectual task (single task). In this case, the answer speed is calculated with respect to the total time of the intellectual task performance time and the dual task performance time. That is, the answer speed is calculated by dividing the total value of the number of times the subject SJ answers when performing the intellectual task (single task) and the number of times the subject SJ answers when performing the dual task by the total time of the intellectual task performance time and the dual task performance time. The intellectual task performance time indicates the time for which the subject SJ is made to perform the intellectual task (single task). The length of the intellectual task performance time is arbitrary. The intellectual task performance time is, for example, 30 seconds.

[0142] Similarly, the rate of correct answers indicates the ratio of the total number of times the subject SJ answered questions when performing the intellectual task (single task) and the total number of times the subject SJ answered questions when performing the dual task to the total number of times the subject SJ answered correctly to questions presented when performing the intellectual task (single task) and the total number of times the subject SJ answered correctly to questions presented when performing the dual task.

[0143] In addition, in this embodiment, the output unit 4 of the third neural network NW3 includes the acquisition unit 413, but the acquisition unit 413 may be omitted.

[0144] The embodiments of the present invention have been described above with reference to the drawings (FIGS. 1 to 13). However, the present invention is not limited to the above embodiments, and can be implemented in various aspects without departing from the gist of the present invention. In addition, the components disclosed in the above embodiments can be modified as appropriate. For example, a certain component among all the components shown in one embodiment may be added to a component of another embodiment, or some of all the components shown in one embodiment may be deleted from the embodiment.

[0145] The drawings are mainly schematic illustrations of each component to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. Furthermore, the configurations of each component shown in the above embodiment are merely examples and are not particularly limited, and it goes without saying that various modifications are possible within a range that does not substantially deviate from the effects of the present invention.

[0146] For example, in the embodiment described with reference to Figs. 1 to 13, the output unit 4 includes the classification unit 412, but the classification unit 412 may be omitted. In this case, a scalar value s is output from the machine learning model ML (the first neural network NW1, the second neural network NW2, or the third neural network NW3). The processing unit 42 of the evaluation unit 40 may classify the cognitive function of the evaluation subject SJ based on the scalar value s output from the machine learning model ML.

[0147] In the embodiment described with reference to Fig. 1 to Fig. 13, the cognitive function assessment system 100 classifies the evaluation subject SJ into a dementia class, or a class including mild cognitive impairment and non-dementia, or into a class including dementia and mild cognitive impairment, or into a non-dementia class, but the cognitive function assessment system 100 may classify the evaluation subject SJ into a dementia class, a mild cognitive impairment class, or a non-dementia class. That is, in the embodiment described with reference to Fig. 1 to Fig. 13, the cognitive function assessment system 100 classifies the cognitive function of each evaluation subject SJ into two classes, but the cognitive function assessment system 100 may classify the cognitive function of each evaluation subject SJ into three classes.

[0148] In addition, although the cognitive function assessment system 100 of the present embodiment classifies the cognitive function of each evaluation subject SJ into two classes, the cognitive function assessment system 100 may classify the cognitive function of each evaluation subject SJ into four or more classes. For example, the cognitive function assessment system 100 may classify each evaluation subject SJ by type of dementia. The types of dementia include Alzheimer's dementia, vascular dementia, Lewy body dementia, frontotemporal dementia, normal pressure hydrocephalus, and the like. For example, the cognitive function assessment system 100 may classify the cognitive function of each evaluation subject SJ into an Alzheimer's dementia class, a Lewy body dementia class, a mild cognitive impairment class, and a non-dementia class. In this case, when training (learning) the first neural network NW1, the second neural network NW2, and the third neural network NW3, label data indicating Alzheimer's disease, label data indicating Lewy body dementia, label data indicating mild cognitive impairment, and label data indicating non-dementia are input to the processing unit 73 as label data (objective functions).

[0149] Furthermore, in the embodiment described with reference to FIGS. 1 to 13, the task presenter 10 has a display, but the task presenter 10 may have an audio output device.

[0150] 1 to 13, the cognitive function assessment system 100 has a left-hand answer switch and a right-hand answer switch as a device (answer detection unit 30) for detecting answers to the intelligence questions (or cognitive questions) by the subject SJ and the test subject who are performing the task set, but the device for detecting answers (answer detection unit 30) is not limited to the answer switch. For example, the cognitive function assessment system 100 may have a gaze direction detection device or a sound collector as a device for detecting answers (answer detection unit 30).

[0151] When a gaze direction detection device is used, for example, the answers of the evaluation subject SJ and the subject can be obtained based on the direction in which the evaluation subject SJ and the subject direct their gaze toward the answer candidate presentation screen 13b shown in FIG. 2(c) and FIG. 3. A known gaze direction detection technology can be adopted for the gaze direction detection device. For example, the gaze direction detection device includes a near-infrared LED and an imaging device. The near-infrared LED irradiates near-infrared light onto the eyes of the evaluation subject SJ and the subject. The imaging device captures images of the eyes of the evaluation subject SJ and the subject. The processing unit 42 and the processing unit 73 analyze the images captured by the imaging device to detect the positions of the pupils (the direction of gaze) of the evaluation subject SJ and the subject.

[0152] When a sound collector is used, for example, the answers of the evaluation target SJ and the test subject can be obtained based on the voices generated by the evaluation target SJ and the test subject in response to the answer candidate presentation screen 13b shown in Fig. 2(c) and Fig. 3. For example, the processing unit 42 and the processing unit 73 can obtain the answers of the evaluation target SJ and the test subject by converting the voices generated by the evaluation target SJ and the test subject into text data by speech recognition processing.

[0153] When a sound collector is used, the cognitive questions are not limited to questions that require the evaluation subject SJ and the test subject to select one of two possible answers. For example, the evaluation subject SJ and the test subject may be asked to answer a calculation question. When a sound collector is used, the cognitive questions may be questions that require the user to answer a word. Questions that require the user to answer a word may be, for example, "word chain questions," "questions that require the user to list words (e.g., words) that start with a sound (letter) arbitrarily selected from the Japanese alphabet," or "questions that require the user to list words (e.g., words) that start with a letter arbitrarily selected from the alphabet."

[0154] In the embodiment described with reference to FIGS. 1 to 13, the training data is generated using the data collecting section 70, but the evaluation section 40 may generate the training data. [Industrial Applicability]

[0155] The present invention can be used to diagnose dementia and mild cognitive impairment. [Explanation of symbols]

[0156] 2a: First motion feature extraction unit 2b: Second motion feature extraction unit 4: Output section 20: Motion detection unit 40: Evaluation section 100: Cognitive function assessment system 413: Acquisition Department L1: First loss function L2: Second loss function ML: Machine learning model NW1: First neural network NW2: Second neural network NW3: Third Neural Network SJ: Evaluator hc : Dual task cost hd : second embedded feature hs : first embedded feature

Claims

1. A cognitive function assessment system for evaluating the cognitive function of a subject, a motion detection unit that captures an image of the subject performing a motor task and a dual task, and generates a plurality of frames in a time series that indicate three-dimensional coordinates of a plurality of joints of the subject obtained from the image capture; an evaluation unit that evaluates the cognitive function of the evaluation subject based on the plurality of frames; Equipped with The evaluation unit has a machine learning model that constructs a neural network, The neural network comprises: a first movement feature extraction unit that extracts first movement features indicating a feature of a spatial positional relationship of the plurality of joints and a feature of a temporal change of the plurality of joints based on a movement task frame group, which is a plurality of frames corresponding to the movement task, among the plurality of frames; a second movement feature extraction unit that extracts second movement features indicating features of spatial positional relationships of the plurality of joints and features of temporal fluctuations of the plurality of joints based on a dual task frame group, which is a plurality of frames corresponding to the dual task, among the plurality of frames; an output unit that outputs evaluation data indicating a cognitive function of the evaluation subject based on the first action feature and the second action feature; Including, The exercise task indicates a task of imposing a first exercise on the subject, The dual task refers to a task in which the subject is simultaneously required to perform a second movement and answer an intelligence question, in this cognitive function assessment system.

2. The cognitive function evaluation system according to claim 1 , wherein the output unit outputs data classifying the subject as positive or negative as the evaluation data.

3. The cognitive function assessment system of claim 2, wherein the output unit outputs, as the assessment data, data classifying the subject into a dementia class or a class including mild cognitive impairment and non-dementia, or outputs data classifying the subject into a class including dementia and mild cognitive impairment, or a non-dementia class.

4. The cognitive function assessment system according to claim 2 or claim 3, wherein the parameter values ​​of the neural network are determined based on a first loss function that optimizes the sum of sensitivity, which indicates the proportion of correctly recognizing positives, and specificity, which indicates the proportion of correctly recognizing negatives.

5. 4. The cognitive function assessment system according to claim 1, wherein the output unit includes an acquisition unit that acquires a feature difference indicating a difference between the first action feature and the second action feature, and outputs the assessment data based on the first action feature, the second action feature, and the feature difference.

6. the output unit includes an acquisition unit that acquires a feature difference indicating a difference between the first action feature and the second action feature, and outputs the evaluation data based on the first action feature, the second action feature, and the feature difference; The cognitive function assessment system according to claim 4 , wherein parameter values ​​of the neural network are determined based on the first loss function and a second loss function that maximizes the feature difference.

7. the first movement feature extraction unit generates a spatial graph indicating a spatial positional relationship of the plurality of joints for each frame and convolves the spatial graph, and generates a time graph indicating a fluctuation of the same joint between adjacent frames and convolves the time graph, thereby extracting the first movement feature; 4. The cognitive function assessment system according to claim 1, wherein the second movement feature extraction unit extracts the second movement feature by generating a spatial graph indicating a spatial positional relationship of the plurality of joints for each frame and convolving the spatial graph, and by generating a time graph indicating a fluctuation of the same joint between adjacent frames and convolving the time graph.

8. The cognitive function assessment system according to claim 1 , wherein the first exercise and the second exercise include the same exercise.

Images