Cognitive function evaluation system

Abstract

A cognitive function evaluation system (100) includes an imaging unit (10), a generation unit (30), a preprocessing unit (40), and an evaluation unit (60). The imaging unit (10) generates an imaging signal by imaging an evaluation subject (SJ) who is performing a dual task, which is a task in which a cognitive problem and a movement problem are simultaneously given. The generation unit (30) generates time-series data of a skeleton, which is a skeleton of the evaluation subject (SJ) who is performing the dual task, and which represents at least a part of the entire body of the evaluation subject (SJ), based on the imaging signal. The preprocessing unit (40) generates skeleton data divided by period by decomposing the time-series data of the skeleton by period, and generates input data by aligning the phases of the skeleton data divided by period. The evaluation unit (60) extracts a feature of a movement of the evaluation subject (SJ) based on the input data, and evaluates the cognitive function of the evaluation subject (SJ) based on the extracted feature of the movement of the evaluation subject (SJ).

Description

Technical Field

[0001] This invention relates to a cognitive function assessment system. Background Technology

[0002] For the assessment of cognitive function, paper-and-pencil tests such as the MMSE (Mini-Mental State Examination) are commonly used. However, the questions in these tests are fixed. Therefore, if these tests are used routinely, the test-taker may remember the content of the questions, resulting in an inaccurate assessment of their cognitive function. Therefore, there is a need for a technology that can routinely assess the cognitive function of test-takers.

[0003] As one technique for evaluating cognitive function, there exists a dual-task technique that simultaneously assigns two tasks to the person being evaluated (e.g., non-patent literature 1, 2). If dual-task evaluation of cognitive function is possible, then the cognitive function of the person being evaluated can be assessed routinely.

[0004] Existing technical documents Non-patent literature Non-patent literature 1: H.Makizako et al., "Relationship between dual-taskperformance and neurocognitive measures in older adults with mild cognitiveimpairment", Geriatr.Gerontol.Int., pp.314-321, 2013. Non-patent literature 2: K.Aoki et al., "Early detection of lower mmse scores inelderly based on dual-task gait", IEEE Access, vol.7, pp.40085-40094, 2019. Summary of the Invention

[0005] The problem that the invention aims to solve The inventors of this invention have conducted in-depth research on techniques for evaluating cognitive function using dual tasks, thus completing this invention. The purpose of this invention is to provide a cognitive function evaluation system that can evaluate the cognitive function of the evaluated individual using dual tasks.

[0006] Methods for solving problems In one aspect of the present invention, a cognitive function evaluation system is a system for evaluating the cognitive function of a subject. This cognitive function evaluation system includes: a camera unit, a generation unit, a preprocessing unit, and an evaluation unit. The camera unit captures images of the subject performing a dual task, which is a task simultaneously assigning a cognitive task and a motor task. The generation unit generates time-series data of a skeleton based on the camera signals. The skeleton is the skeleton of the subject performing the dual task and represents at least a portion of the subject's entire body. The preprocessing unit decomposes the time-series data of the skeleton periodically to generate periodically separated skeleton data and aligns the phases of the periodically separated skeleton data to generate input data. The evaluation unit extracts features of the subject's movements based on the input data and evaluates the subject's cognitive function based on the extracted features of the subject's movements.

[0007] In one embodiment, the time-series data of the skeleton represents the time-series data of each of a plurality of joints constituting the skeleton. The plurality of joints represents at least a portion of all joints constituting the skeleton. The preprocessing unit decomposes the time-series data of the joints by period to generate periodically separated joint data, and aligns the phases of the periodically separated joint data to generate the input data.

[0008] In one embodiment, the preprocessing unit samples data from a certain number of identical phase points from each of the periodically separated joint point data to generate the input data.

[0009] In one embodiment, the preprocessing unit selects a certain number of consecutive periodically separated joint point data with the highest correlation from the periodically separated joint point data according to the joint point to generate the input data.

[0010] In one implementation, the input data represents the phase-aligned, periodically separated keypoint data. The phase-aligned, periodically separated keypoint data represents spatial dimension data, phase dimension data, and period dimension data. The evaluation unit extracts features of the evaluated subject's actions by performing convolutions on the spatial dimension data, the phase dimension data, and the period dimension data, respectively.

[0011] In one embodiment, the evaluation unit includes a first convolutional layer and a second convolutional layer. The first convolutional layer convolves the phase dimension data included in the input data. The second convolutional layer extracts features of the evaluated person's actions by convolving the spatial dimension data, the phase dimension data, and the periodic dimension data included in the input data obtained by convolving the phase dimension data by the first convolutional layer.

[0012] In one embodiment, the evaluation unit further includes a multiplier. The multiplier multiplies the adjacency matrix representing the positional relationships of the plurality of key points by the output of the first convolutional layer. The output of the multiplier is then input to the second convolutional layer.

[0013] In one embodiment, the evaluation unit further includes a residual neural network that performs calculations on the difference between the output of the second convolutional layer and the input data.

[0014] In one embodiment, the evaluation unit includes a plurality of convolutional blocks connected in series. Each of the plurality of convolutional blocks performs convolution on the data in the spatial dimension, the data in the phase dimension, and the data in the period dimension.

[0015] In one embodiment, the cognitive function evaluation system further includes a response detection unit for detecting the responses made by the evaluated subject to a cognitive task. The response detection unit detects at least one of the responses of the evaluated subject performing the dual task and the responses of the evaluated subject performing the cognitive task assigned to them. The evaluation unit further extracts features of the responses detected by the response detection unit and evaluates the evaluated subject's cognitive function based on the features of the evaluated subject's actions and the features of the responses.

[0016] Invention Effects The cognitive function evaluation system according to the present invention can evaluate the cognitive function of the evaluated person by using a dual task. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating a cognitive function evaluation system according to an embodiment of the present invention.

[0018] Figure 2 This is a block diagram illustrating the hardware configuration of a cognitive function evaluation system according to an embodiment of the present invention.

[0019] Figure 3Figures (a) to (d) are illustrations showing an example of a task presented to an evaluator by a task presentation unit included in a cognitive function evaluation system according to an embodiment of the present invention.

[0020] Figure 4 This is a diagram illustrating another example of a cognitive task presented to a person being evaluated by a task presentation unit included in a cognitive function evaluation system according to an embodiment of the present invention.

[0021] Figure 5 (a) is a diagram showing the skeleton generated by the skeleton generation unit included in the cognitive function evaluation system according to an embodiment of the present invention. Figure 5 (b) is a diagram showing the normalization process performed by the skeleton generation unit.

[0022] Figure 6 This is a diagram showing an example of time-series data of the movement of a joint corresponding to the right knee of an evaluated subject performing stationary steps, and an example of multiple periodically separated joint data after phase alignment.

[0023] Figure 7 This is a diagram showing an example of a pair of stacked absolute correlation matrices and a sliding window.

[0024] Figure 8 This diagram illustrates the structure of the evaluation unit included in the cognitive function evaluation system according to an embodiment of the present invention.

[0025] Figure 9 This is a diagram illustrating the neural network of the action feature extractor included in the cognitive function evaluation system according to an embodiment of the present invention.

[0026] Figure 10 This is a schematic diagram illustrating spatial convolution processing, phase convolution processing, and periodic convolution processing.

[0027] Figure 11 This is a diagram illustrating the neural network of the first graph convolutional block included in the cognitive function evaluation system according to an embodiment of the present invention.

[0028] Figure 12 This is a diagram illustrating the neural network of the response feature extractor included in the cognitive function evaluation system according to an embodiment of the present invention.

[0029] Figure 13 This is a diagram illustrating the neural network of the fusion unit included in the cognitive function evaluation system according to an embodiment of the present invention. Detailed Implementation

[0030] The following refers to the attached diagram ( Figures 1 to 13This section describes the implementation methods of the cognitive function evaluation system of the present invention. However, the present invention is not limited to the following embodiments and can be implemented in various ways without departing from its spirit. Furthermore, descriptions of repeated parts are sometimes appropriately omitted. Additionally, in the accompanying drawings, the same or corresponding parts are labeled with the same reference numerals and are not described repeatedly.

[0031] Figure 1 This is a diagram illustrating the cognitive function assessment system 100 of this embodiment. The cognitive function assessment system 100 evaluates the cognitive function of the person being assessed, SJ. More specifically, the cognitive function assessment system 100 of this embodiment classifies the cognitive function of the person being assessed, SJ, into: a category of dementia or cognitive impairment, and a category that includes mild cognitive impairment (MCI) and non-dementia. Alternatively, the cognitive function assessment system 100 of this embodiment classifies the cognitive function of the person being assessed, SJ, into: a category that includes one of dementia and cognitive impairment, a category that includes mild cognitive impairment, and a category that includes non-dementia. Hereinafter, dementia or cognitive impairment will sometimes be referred to as "dementia".

[0032] In addition, mild cognitive impairment is a pre-dementia stage, representing an intermediate state between normal and dementia. Specifically, mild cognitive impairment indicates the following state: although cognitive functions such as memory and attention decline, cognitive functions have not yet declined to the point of hindering daily life.

[0033] like Figure 1 As shown, the cognitive function evaluation system 100 includes a camera unit 10, a response detection unit 20, and an information processing unit 100a.

[0034] Camera unit 10 records footage of the subject SJ performing a dual task to generate video signals. The dual task involves simultaneously presenting the subject SJ with a cognitive task and a motor task. The cognitive task requires the subject SJ to answer cognitive questions, such as calculation problems, positional memory problems, or rock-paper-scissors problems. The motor task includes activities such as marching in place, walking, running, or hopping. The video signals are used to extract features of the subject SJ's movements.

[0035] Specifically, the camera unit 10 captures images of the evaluated subject SJ to generate video images. For example, the camera unit 10 generates video data or recorded data. The camera unit 10 may, for example, have a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. For example, the camera unit 10 may be a video camera or an RGB camera. Alternatively, the camera unit 10 may also be a camera mounted on a smartphone.

[0036] For example, the camera unit 10 is positioned in front of the person being evaluated, SJ. By positioning the camera unit 10 in front of the person being evaluated, SJ's movement characteristics can be extracted more reliably.

[0037] In this embodiment, the camera unit 10 further films the person being evaluated, SJ, who is performing a sports task (single task). The sports task is a task that assigns a sports assignment to the person being evaluated, SJ.

[0038] In detail, in this embodiment, the person being evaluated, SJ, performs these tasks sequentially in the order of cognitive task (single task), motor task (single task), and dual task. The cognitive task refers to a task that requires the person being evaluated, SJ, to answer a cognitive question. The camera unit 10 films the person being evaluated, SJ, while he / she performs the motor task. Furthermore, the camera unit 10 films the person being evaluated, SJ, while he / she performs the dual task. Hereinafter, a set of tasks performed by the person being evaluated, SJ, will sometimes be referred to as a "task set."

[0039] More specifically, the person being evaluated, SJ, performs a cognitive task for a specified time (e.g., 30 seconds). After performing the cognitive task, SJ performs a motor task for a specified time (e.g., 20 seconds). Then, after the motor task, SJ performs a dual task for a specified time (e.g., 30 seconds). Hereinafter, the time SJ spends performing the cognitive task will sometimes be recorded as "Cognitive Task Execution Time." Similarly, the time SJ spends performing the motor task will sometimes be recorded as "Motor Task Execution Time," and the time SJ spends performing the dual task will sometimes be recorded as "Dual Task Execution Time." The length of the cognitive task execution time can be arbitrarily set (determined). Similarly, the lengths of the motor task execution time and the dual task execution time can also be arbitrarily set (determined).

[0040] Furthermore, the motion task and the motion task of the dual task can be the same or different. Similarly, the cognitive task and the cognitive task of the dual task can be the same or different. In this embodiment, the motion task and the motion task of the dual task are the same. Similarly, the cognitive task and the cognitive task of the dual task are the same.

[0041] The response detection unit 20 detects the responses of the evaluated subject SJ to a cognitive task and generates a detection signal. In this embodiment, the response detection unit 20 detects the responses of the evaluated subject SJ who is performing a cognitive task, as well as the responses of the evaluated subject SJ who is performing a dual task. For example, the response detection unit 20 may have a response switch for the left hand (not shown) and a response switch for the right hand (not shown). The detection signal is used to extract features of the responses made by the evaluated subject SJ.

[0042] The camera signal is input from the camera unit 10 to the information processing unit 100a. The information processing unit 100a processes the camera signal, extracts the features of the evaluated person SJ's actions, and evaluates the evaluated person SJ's cognitive function based on the extracted features of the evaluated person SJ's actions. The information processing unit 100a can be configured as, for example, a computer such as a personal computer.

[0043] In this embodiment, the detection signal is further input from the response detection unit 20 to the information processing unit 100a. The information processing unit 100a processes the detection signal and extracts features of the respondent SJ's response to the cognitive task. Then, the information processing unit 100a evaluates the respondent SJ's cognitive function based on the features of the respondent SJ's actions and the features of the respondent SJ's response to the cognitive task.

[0044] More specifically, the information processing unit 100a functions as the skeleton generation unit 30, the preprocessing unit 40, the response calculation unit 50, and the evaluation unit 60.

[0045] The skeleton generation unit 30 generates time-series data of the skeleton SK representing the subject SJ performing a dual task, based on the camera signal generated by the camera unit 10. In other words, the skeleton generation unit 30 generates a skeleton sequence representing the subject SJ performing a dual task. The time-series data of the skeleton SK (skeleton sequence) represents the actions of the subject SJ. The skeleton generation unit 30 is an example of a "generation unit".

[0046] In this embodiment, the skeleton generation unit 30 generates a skeleton sequence (time series data of skeleton SK) representing the actions of the evaluated subject SJ after performing a cognitive task (the actions of the evaluated subject SJ who perform these tasks sequentially in the order of motor task and dual task).

[0047] The preprocessing unit 40 detects the period of the time series data of the skeleton SK. Then, the preprocessing unit 40 decomposes the time series data of the skeleton SK by each period to generate multiple periodically separated skeleton data SKDs, and aligns the phases of the periodically separated skeleton data SKDs to generate first input data. Therefore, the first input data represents the multiple periodically separated skeleton data SKDs after phase alignment. Each periodically separated skeleton data SKD represents multiple skeleton SKs arranged along the phase dimension. More specifically, each periodically separated skeleton data SKD represents the skeleton SK at a specific phase point.

[0048] In other words, the preprocessing unit 40 decomposes the skeleton sequence into multiple periodically separated skeleton sequences by each period, and aligns the phases of each periodically separated skeleton sequence to generate the first input data. Therefore, the first input data represents the multiple periodically separated skeleton sequences after phase alignment. Each periodically separated skeleton sequence represents the movement of the skeleton SK along the phase dimension.

[0049] In detail, the actions of the subject SJ performing a motor task are periodic. For example, if the motor task is any of the following: marching in place, walking, running, or hopping, the actions of the subject SJ performing the motor task are quasi-periodic. Similarly, the actions of the subject SJ performing a dual task are also periodic. Therefore, the preprocessing unit 40 is able to detect the periodicity of the time series data of the skeleton SK.

[0050] The response calculation unit 50 generates second input data based on the detection signal generated by the response detection unit 20. The second input data, for example, represents response speed and accuracy. Response speed represents the number of times the evaluated subject SJ responds per unit time. The unit time is, for example, 1 second. Accuracy represents the ratio of the number of responses to the number of correct answers. The number of responses represents the number of times the evaluated subject SJ responds. The number of correct answers represents the number of correct responses.

[0051] In this embodiment, the response speed includes a first response speed and a second response speed. The first response speed is calculated by dividing the number of responses made by the appraisee SJ during the cognitive task by the cognitive task execution time. The second response speed is calculated by dividing the number of responses made by the appraisee SJ during the dual task by the dual task execution time.

[0052] Similarly, accuracy includes a first accuracy rate and a second accuracy rate. The first accuracy rate represents the ratio of the number of times the subject SJ responded to a question presented during the cognitive task to the number of times the subject SJ answered correctly. The second accuracy rate represents the ratio of the number of times the subject SJ responded to a question presented during the dual task to the number of times the subject SJ answered correctly.

[0053] The evaluation unit 60 extracts features of the actions of the person being evaluated, SJ, based on the data input from the preprocessing unit 40 (first input data). Then, the evaluation unit 60 evaluates the cognitive function of the person being evaluated, SJ, based on the extracted features.

[0054] In this embodiment, the evaluation unit 60 extracts features of the actions of the evaluated subject SJ after performing the cognitive task (the actions of the evaluated subject SJ who performed the tasks sequentially in the order of the motor task and the dual task) based on the first input data. Furthermore, the evaluation unit 60 further extracts features of the evaluated subject SJ's responses to the cognitive task based on the data (second data) input from the response calculation unit 50. Then, the evaluation unit 60 evaluates the cognitive function of the evaluated subject SJ based on the features of the evaluated subject SJ's actions and the features of the evaluated subject SJ's responses.

[0055] Next, refer to Figure 2 This embodiment describes the cognitive function evaluation system 100. Figure 2 This is a block diagram illustrating the hardware configuration of the cognitive function evaluation system 100 according to this embodiment. Figure 2 As shown, the cognitive function evaluation system 100 also includes a task presentation unit 70, a recognition information acquisition unit 80, and an evaluation result presentation unit 90. Furthermore, the information processing unit 100a includes a processing unit 101 and a storage unit 102.

[0056] The task presentation unit 70 presents a task for the person being evaluated, SJ, to perform. In this embodiment, the task presentation unit 70 may be a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The task presentation unit 70 displays a screen showing the task for the person being evaluated, SJ. This screen may be positioned, for example, in front of the person being evaluated, SJ.

[0057] For each participant SJ using the cognitive function assessment system 100, inherent identification information is pre-assigned. The identification information acquisition unit 80 acquires the identification information assigned to each participant SJ. The identification information acquisition unit 80 may include, for example, a card reader, a keyboard, or a touch panel. When the identification information acquisition unit 80 includes a card reader, each participant SJ allows the card reader to read the identification information carried on the card. When the identification information acquisition unit 80 includes a keyboard or a touch panel, each participant SJ inputs the identification information assigned to them by operating the keyboard or touch panel.

[0058] Storage unit 102 stores various computer programs and various data. Specifically, storage unit 102 has a storage device. This storage device includes, for example, a semiconductor memory (semiconductor storage device). As a semiconductor memory, storage unit 102 may also include, for example, ROM (Read Only Memory) and RAM (Random Access Memory). As a semiconductor memory, storage unit 102 may further include VRAM (Video RAM). In addition, storage unit 102 may also have a storage device. As a storage device, storage unit 102 may also include, for example, at least one of HDD (Hard Disk Drive) and SSD (Solid State Drive). Storage unit 102 may further include a removable medium.

[0059] In this embodiment, the storage unit 102 stores the machine learning model ML. The machine learning model ML is used as a reference. Figure 1 The evaluation unit 60 is a computer program that performs its functions, and is based on the preprocessing unit 40 (see reference). Figure 1 The data output (first input data) is used to output evaluation data representing the evaluation results of the cognitive function of the evaluated subject SJ. In this embodiment, the machine learning model ML is based on the data from the preprocessing unit 40 (refer to...). Figure 1 The output data (first input data) and the data from the response calculation unit 50 (refer to) Figure 1 The output data (second input data) is used to output evaluation data.

[0060] Additionally, the explanatory variables for the multiple training data used to enable the machine learning model (ML) to learn are the first and second input data collected by having multiple subjects perform a set of tasks. Furthermore, the objective variable for the multiple training data is the labeled data (positive answer labels) assigned to multiple subjects.

[0061] Specifically, in generating a machine learning model (ML) to classify the cognitive function of the person being evaluated (SJ) into categories including dementia and mild cognitive impairment (MCI) and non-dementia, the label data indicates whether the subject belongs to the dementia category (positive) or the category including MCI and non-dementia (negative). Similarly, in generating a machine learning model (ML) to classify the cognitive function of the person being evaluated (SJ) into categories including dementia and MCI and non-dementia, the label data indicates whether the subject belongs to the category including dementia and MCI (positive) or the category non-dementia (negative). For example, the label data could be based on a physician's diagnosis.

[0062] The processing unit 101 performs various processes, such as numerical calculations, information processing, and device control, by executing various computer programs stored in the storage unit 102. Specifically, the processing unit 101 has a processor. This processor may include, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit). Alternatively, the processing unit 101 may also have a quantum computer.

[0063] In detail, the processing unit 101 executes various computer programs stored in the storage unit 102 as... Figure 1 The skeleton generation unit 30, preprocessing unit 40, response calculation unit 50, and evaluation unit 60 shown in the figure function as such.

[0064] Specifically, when the processing unit 101 acquires identification information through the identification information acquisition unit 80, it controls the task presentation unit 70 to present a task for the person being evaluated, SJ, to perform. Then, based on the video signal output from the camera unit 10 and the detection signal output from the response detection unit 20, the processing unit 101 acquires the evaluation result of the person being evaluated, SJ's, cognitive function.

[0065] In this embodiment, the evaluation results include a classification of cognitive function. The classification results show the category into which the evaluated individual SJ's cognitive function is categorized. Specifically, the categories into which the evaluated individual SJ's cognitive function is categorized include two categories. One of the two categories represents the dementia category. The other of the two categories represents the category that includes mild cognitive impairment and non-dementia. Alternatively, one of the two categories represents the category that includes both dementia and mild cognitive impairment. The other of the two categories represents the non-dementia category.

[0066] When processing unit 101 obtains the evaluation results of the cognitive function of the evaluated subject SJ, it establishes an association between the evaluation results of the cognitive function of the evaluated subject SJ and the identification information assigned to the evaluated subject SJ, and stores it in storage unit 102. As a result, the historical record of the evaluation results of cognitive function is stored in storage unit 102 in association with the identification information.

[0067] The evaluation result presentation unit 90, controlled by the processing unit 101, presents the evaluation results of cognitive function or a historical record of the evaluation results of cognitive function to the evaluated subject SJ. The evaluation result presentation unit 90 may include, for example, a printer. Alternatively, the evaluation result presentation unit 90 may also include a communication device.

[0068] When the cognitive function assessment system 100 includes a printer as an assessment result presentation unit 90, the processing unit 101 controls the printer to output paper containing the assessment results of cognitive function. Alternatively, the processing unit 101 creates a table or graph showing the historical record of the cognitive function assessment results and outputs paper containing such a table or graph. When the cognitive function assessment system 100 includes a communication device as an assessment result presentation unit 90, the processing unit 101 controls the communication device, for example, to send an email containing the assessment results of the assessed person SJ's cognitive function to an email address pre-registered by the assessed person SJ. Alternatively, the processing unit 101 may also send an email containing a table or graph showing the historical record of the cognitive function assessment results.

[0069] Next, refer to Figure 3 (a) to Figure 3 (d) illustrates an example of a task presented by the task presentation department 70 to the evaluated person SJ. Figure 3 (a) to Figure 3 (d) is a diagram illustrating an example of a task presented by the task presentation unit 70 included in the cognitive function evaluation system 100 of this embodiment to the evaluated subject SJ. Figure 3 (a) to Figure 3 In the example shown in (d), the physical task given to the evaluated subject SJ is "standing still", and the cognitive task (cognitive problem) given to the evaluated subject SJ is "calculation problem".

[0070] like Figure 3 As shown in (a), the task presentation unit 70, controlled by the processing unit 101, displays a first notification screen 11 on the monitor to notify the evaluated subject SJ that the task has begun. Next, as... Figure 3As shown in (b), the task presentation unit 70 is controlled by the processing unit 101 and presents a cognitive task. Specifically, the task presentation unit 70 displays a problem presentation screen 12a on the display showing a calculation problem (cognitive task). For example, the task presentation unit 70 displays a subtraction problem on the display as a calculation problem.

[0071] In addition, Figure 3 In the example shown in (b), the calculation problem is a subtraction problem, but calculation problems are not limited to subtraction problems. Calculation problems can also be addition problems. Alternatively, calculation problems can include both subtraction and addition problems.

[0072] The task presentation unit 70, controlled by the processing unit 101, terminates the display of the cognitive task (calculation problem) at a predetermined time. Specifically, it removes the problem presentation screen 12a from the display. After the problem presentation screen 12a disappears from the display, the task presentation unit 70, controlled by the processing unit 101, displays an answer candidate presentation screen 12b showing two answer candidates on the display. The person being evaluated, SJ, presses one of the two answer switches included in the answer detection unit 20 to select an answer.

[0073] After the person being evaluated, SJ, selects an answer, the task presentation unit 70, controlled by the processing unit 101, displays the next calculation problem (the next cognitive task) on the screen. Specifically, the next calculation problem displayed on the screen is different from the previous one. Thereafter, until the prescribed cognitive task execution time has elapsed, the calculation problems (cognitive tasks) that the person being evaluated, SJ, needs to answer are repeatedly presented.

[0074] like Figure 3 As shown in (c), after the prescribed cognitive task execution time has elapsed, the task presentation unit 70 presents a motor task to the person being evaluated, SJ. Specifically, the task presentation unit 70 is controlled by the processing unit 101 and displays a motor presentation screen 13 on the monitor, which shows the motor task (stepping) given to the person being evaluated, SJ.

[0075] In this embodiment, the exercise task assigned to the evaluated subject SJ in the dual-task scenario is the same as the exercise task assigned to the evaluated subject SJ in the physical task scenario. Furthermore, the cognitive task assigned to the evaluated subject SJ in the dual-task scenario is the same as the cognitive task assigned to the evaluated subject SJ in the cognitive task scenario. Therefore, after the prescribed physical task execution time has elapsed, the task presentation unit 70, as referred to... Figure 3 As described in (b), computational questions (cognitive tasks) that the evaluator SJ needs to answer are repeatedly presented until the prescribed dual task execution time has elapsed.

[0076] like Figure 3As shown in (d), after the prescribed dual task execution time has elapsed, the task presentation unit 70 is controlled by the processing unit 101 to display a second notification screen 14 on the monitor. The second notification screen 14 is a screen that notifies the evaluated person SJ that the task has ended.

[0077] Furthermore, the cognitive tasks presented by the task presentation unit 70 are not limited to calculation problems. For example, cognitive tasks could also be spatial memory problems or rock-paper-scissors problems.

[0078] Figure 4 This diagram illustrates another example of a cognitive task presented by the task presentation unit 70 included in the cognitive function evaluation system 100 of this embodiment to the person being evaluated, SJ. Figure 4 In the example shown, the cognitive topic is "location memory problem". Figure 4 As shown, when the cognitive task is a location memory problem, the task presentation unit 70 is controlled by the processing unit 101 and displays a problem presentation screen 12a on the display. The problem presentation screen 12a is a screen in which a graphic is configured in one of the four areas of configurable graphics.

[0079] The task presentation unit 70, controlled by the processing unit 101, displays an answer candidate presentation screen 12b on the display after the question presentation screen 12a is removed from the display. The answer candidate presentation screen 12b is a screen where graphics are arranged in one of the four configurable graphic areas. In the answer candidate presentation screen 12b, the two answer candidates, "Yes" and "No," are displayed along with the question text. The question text indicates a question that can be answered with "Yes" or "No." Here, the person being evaluated, SJ, is asked whether the position of the graphics is the same between the question presentation screen 12a and the answer candidate presentation screen 12b.

[0080] If the person being evaluated, SJ, determines that the graphic is positioned in the same location on both the question display screen 12a and the answer candidate display screen 12b, then SJ presses the answer switch on the left hand and selects "Yes". Alternatively, if the person being evaluated, SJ, determines that the graphic is positioned in a different location on both the question display screen 12a and the answer candidate display screen 12b, then SJ presses the answer switch on the right hand and selects "No".

[0081] Next, the "rock-paper-scissors problem" is explained. When the cognitive task given to the examinee SJ is a "rock-paper-scissors problem," for example, the task presentation unit 70 displays one of "rock," "scissors," and "paper" in the question presentation screen 12a. Then, in the answer candidate presentation screen 12b, one of "rock," "scissors," and "paper," the question text, and "yes" and "no" are displayed. In the answer candidate presentation screen 12b, for example, the examinee SJ is asked whether the gesture displayed in the answer candidate presentation screen 12b can defeat the gesture displayed in the question presentation screen 12a.

[0082] If the person being evaluated, SJ, determines that the gesture displayed on the answer candidate display screen 12b is capable of defeating the gesture displayed on the question display screen 12a, then the person being evaluated, SJ, presses the answer switch on the left hand and selects "Yes". Alternatively, if the person being evaluated, SJ, determines that the gesture displayed on the answer candidate display screen 12b is not capable of defeating the gesture displayed on the question display screen 12a, then the person being evaluated, SJ, presses the answer switch on the right hand and selects "No".

[0083] Next, refer to Figure 1 and Figure 5 (a) is used to explain the skeleton generation part 30. Figure 5 (a) is a diagram showing the skeleton SK generated by the skeleton generation unit 30 included in the cognitive function evaluation system 100 of this embodiment. As described above, the skeleton generation unit 30 generates time-series data (skeleton sequence) of the skeleton SK based on the camera signal generated by the camera unit 10.

[0084] For example, the skeleton generation unit 30 (processing unit 101) can also process each frame constituting the video data or recorded data using a machine learning model to generate a skeleton SK for each frame. The algorithm of the machine learning model can be, for example, RTMpose. More specifically, the skeleton generation unit 30 can also resample the video data or recorded data at 10 frames per second and generate the skeleton SK for each resampled frame.

[0085] like Figure 5 As shown in (a), in this embodiment, the skeleton generation unit 30 generates a two-dimensional skeleton SK per frame. The skeleton SK represents a graph structure containing multiple joints J. Therefore, the time-series data (skeleton sequence) of the skeleton SK represents the time-series data of each of the multiple joints J constituting the skeleton SK.

[0086] In this embodiment, the skeleton SK represents the whole body of the subject SJ. Figure 5 The skeleton SK shown in (a) includes 17 joints J. Furthermore, the joints J constituting the skeleton SK may not correspond to joints in the human body. For example, Figure 5The skeleton SK shown in (a) includes a first joint J1, a second joint J2, a third joint J3, a fourth joint J4, and a fifth joint J5. The first joint J1 corresponds to the right ear of the person being evaluated, SJ. The second joint J2 corresponds to the right eye of the person being evaluated, SJ. The third joint J3 corresponds to the nose of the person being evaluated, SJ. The fourth joint J4 corresponds to the left eye of the person being evaluated, SJ. The fifth joint J5 corresponds to the left ear of the person being evaluated, SJ.

[0087] Next, refer to Figure 1 and Figure 5 (b) is used to explain the skeleton generation part 30. Figure 5 Figure (b) illustrates the normalization process performed by the skeleton generation unit 30. In this embodiment, the skeleton generation unit 30 normalizes the skeleton SK of each frame to generate time-series data (skeleton sequence) of the skeleton SK. By normalizing the skeleton SK, the influence of individual physical differences on the evaluation results of cognitive function can be avoided. Furthermore, by normalizing the skeleton SK, the influence of the location where the evaluated subject SJ performs the task on the evaluation results of cognitive function can be avoided.

[0088] Specifically, such as Figure 5 As shown in (b), the skeleton generation unit 30 scales the skeleton SK so that the length of the spine is a unit length. Furthermore, the skeleton generation unit 30 moves the skeleton SK so that the hip joint of the skeleton SK is located at the origin of the xy coordinate system. Here, the length of the spine represents the length between the hip joint and the cervical joint. Additionally, the hip joint and cervical joint may not be part of the joint points J (here, 17 joint points J) of the skeleton SK. For example, the skeleton generation unit 30 may calculate the position of the cervical joint based on the average position of the left and right shoulder joints. Furthermore, the skeleton generation unit 30 may calculate the position of the hip joint based on the average position of the pelvis.

[0089] Next, refer to Figure 1 and Figure 6 The preprocessing unit 40 will now be explained. In this embodiment, the preprocessing unit 40 detects the period of the time series data for each joint point J. Then, the preprocessing unit 40 decomposes the time series data for each joint point J into multiple periodically separated joint point data for each joint point J. Then, the preprocessing unit 40 aligns the phases of the multiple periodically separated joint point data for each joint point J to generate first input data. Therefore, the first input data represents the multiple periodically separated joint point data after phase alignment for each joint point J. Each periodically separated joint point data represents the movement of the corresponding joint point J along the phase dimension.

[0090] For example, the preprocessing unit 40 may decompose the time-series data of each joint J constituting the skeleton SK into periods, thereby generating multiple periodically separated joint point data for each joint J. These multiple periodically separated joint point data represent the movement of the joint J along the phase dimension in periods. Furthermore, in the following description, the multiple periodically separated joint point data after phase alignment may sometimes be referred to as a "periodically separated joint point data set". The first input data represents the periodically separated joint point data set for each joint J.

[0091] Specifically, the time-series data of each joint J constituting the two-dimensional skeleton SK includes: time-series data representing the movement of joint J in the x-direction and time-series data representing the movement of joint J in the y-direction. The preprocessing unit 40 decomposes the time-series data (one-dimensional time-series data) representing the movement of joint J in the x-direction for each joint J into one cycle, thereby generating periodically separated first joint point data for that joint J. The periodically separated first joint point data represents the first joint point data for each cycle. Similarly, the preprocessing unit 40 decomposes the time-series data (one-dimensional time-series data) representing the movement of joint J in the y-direction for each joint J into one cycle, thereby generating periodically separated second joint point data for that joint J. The periodically separated second joint point data represents the second joint point data for each cycle. Then, the preprocessing unit 40 aligns the phases of the multiple first joint point data and the multiple second joint point data for each joint J.

[0092] For example, the preprocessing unit 40 (processing unit 101) can also decompose the one-dimensional time series data into periods based on the Self-DTW algorithm, and obtain samples corresponding to the same phase from each decomposed period, aligning the phases of multiple periodically separated keypoint data. In other words, the preprocessing unit 40 can also align the phases of a certain number N from each decomposed period. phase Data from each phase point is sampled, and the phases of multiple periodically separated keypoint data are aligned. This is achieved by sampling a certain number of N... phase Sampling the data at each phase point ensures that the number of phases contained in the periodically separated keypoint data is uniformly set to a certain number N. phase More specifically, the preprocessing unit 40 (processing unit 101) can also convert one-dimensional time series data into an "N×M" matrix based on the Self-DTW algorithm. Here, "N" represents the number of periods, and "M" represents the time series points of each period after phase alignment.

[0093] Additionally, in the following description, the first joint data separated by period is sometimes referred to as "first periodically separated joint data". Similarly, the second joint data separated by period is sometimes referred to as "second periodically separated joint data". Furthermore, the first joint data separated by period after phase alignment is sometimes referred to as "first periodically separated joint data group". Similarly, the second joint data separated by period after phase alignment is sometimes referred to as "second periodically separated joint data group". The periodically separated joint data groups for each joint J represent the first periodically separated joint data group and the second periodically separated joint data group, respectively.

[0094] Figure 6 The diagram shows the following: Joint point J6 corresponding to the right knee of the subject SJ who is performing stationary marching (see reference). Figure 5 An example of time-series data of the action of (a) and an example of multiple periodically separated keypoint data after phase alignment (a periodically separated keypoint data group).

[0095] In detail, Figure 6 The time series data G1 in the upper left corner represents the joint point J6 corresponding to the right knee of a healthy individual (the non-dementia subject SJ) (see reference). Figure 5 The action of (a) in the y direction. Figure 6 The lower left corner of the key data group G2 represents multiple key data groups separated by period after the time series data G1 is decomposed into each period and phase aligned (the second key data group separated by period). Figure 6 The time series data G3 in the upper right corner represents the joint point J6 corresponding to the right knee of the dementia-stricken subject SJ (see reference). Figure 5 The action of (a) in the y direction. Figure 6 The lower right corner of the key data group G4 represents multiple key data groups separated by period after the time series data G3 is decomposed into each period and phase aligned (the second key data group separated by period).

[0096] In addition, Figure 6 In the upper right and upper left figures respectively, the horizontal axis represents time, and the vertical axis represents the amplitude of the joint J6 in the y-direction. Figure 6 In the lower right and lower left diagrams, the first horizontal axis HA1 represents the period, the second horizontal axis HA2 represents the phase, and the vertical axis represents the amplitude of joint point J6 in the y-direction. More specifically, the vertical axis represents the xy coordinates with the hip joint as the origin (see reference). Figure 5 The y-coordinate in (b)).

[0097] In addition, by decomposing the time series data representing the movement of joint J6 in the y direction corresponding to the right knee of the subject SJ who is performing stationary marching into place into each cycle, periodic joint data is generated, with 2 steps of stationary marching into place as 1 cycle.

[0098] like Figure 6 As shown, compared to the periodicity of movements in healthy individuals, the movements of the dementia patient SJ are not periodic. Therefore, by using periodically separated skeletal sequences (periodically separated keypoint data), the cognitive function of the patient SJ can be evaluated with greater precision.

[0099] Furthermore, by using multiple periodically separated skeleton sequences (periodically separated keypoint data) after phase alignment, it is possible to analyze the variations in samples between different periods corresponding to the same phase. Therefore, the cognitive function of the subject's SJ can be evaluated with greater precision.

[0100] Furthermore, according to this embodiment, samples corresponding to different periods of the same phase are arranged adjacently in three-dimensional space. Therefore, compared to convolving one-dimensional time series data, a smaller convolution kernel size can be used for convolution. As a result, both long-term and short-term relationships between samples can be analyzed. Therefore, the cognitive function of the evaluated subject SJ can be evaluated with higher precision.

[0101] Next, refer to Figure 1 and Figure 7 Let's explain the preprocessing unit 40. When processing data using a machine learning model ML (neural network NW), the number of variables input into the machine learning model ML needs to be a certain number. Therefore, when processing multiple periodically separated skeleton data SKDs after phase alignment using the machine learning model ML, the number of periodically separated skeleton data SKDs (the number of periods) needs to be a certain number. Therefore, in this embodiment, the preprocessing unit 40 generates the first input data by ensuring that the number of periodically separated skeleton data SKDs (the number of periods) is a certain number.

[0102] Specifically, the preprocessing unit 40 extracts M from the time series data of each key point J. periods In each of the cycles, choose N. periods A certain number of cycles are used to unify the number of keypoint data (number of cycles) separated by cycles for each keypoint J. For example, the preprocessing unit 40 selects a certain number (N) consecutive data points with the highest correlation for each keypoint J. periods The number of periodic keypoint data (separated by period) extracted from one-dimensional time series data. Additionally, the number of periodic keypoint data (periods) extracted from one-dimensional time series data (M)periods The number of (N) is less than a certain quantity. periods In the case of (one), the preprocessing unit 40 repeatedly uses the extracted M periods A portion or all of the periodically separated keypoint data, such that the number of periodically separated keypoint data is less than a certain number (N). periods (N) consistent. For example, in "N" periods Under the condition of "10", after extracting the keypoint data corresponding to the 7 periods {P1, P2, P3, P4, P5, P6, P7}, the preprocessing unit 40 adds the keypoint data corresponding to 3 of the extracted 7 periods {P1, P2, P3} to the 7 keypoint data. As a result, the keypoint data corresponding to the 10 periods {P1, P2, P3, P4, P5, P6, P7, P1, P2, P3} is input into the machine learning model ML.

[0103] For example, the preprocessing unit 40 can also calculate a pair of absolute correlation matrices in the x and y dimensions for each joint J based on the following equation (1). Furthermore, in equation (1), "M" represents M. periods .

[0104] C jointx C jointy ∈ [0, 1] M × M ···(1) In formula (1), “C” jointx "and "C jointy “ represents the Pearson correlation coefficient between the periods extracted from the time series data of the key point J in the i-th period and the j-th period. More specifically, the Pearson correlation coefficient is defined by the following equation (2).

[0105] [Mathematical Formula 1] [Mathematical Formula 2] [Mathematical Formula 3] In addition, Equation (3) included in Equation (2) represents the covariance between the periods extracted from the time series data of the key point J in the i-th period and the j-th period, and Equation (4) included in Equation (2) represents the standard deviation between the periods extracted from the time series data of the key point J in the i-th period and the j-th period.

[0106] Furthermore, the preprocessing unit 40, in order to extract M periodsRetrieve a certain number (N) consecutive periods with the highest relevance from a given number of periods. periods The starting index k of a period (joint point data separated by period) is used to determine the relationship between a pair of absolute correlation matrices (C). jointx C jointy The layers are stacked. Then, the preprocessing unit 40 utilizes N... periods ×N periods A sliding window SW of size is used to retrieve the starting index k from a pair of stacked absolute correlation matrices. Figure 7 This shows a pair of absolute correlation matrices (C) after stacking. jointx C jointy A diagram of an example of a sliding window SW. For example, the preprocessing unit 40 may also retrieve the starting index k that maximizes the absolute correlation within the sliding window SW based on the following equation (5).

[0107] [Mathematical Formula 4] In equation (5), “x” and “y” are sets [1, M] shown in equations (6) and (7). periods -N periods ] elements x∈[1,M periods -N periods ]...(6) y∈[1,M periods -N periods ]...(7) When the preprocessing unit 40 retrieves the starting index k, it extracts N consecutive values ​​starting from the starting index k. periods A certain number of periods {P(k), P(k+1), ..., P(k+N)} periods -1)}. Then, the preprocessing unit 40 will be connected with the period {P(k), P(k+1), ..., P(k+N)}. periods -1)} The corresponding periodically separated keypoint data is input into the evaluation unit 60 (refer to...). Figure 1 ).

[0108] As described above, the preprocessing unit 40 decomposes the one-dimensional time series data into periods according to each key point J, and selects a certain number (N) of the periodically separated key point data after phase alignment. periods The selected periodically separated joint data (group of periodically separated joint data) is input as the first input data (group of periodically separated joint data) to the evaluation unit 60 (refer to...). Figure 1 ).

[0109] Next, refer to Figure 8 This embodiment describes the cognitive function evaluation system 100. Figure 8This diagram illustrates the configuration of the evaluation unit 60 included in the cognitive function evaluation system 100 of this embodiment. More specifically, Figure 8 Shown by Figure 2 The machine learning model ML shown is a neural network NW constructed from it. The neural network NW is, for example, a neural network performing deep learning.

[0110] like Figure 8 As shown, in this embodiment, the neural network NW includes an action feature extractor 61, an answer feature extractor 62, a connector 63, and a fusion unit 64.

[0111] Based on the first input data received from the preprocessing unit 40, the action feature extractor 61 extracts features of the actions of the subject SJ who is performing a dual task. Then, the action feature extractor 61 outputs a first feature vector representing the features of the subject SJ's actions. In this embodiment, the action feature extractor 61 extracts features of the actions of the subject SJ after performing a cognitive task (actions of the subject SJ performing the motor task and the dual task in sequence).

[0112] In detail, the motion feature extractor 61 extracts the features of the evaluated subject SJ's movements based on multiple periodically separated skeleton data SKDs (periodically separated skeleton sequences) after phase alignment. In this embodiment, the motion feature extractor 61 extracts the features of the evaluated subject SJ's movements based on periodically separated joint data groups (multiple periodically separated joint data after phase alignment) for each joint J. More specifically, the motion feature extractor 61 extracts the features of the evaluated subject SJ's movements based on a certain number (N) periods The first set of keypoint data separated by period (N) and a certain number (N) periods The second set of joint data (separated by cycle) was used to extract the characteristics of the actions of the evaluated subject SJ.

[0113] More specifically, each joint J's first periodically separated joint data set represents data in the phase dimension, the period dimension, and the spatial dimension (x-dimension). Similarly, each joint J's second periodically separated joint data set represents data in the phase dimension, the period dimension, and the spatial dimension (y-dimension). In other words, the first input data represents data in the phase dimension, the period dimension, and the spatial dimension. Therefore, the first input data has three dimensions (spatial dimension, phase dimension, and period dimension). The action feature extractor 61 extracts the action features of the evaluated subject SJ by convolving the first input data along each dimension (spatial dimension, phase dimension, and period dimension).

[0114] In addition, the number of variables input from the preprocessing unit 40 to the motion feature extractor 61 is "N".dim ×N periods ×N phase ×N joint Here, "N" dim "N" indicates the dimension (spatial dimension) of the skeleton SK. In this embodiment, since a two-dimensional skeleton SK is used, "N" represents the dimension (spatial dimension) of the skeleton SK. dim =2". As mentioned before, "N" periods "N" indicates the number of periods. phase " indicates the number of phases. Additionally, "N" indicates the number of phases. joint "Indicates the number of key points J. For example, when using reference..." Figure 5 In the case of (a) describing the joint data groups of all joints J of the skeleton SK separated by period, the number N of joints J is... joint 17 (N) joint =17).

[0115] The response feature extractor 62 extracts features of the responses of the evaluated subject SJ who is performing a dual task based on the second input data input from the response processing unit 50. Then, the response feature extractor 62 outputs a second feature vector representing the features of the evaluated subject SJ's responses. In this embodiment, the response feature extractor 62 extracts features of the evaluated subject SJ's responses to cognitive topics based on the responses of the evaluated subject SJ performing a cognitive task and the responses of the evaluated subject SJ performing a dual task.

[0116] In detail, the response feature extractor 62 extracts features of the respondent SJ's responses based on the respondent SJ's response speed and accuracy on cognitive tasks. More specifically, the response feature extractor 62 extracts features of the respondent SJ's responses based on a first response speed (response speed when performing a cognitive task), a second response speed (response speed when performing a dual task), a first accuracy (accuracy when performing a cognitive task), and a second accuracy (accuracy when performing a dual task).

[0117] More specifically, the answer feature extractor 62 extracts features of the respondent SJ's answers by performing convolution on the second input data. Furthermore, the number of variables input from the answer processing unit 50 to the answer feature extractor 62 is four (first answer speed, second answer speed, first accuracy, and second accuracy).

[0118] Connector 63 concatenates the first feature vector output from action feature extractor 61 with the second feature vector output from response feature extractor 62 to form a third feature vector. The third feature vector represents a cross-modal feature vector. The third feature vector is then input to fusion unit 64.

[0119] The fusion unit 64 fuses the first and second feature vectors to output evaluation data (data representing the evaluation results of the evaluated subject SJ's cognitive function). That is, the fusion unit 64 fuses the features of the evaluated subject SJ's actions with the features of the evaluated subject SJ's responses to output evaluation data. For example, the evaluation data represents a positive prediction probability. A positive result, for example, indicates that the cognitive function belongs to the dementia category. Or, a positive result indicates that the cognitive function belongs to a category that includes both dementia and mild cognitive impairment. In other words, as evaluation data, the fusion unit 64 outputs data classifying the evaluated subject SJ's cognitive function into the dementia category (positive) and the category that includes mild cognitive impairment and non-dementia (negative). Alternatively, as evaluation data, the fusion unit 64 outputs data classifying the evaluated subject SJ's cognitive function into the category that includes both dementia and mild cognitive impairment (positive) and the category that does not include dementia (negative).

[0120] According to this embodiment, since the first feature vector and the second feature vector are concatenated and input into the fusion unit 64, the cognitive function of the person being evaluated (SJ) can be evaluated without losing a portion of the information representing the characteristics of the actions of the person being evaluated (SJ). Similarly, the cognitive function of the person being evaluated (SJ) can be evaluated without losing a portion of the information representing the characteristics of the responses of the person being evaluated (SJ). Therefore, the cognitive function of the person being evaluated (SJ) can be evaluated with higher accuracy.

[0121] Next, refer to Figure 9 Here is an example illustrating the network structure of the action feature extractor 61. Figure 9 This is a diagram illustrating the neural network of the action feature extractor 61 included in the cognitive function evaluation system 100 of this embodiment.

[0122] Figure 9 The action feature extractor 61 shown includes a normalization layer 611, a first graph convolutional block 612, a second graph convolutional block 613, a third graph convolutional block 614, and a pooling layer 615. These layers are concatenated in the order of normalization layer 611, first graph convolutional block 612, second graph convolutional block 613, third graph convolutional block 614, and pooling layer 615. Furthermore, the first graph convolutional block 612, second graph convolutional block 613, and third graph convolutional block 614 are an example of "multiple convolutional blocks".

[0123] The first input data (a group of joint data points J separated by period) is input from the preprocessing unit 40 to the normalization layer 611. The normalization layer 611 normalizes the first input data. For example, the normalization layer 611 may also include a batch normalization layer. In this embodiment, since the first input data has three dimensions, the normalization layer 611 may include a three-dimensional batch normalization layer. The normalized first input data is input to the first graph convolution block 612.

[0124] The first convolutional block 612 performs convolutions on the phase dimension, period dimension, and spatial dimension data contained in the normalized first input data (the periodically separated keypoint data groups of each keypoint J), and outputs a first feature map. The first feature map is then input into the second convolutional block 613.

[0125] The second convolutional block 613 and the third convolutional block 614 are constructed in roughly the same way as the first convolutional block 612. The second convolutional block 613 convolves the phase-dimensional data, periodic-dimensional data, and spatial-dimensional data contained in the first feature map, respectively, to output the second feature map. The third convolutional block 614 convolves the phase-dimensional data, periodic-dimensional data, and spatial-dimensional data contained in the second feature map, respectively, to output the third feature map.

[0126] Furthermore, the number of channels in the second convolutional block 613 can be less than the number of channels in the first convolutional block 612. Additionally, the number of channels in the third convolutional block 614 can be greater than the number of channels in the second convolutional block 613. The number of channels in the third convolutional block 614 can also be the same as the number of channels in the first convolutional block 612.

[0127] In this embodiment, the feature map (third feature map) from the third convolutional block 614 is input to the pooling layer 615. The pooling layer 615 converts the feature map (third feature map) into a first feature vector. For example, the pooling layer 615 may also include a global average pooling layer. For example, the pooling layer 615 converts the feature map (third feature map) into a feature vector (first feature vector) with a vector length of 64.

[0128] According to this embodiment, since the first input data is convolved by multiple convolutional blocks (here, 3 graph convolutional blocks) connected in series, the number of operations in one convolutional block (one graph convolutional block) can be reduced.

[0129] Next, refer to Figure 10 This illustrates spatial convolution (CV1) for convolving data in the spatial dimension, phase convolution (CV2) for convolving data in the phase dimension, and periodic convolution (CV3) for convolving data in the period dimension. Figure 10This is a schematic diagram illustrating spatial convolution processing CV1, phase convolution processing CV2, and periodic convolution processing CV3. Figure 10 In the diagram, the first horizontal axis HA1 represents the period, and the second horizontal axis HA2 represents the phase.

[0130] exist Figure 10 The diagram illustrates the periodically separated skeleton data SKD(i) and periodically separated skeleton data SKD(i+1) after phase alignment. Periodically separated skeleton data SKD(i) represents the periodically separated skeleton data SKD of the i-th period (i) extracted from the skeleton sequence. Periodically separated skeleton data SKD(i+1) represents the periodically separated skeleton data SKD of the (i+1)-th period (i+1) extracted from the skeleton sequence. Both periodically separated skeleton data SKD(i) and periodically separated skeleton data SKD(i+1) contain skeletons SKD of specific phases. Here, using periodically separated skeleton data SKD(i) and periodically separated skeleton data SKD(i+1) as examples, the convolution process on all joints J constituting the skeleton SK is explained.

[0131] Spatial convolution processing CV1 represents the convolution process performed on all joints J contained in the same skeleton SK. Here, the same skeleton SK refers to the skeleton SK of the same frame. Therefore, for each skeleton SK in each phase of period (i), all joints J are convolved to extract the spatial positional relationship features of all joints J. Similarly, for each skeleton SK in each phase of period (i+1), all joints J are convolved to extract the spatial positional relationship features of all joints J.

[0132] Phase convolution processing CV2 represents the process of convolving each joint J (the same joint J) within the same period along the phase dimension. For example, joint J7 contained in each phase of period (i) is convolved along the phase dimension. Similarly, joint J7 contained in each phase of period (i+1) is convolved along the phase dimension. The other joint Js that make up the skeleton SK are also convolved in the same way.

[0133] Periodic convolution processing (CV3) refers to the process of convolving each joint J corresponding to the same phase (the same joint J) along the periodic dimension. For example, joint J8 corresponding to the same phase at period (i) is convolved with joint J8 at period (i+1) along the periodic dimension. The other joint Js that make up the skeleton SK are also convolved in the same way.

[0134] According to this embodiment, the motion of each joint J can be convolved along the phase dimension. Furthermore, the motion of each joint J can be convolved along the periodic dimension. Therefore, periodic features of the motion of each joint J can be extracted. Thus, periodic features of the motion of the evaluated subject SJ can be extracted.

[0135] Next, refer to Figure 11 Here is an example illustrating the network structure of convolutional block 612 in the first figure. Figure 11 This is a diagram illustrating the neural network of the first graph convolutional block 612 included in the cognitive function evaluation system 100 of this embodiment. (See diagram for example.) Figure 11 As shown, the first graph convolutional block 612 may also include a first graph convolutional layer 81, a second graph convolutional layer 82, a third graph convolutional layer 83, a multiplier 84, and an adder 85.

[0136] The first input data (groups of joint data separated by period for each joint J) after normalization by normalization layer 611 is input to the first graph convolutional layer 81. The first graph convolutional layer 81 generates a feature map by convolving the phase dimension data contained in the first input data. That is, the first graph convolutional layer 81 performs a reference... Figure 10 The phase convolution process CV2 is illustrated. The first convolutional layer 81 is an example of a "first convolutional layer".

[0137] For example, the first convolutional layer 81 can also include a convolutional kernel size of "1×K". A 3D graph convolutional layer with a kernel size of "×1" is used. A 3D graph convolutional layer of “×1” can reduce the number of channels that need to be processed for the spatial and periodic dimensions of the data contained in the first input data, thereby reducing the number of convolution operations.

[0138] Multiplier 84 multiplies the feature map output from the first graph convolutional layer 81 with the adjacency matrix A. Adjacency matrix A represents the spatial positional relationship (in this embodiment, a two-dimensional positional relationship) of each joint J constituting the skeleton SK. In this embodiment, periodically separated joint data sets of each joint J are input to the first graph convolutional block 612. Multiplier 84 adds the positional relationship information of each joint J to the periodically separated joint data sets of each joint J, thereby enabling more precise execution of the reference... Figure 10 The spatial convolution process CV1 and the periodic convolution process CV3 are described.

[0139] The first input data, after convolving the phase dimension data through the first graph convolutional layer 81, is input to the second graph convolutional layer 82. In this embodiment, the input is the output of the multiplier 84 (a feature map with added information on the positional relationships of each key point J). The second graph convolutional layer 82 simultaneously performs a reference operation on the data from the multiplier 84. Figure 10 The description includes spatial convolution processing CV1, phase convolution processing CV2, and periodic convolution processing CV3. That is, the second graph convolutional layer 82 extracts features of the evaluated subject SJ's actions by simultaneously convolving the spatial, phase, and periodic data contained in the first input data obtained by convolving the phase dimension data from the first graph convolutional layer 81. More specifically, the second graph convolutional layer 82 outputs a feature map representing the features of the evaluated subject SJ's actions. For example, the first graph convolutional layer 81 may also include a convolutional kernel size of "Kp×K". The three-dimensional graph convolutional layer “×Ks”. The second graph convolutional layer 82 is an example of a “second convolutional layer”.

[0140] In this embodiment, the neural network of the first graph convolutional block 612 includes a residual neural network (ResNet). Specifically, the residual neural network RN is constructed by the third graph convolutional layer 83 and the adder 85. The residual neural network RN performs calculations on the difference (residual) between the output of the second graph convolutional layer 82 and the first input data, and outputs the result as a first feature map from the first graph convolutional block 612.

[0141] In detail, the first input data (a group of joint data points J separated by period) after normalization by normalization layer 611 is input to the third graph convolutional layer 83. The third graph convolutional layer 83 is a 3D graph convolutional layer with a kernel size of "1×1×1". Adder 85 adds the output (feature map) of the second graph convolutional layer 82 and the output (feature map) of the third graph convolutional layer 83 to form the first feature map. As a result, the first feature map is output from the first graph convolutional block 612. In addition, the third graph convolutional layer 83 is an example of a "third convolutional layer".

[0142] According to this embodiment, since the neural network of the first graph convolutional block 612 includes a residual neural network (RN), gradient vanishing is less likely, thus improving learning efficiency. Furthermore, according to this embodiment, by using a third graph convolutional layer 83 with a kernel size of "1×1×1", the dimensionality of the feature map added to the output of the second graph convolutional layer 82 can be reduced (dimensionality reduction). Therefore, the efficiency of computational processing in the first graph convolutional block 612 can be improved.

[0143] In addition, the network structures of convolutional block 613 in the second figure and convolutional block 614 in the third figure are roughly the same as those of convolutional block 612 in the first figure, so their description is omitted.

[0144] The above is as follows (refer to) Figure 11 As explained, the first input data, after the phase is convolved by the first graph convolutional layer 81, is input to the second graph convolutional layer 82. Therefore, the amount of data processed by the second graph convolutional layer 82 can be reduced, and the number of operations of the second graph convolutional layer 82 can be reduced.

[0145] Next, refer to Figure 12 Here is an example illustrating the network structure of the feature extractor 62. Figure 12 This is a diagram illustrating the neural network of the response feature extractor 62 included in the cognitive function evaluation system 100 of this embodiment. Figure 12 The illustrated response feature extractor 62 includes a normalization layer 621, a convolutional layer 622, a convolutional layer 623, a normalization layer 624, a convolutional layer 625, and a pooling layer 626. These layers are connected in series in the order of normalization layer 621, convolutional layer 622, convolutional layer 623, normalization layer 624, convolutional layer 625, and pooling layer 626.

[0146] The second input data is input from the response calculation unit 50 to the normalization layer 621. The normalization layer 621 normalizes the second input data. For example, the normalization layer 621 includes a one-dimensional batch normalization layer.

[0147] The second input data, normalized by normalization layer 621, is then convolved sequentially by convolutional layers 622 and 623. For example, convolutional layer 622 includes a one-dimensional convolutional layer with a kernel size of 3. Convolutional layer 623 performs depthwise convolution on the output (feature map) of convolutional layer 622. For example, convolutional layer 623 includes a one-dimensional convolutional layer that performs depthwise convolution on the output of convolutional layer 622.

[0148] Normalization layer 624 normalizes the output (feature map) of convolutional layer 623. For example, normalization layer 624 includes a one-dimensional batch normalization layer. Convolutional layer 625 is the same as convolutional layer 623, performing convolution on the output of normalization layer 624 in the depth direction. For example, convolutional layer 625 includes a one-dimensional convolutional layer that convolutions on the output of normalization layer 624 in the depth direction.

[0149] The feature map is input from convolutional layer 625 to pooling layer 626. Pooling layer 626 transforms the feature map into a feature vector (second feature vector). For example, pooling layer 626 includes a global average pooling layer. For example, pooling layer 626 transforms the feature map into a feature vector with a vector length of 160.

[0150] Next, refer to Figure 13 Here is an example illustrating the network structure of the fusion unit 64. Figure 13 This is a diagram illustrating the neural network of the fusion unit 64 included in the cognitive function evaluation system 100 of this embodiment. Figure 13 The fusion unit 64 shown includes a normalization layer 641, a fully connected layer 642, a fully connected layer 643, and an output layer 644. These layers are connected in series in the order of normalization layer 641, fully connected layer 642, fully connected layer 643, and output layer 644.

[0151] The third feature vector is input from connector 63 to normalization layer 641. As described above, connector 63 connects the first and second feature vectors to form the third feature vector. Normalization layer 641 normalizes the third feature vector. For example, normalization layer 641 includes a one-dimensional batch normalization layer.

[0152] The normalized third feature vector is linearly combined (linearly transformed) through fully connected layers 642 and 643 before being input into output layer 644. Through fully connected layers 642 and 643, the features of the subject SJ's actions and responses are fused. Output layer 644, based on the output of fully connected layer 643, outputs evaluation data (data representing the evaluation results of the subject SJ's cognitive function). Specifically, output layer 644 operates on the output of fully connected layer 643 using the sigmoid function and outputs data representing the positive prediction probability.

[0153] Furthermore, during the training of the machine learning model ML, the processing unit 101 can also use the binary cross-entropy (BCE) loss to determine the values ​​of the multiple parameters contained in the neural network constituting the fusion unit 64. Hereinafter, the values ​​of the multiple parameters contained in the neural network constituting the fusion unit 64 will sometimes be referred to as "parameter values ​​of the fusion unit 64".

[0154] Binary cross-entropy loss L BCE It is the loss function that minimizes the difference between the predicted probability distribution and the probability distribution of the correct answer. Therefore, by utilizing the binary cross-entropy loss L... BCE The parameters of the fusion unit 64 are adjusted to classify the cognitive function of the evaluated subject SJ with higher accuracy. Specifically, the processing unit 101 adjusts the parameters of the fusion unit 64 to adjust the binary cross-entropy loss L... BCE The value (loss value) becomes the minimum value.

[0155] Next, an example of the loss function L used during the training of the machine learning model ML will be described. In this embodiment, during the training of the machine learning model ML, a loss function L combined with the first loss function L is used.same Second loss function L opp and binary cross-entropy loss L BCE The loss function L. The processing unit 101 uses the first loss function L during the training of the machine learning model ML. same Second loss function L opp The values ​​(parameter values) of multiple parameters contained in the neural network constituting the action feature extractor 61 are determined hereafter. Hereinafter, the values ​​of multiple parameters contained in the neural network constituting the action feature extractor 61 are sometimes referred to as "parameter values ​​of action feature extractor 61".

[0156] In detail, the processing unit 101 calculates the first loss function L same and the second loss function L opp The parameter values ​​of the action feature extractor 61 are determined so that the first feature vectors generated from the first input data (descriptive variables) with the same label (target variable) become more similar feature vectors, and the first feature vectors generated from the first input data (descriptive variables) with different labels (target variables) become different feature vectors. The first loss function L... same and the second loss function L opp It is a loss function used to adjust the parameter values ​​of the action feature extractor 61 so that similar first feature vectors are generated for samples of the same class, and different first feature vectors are generated for samples of different classes.

[0157] Hereinafter, the first feature vector generated from the first input data that is assigned the same label will sometimes be referred to as "the first feature vector of the same label data". Similarly, the first feature vector generated from the first input data that is assigned different labels will sometimes be referred to as "the first feature vector of the different label data".

[0158] Specifically, the first loss function L same The first loss function L minimizes the cosine distance (pairwise cosine distance) between the first feature vectors of data with the same label. During the training of the machine learning model ML, the processing unit 101 calculates the first loss function L for each batch. same The processing unit 101 adjusts the parameter values ​​of the action feature extractor 61 to minimize the cosine distance between the first feature vectors of data with the same label. In other words, the processing unit 101 adjusts the parameter values ​​of the action feature extractor 61 to minimize the cosine distance between the first feature vectors of data with the same label. same The value (loss value) becomes the minimum. Therefore, the parameter values ​​of the action feature extractor 61 are adjusted so that the first feature vectors of the same labeled data become more similar feature vectors. As a result, the accuracy of the classification of the cognitive function of the evaluated subject SJ is improved.

[0159] In detail, during batch learning, the processing unit 101 divides each training data corresponding to each first feature vector into positive F+ sub-batches and negative F- sub-batches based on the label data (positive solution labels). Then, the processing unit 101 calculates the loss L of the positive F+ sub-batches, for example, based on the following equations (8) and (9). same + (loss value) and the loss L of the negative F- sub-batch same - (loss value).

[0160] [Mathematical Formula 5] [Mathematical Formula 6] Finally, the processing unit 101 calculates the first loss function L. same For example, the first loss function L same It can also be calculated based on the following equation (10). By calculating the first loss function L shown in equation (10) same It is possible to calculate the loss L for the sub-batch with positive F+. same + (loss value) and the loss L of the negative F- sub-batch same - The weighted sum of the loss values ​​(loss values).

[0161] [Mathematical Expression 7] Next, the second loss function L will be explained. opp The second loss function L opp The second loss function, L, minimizes the cosine similarity (pairwise cosine similarity) between the first feature vectors of data with different labels. During the training of the machine learning model ML, the processing unit 101 calculates the second loss function L according to each batch. opp The processing unit 101 adjusts the parameter values ​​of the action feature extractor 61 to minimize the cosine similarity between the first feature vectors of different labeled data. In other words, the processing unit 101 adjusts the parameter values ​​of the action feature extractor 61 to minimize the cosine similarity between the first feature vectors of different labeled data. opp The value (loss value) becomes the minimum. Therefore, the parameter values ​​of the action feature extractor 61 are adjusted so that the first feature vector of different labeled data becomes a different feature vector. As a result, the accuracy of the classification of the cognitive function of the evaluated subject SJ is improved. For example, the second loss function L opp It can also be calculated based on the following formula (11).

[0162] [Mathematical Formula 8] Furthermore, the processing unit 101 can also store multiple first feature vectors used to calculate the loss (loss value calculation) of the (n-1)th batch, and reuse the stored multiple first feature vectors for loss calculation if the nth batch does not contain either positive or negative samples. As a result, even if the training dataset for the machine learning model ML is imbalanced, batch processing anomalies can be avoided because loss calculation is always performed.

[0163] Furthermore, before starting training the machine learning model (ML), the operators ensured that the first batch contained at least one positive and one negative sample. As a result, batch anomalies were avoided because multiple first feature vectors used in calculating the loss of the first batch containing both positive and negative samples were preserved.

[0164] Next, the loss function L will be explained. As mentioned above, the loss function L combines the first loss function L... same Second loss function L opp and binary cross-entropy loss L BCE The loss function. As mentioned above, the binary cross-entropy loss L... BCE Used to adjust the parameter values ​​of fusion unit 64.

[0165] During the training of the machine learning model ML, the processing unit 101 calculates the loss function L for each batch. As a result, the first loss function L is calculated for each batch. same Second loss function L opp and binary cross-entropy loss L BCE For example, the loss function L can also be calculated based on the following equation (12). Furthermore, in equation (12), "α" represents the first loss function L. same The coefficient, "β" represents the second loss function L opp The coefficient.

[0166] [Mathematical Expression 9] As shown in equation (12), the binary cross-entropy loss L is calculated by using the sum of α and β, "α + β". BCE Weighting is applied to reduce the binary cross-entropy loss L. BCE The impact is greater than that of the first loss function L same and the second loss function L opp This improves the accuracy of cognitive function classification. Furthermore, the total loss (loss value) is normalized using "1 / (2α+2β)".

[0167] Above, refer to Figures 1 to 13Embodiments of the present invention have been described. According to this embodiment, the cognitive function of the subject SJ can be evaluated using a dual-task approach. More specifically, the cognitive function of the subject SJ can be evaluated based on the periodicity of the subject SJ's movements (characteristics of periodic movements). Therefore, the cognitive function of the subject SJ can be evaluated with higher precision.

[0168] Furthermore, according to this embodiment, by evaluating the cognitive function of the subject SJ based on the periodicity of their movements (characteristics of periodic movements), the influence of individual differences in motor ability on the evaluation results of cognitive function can be avoided. Therefore, the cognitive function of the subject SJ can be evaluated with higher accuracy.

[0169] Furthermore, according to this embodiment, the phases of the periodic movements of the person being evaluated, SJ, can be aligned. Therefore, the impact of phase differences between cycles on the evaluation of cognitive function can be avoided. As a result, the cognitive function of the person being evaluated, SJ, can be evaluated with higher accuracy.

[0170] The above is with reference to the attached diagram ( Figures 1 to 13 The embodiments of the present invention have been described. However, the present invention is not limited to the above embodiments and can be implemented in various ways without departing from its spirit. Furthermore, the various constituent elements disclosed in the above embodiments can be appropriately modified. For example, one of the constituent elements shown in one embodiment can be added to the constituent elements of another embodiment, or several constituent elements shown in one embodiment can be deleted from that embodiment.

[0171] The accompanying drawings are provided to facilitate understanding of the invention and are schematic representations of the constituent elements. The thickness, length, number, and spacing of these elements may sometimes differ from the actual figures due to drawing conventions. Furthermore, the configuration of the constituent elements shown in the above embodiments is merely an example and is not intended to limit the invention. Various modifications can be made without substantially departing from the effects of the invention.

[0172] For example, in reference Figures 1 to 13 In the described implementation, in order to evaluate the cognitive function of the evaluated subject SJ, the answers of the evaluated subject SJ who is performing a cognitive task and the answers of the evaluated subject SJ who is performing a dual task are used. However, the evaluation unit 60 may also extract the features of the answer from only one of these answers.

[0173] In addition, in reference Figures 1 to 13In the described implementation, the respondent SJ's responses are used to evaluate the cognitive function of the person being evaluated, but these responses can be omitted. In other words, the cognitive function of the person being evaluated can also be evaluated solely based on the characteristics of their actions.

[0174] In addition, in reference Figures 1 to 13 In the described implementation, in order to evaluate the cognitive function of the evaluated subject SJ, a video signal representing the actions of the evaluated subject SJ performing a motor task is used. However, the video signal representing the actions of the evaluated subject SJ performing a motor task may be omitted.

[0175] In addition, in reference Figures 1 to 13 In the described implementation, the subject SJ performs a task set once, which is given sequentially in the order of cognitive task, motor task, and dual task. However, the subject SJ can also perform the task set multiple times consecutively. In this case, first input data and second input data are generated according to each task set. The action feature extractor 61 processes the first input data according to each task set and extracts the features of the subject SJ's actions according to each task set. Similarly, the response feature extractor 62 processes the second input data according to each task set and extracts the features of the subject SJ's responses according to each task set. Therefore, the first feature vector represents the features of the subject SJ's actions extracted according to each task set, and the second feature vector represents the features of the subject SJ's responses extracted according to each task set. As a result, for example, if the subject SJ performs the task set three times consecutively, the vector length of the first feature vector is "64×3", and the vector length of the second feature vector is "160×3". Therefore, the fusion unit 64 fuses the first feature vector (192-dimensional feature vector) with a vector length of 192 and the second feature vector (480-dimensional feature vector) with a vector length of 480.

[0176] In addition, in reference Figures 1 to 13 In the described implementation, the subject SJ performs the cognitive task, motor task, and dual task in that order, but the order in which the subject SJ performs the cognitive task, motor task, and dual task can be changed.

[0177] In addition, in reference Figures 1 to 13In the described implementation, the subject SJ performs cognitive tasks, motor tasks, and dual tasks. However, the task performed by the subject SJ only needs to include at least a dual task. For example, the subject SJ may only perform a dual task. Alternatively, the subject SJ may perform a motor task and a dual task, or a cognitive task and a dual task. Furthermore, the task set performed by the subject SJ may be a task that involves performing any one of the cognitive, motor, and dual tasks more than twice. For example, the task set may include more than two dual tasks.

[0178] In addition, in reference Figures 1 to 13 In the described embodiment, the cognitive function of the person being evaluated, SJ, is classified into two categories. However, the evaluation unit 60 may also classify the cognitive function of the person being evaluated, SJ, into three or more categories. Specifically, the evaluation unit 60 may classify the cognitive function of the person being evaluated, SJ, into categories of dementia, mild cognitive impairment, and non-dementia. Alternatively, the evaluation unit 60 may classify the cognitive function of the person being evaluated, SJ, according to the type of dementia. Types of dementia include Alzheimer's dementia, vascular dementia, Lewy body dementia, frontotemporal dementia, and normal pressure hydrocephalus, etc. For example, the evaluation unit 60 may also classify the cognitive function of the person being evaluated, SJ, into categories of Alzheimer's dementia, Lewy body dementia, mild cognitive impairment, and non-dementia.

[0179] In addition, in reference Figures 1 to 13 In the described implementation, the labeled training data uses the doctor's diagnosis; however, labeled data can also be created using the cognitive function score of the subject (SJ). As a cognitive function score, for example, the MMSE (Mini Mental State Examination) score or a general intelligence assessment scale such as the Hasegawa Mini-Mental State Examination can be used. In diagnoses using the MMSE score, an MMSE score below 23 is considered a suspicion of dementia; an MMSE score above 23 but below 27 is considered a suspicion of mild cognitive impairment. Furthermore, an MMSE score above 27 is considered not to be dementia.

[0180] In addition, in reference Figures 1 to 13 In the described embodiment, the evaluation unit 60 performs category classification, but the evaluation unit 60 can also perform regression classification. For example, the evaluation unit 60 can also determine the cognitive function score of the evaluated subject SJ.

[0181] In addition, in reference Figures 1 to 13In the described embodiment, the imaging unit 10 has a camera or an RGB camera, but the imaging unit 10 may also have a depth camera. In this case, the motion features of the evaluated subject SJ can also be extracted from the three-dimensional skeleton sequence.

[0182] In addition, in reference Figures 1 to 13 In the described implementation, the skeleton SK consists of 17 joints J. However, as long as it can represent the whole body of the person being evaluated SJ, the number of joints J constituting the skeleton SK is not particularly limited. For example, the skeleton SK can also consist of 25 joints J.

[0183] In addition, in reference Figures 1 to 13 In the described embodiment, the skeleton generation unit 30 generates a skeleton SK representing the entire body of the evaluated subject SJ. However, the skeleton generation unit 30 may also generate a skeleton representing a part of the evaluated subject SJ's body. For example, the skeleton generation unit 30 may generate a skeleton representing the lower limbs of the evaluated subject SJ. Alternatively, the skeleton generation unit 30 may generate a skeleton representing the right or left half of the evaluated subject SJ, or a skeleton representing the knees or waist of the evaluated subject SJ.

[0184] In addition, in reference Figures 1 to 13 In the described implementation, the motion features of all joints J constituting the skeleton SK are extracted, but it is also possible to extract the motion features of a portion of the joints J constituting the skeleton SK.

[0185] In addition, in reference Figures 1 to 13 In the described embodiment, the three convolutional blocks (first graph convolutional block 612, second graph convolutional block 613, and third graph convolutional block 614) each have a residual neural network RN (third graph convolutional layer 83 and adder 85), but the residual neural network RN (third graph convolutional layer 83 and adder 85) can also be omitted.

[0186] In addition, in reference Figures 1 to 13 In the described embodiment, the three convolutional blocks (first convolutional block 612, second convolutional block 613, and third convolutional block 614) each have a multiplier 84, but the multiplier 84 may be omitted.

[0187] In addition, in reference Figures 1 to 13 In the described embodiment, the three convolutional blocks (first graph convolutional block 612, second graph convolutional block 613, and third graph convolutional block 614) each have a first graph convolutional layer 81, but the first graph convolutional layer 81 can also be omitted. For example, the three convolutional blocks (first graph convolutional block 612, second graph convolutional block 613, and third graph convolutional block 614) can also each have only a second graph convolutional layer 82.

[0188] In addition, in reference Figures 1 to 13In the described embodiment, the action feature extractor 61 has three convolutional blocks (first graph convolutional block 612, second graph convolutional block 613, and third graph convolutional block 614), but the action feature extractor 61 may also have one convolutional block (graph convolutional block), or two or more convolutional blocks (graph convolutional blocks).

[0189] In addition, in reference Figures 1 to 13 In the described embodiment, the task presentation unit 70 has a display, but the configuration of the task presentation unit 70 is not particularly limited as long as it can present the task to be performed by the evaluated person SJ. For example, the task presentation unit 70 may also have an audio output device.

[0190] In addition, in reference Figures 1 to 13 In the described embodiment, the response detection unit 20 has a response switch for the left hand and a response switch for the right hand. However, the configuration of the response detection unit 20 is not particularly limited as long as it can detect responses to cognitive questions. For example, the response detection unit 20 may also have a gaze direction detection device or a microphone.

[0191] When using a gaze direction detection device, for example, based on the evaluation subject SJ... Figure 3 (b) and Figure 4 The gaze direction shown in the answer candidate presentation screen 12b allows for the acquisition of the respondent SJ's answer. For the gaze direction detection device, known gaze direction detection techniques can be employed. For example, the gaze direction detection device includes a near-infrared LED and a camera device. The near-infrared LED illuminates near-infrared light into the respondent SJ's eyes. The camera device captures an image of the respondent SJ's eyes. The processing unit 101 analyzes the image captured by the camera device and detects the position of the respondent SJ's pupils (the direction of the gaze).

[0192] When using a microphone, for example, based on the evaluated person SJ... Figure 3 (b) and Figure 4 The sound emitted from the answer candidate presentation screen 12b can be used to obtain the response from the person being evaluated, SJ. For example, the processing unit 101 can convert the sound emitted by the person being evaluated, SJ, into text data through speech recognition processing, thereby obtaining the response from the person being evaluated, SJ.

[0193] Furthermore, when using a microphone, cognitive tasks are not limited to questions where the person being evaluated, SJ, chooses one answer from two candidate answers. For example, the person being evaluated, SJ, could also answer calculation questions. Additionally, when using a microphone, cognitive tasks can also be questions requiring a verbal response. Examples of verbal response questions include "chain word questions," "questions about identifying sounds (words) that begin with any of the fifty sounds (hiragana and katakana)," or "questions about identifying sounds (words) that begin with any of the letters in the English alphabet."

[0194] Industrial availability This invention can be used for the diagnosis of dementia and mild cognitive impairment.

[0195] Explanation of reference numerals in the attached figures 10: Camera Department 20: Answer the testing department 30: Skeleton generation section (generation section) 40: Pre-processing Department 60: Evaluation Department 81: First convolutional layer (first convolutional layer) 82: Second convolutional layer (second convolutional layer) 84: Multiplier 100: Cognitive Function Assessment System 612: First image: Convolutional block (convolutional block) 613: Second diagram: Convolutional block (convolutional block) 614: Third Figure - Convolutional Block (Convolutional Block) A: Adjacency matrix CV1: Spatial Convolution Processing CV2: Phase Convolution Processing CV3: Periodic Convolution Processing G1: Time series data of key points G2: Joint data groups separated by period G3: Time series data of key points G4: Joint data groups separated by period J: Key Point RN: Residual Neural Network SJ: The person being evaluated SK: skeleton SKD: Skeleton data separated by period.

Claims

1. A cognitive function assessment system for evaluating the cognitive function of an individual, the cognitive function assessment system comprising: The camera department captures images of the evaluated person performing a dual task, which is a task that simultaneously presents a cognitive task and a motor task. The generation unit generates time-series data of a skeleton based on the camera signal. The skeleton is the skeleton of the person being evaluated who is performing the dual task, and it is a skeleton that represents at least a part of the whole body of the person being evaluated. The preprocessing unit decomposes the time series data of the skeleton by period to generate periodically separated skeleton data, and aligns the phases of the periodically separated skeleton data to generate input data. as well as The evaluation department extracts features of the evaluated person's actions based on the input data, and evaluates the evaluated person's cognitive function based on the extracted features of the evaluated person's actions.

2. The cognitive function evaluation system as described in claim 1, The time-series data of the skeleton represents the time-series data of each of the multiple joints that make up the skeleton. The plurality of joints represents at least a portion of all joints constituting the skeleton. The preprocessing unit decomposes the time-series data of the joint points by period to generate periodically separated joint point data, and aligns the phases of the periodically separated joint point data to generate the input data.

3. The cognitive function evaluation system as described in claim 2, The preprocessing unit samples a certain number of data points of the same phase from each of the periodically separated joint data to generate the input data.

4. The cognitive function evaluation system as described in claim 2 or 3, The preprocessing unit selects a certain number of consecutive, periodically separated joint data points with the highest correlation from the joint data points separated by period, according to the joint points, to generate the input data.

5. The cognitive function evaluation system as described in claim 2 or 3, The input data represents the phase-aligned, periodically separated keypoint data. The phase-aligned, periodically separated keypoint data represents data in the spatial dimension, phase dimension, and period dimension. The evaluation unit extracts the features of the evaluated person's actions by performing convolution on the data in the spatial dimension, the phase dimension, and the period dimension, respectively.

6. The cognitive function evaluation system as described in claim 5, The evaluation department includes: The first convolutional layer convolves the phase dimension data contained in the input data. as well as The second convolutional layer extracts the features of the evaluated person's actions by convolving the spatial dimension data, the phase dimension data, and the period dimension data contained in the input data obtained by convolving the phase dimension data by the first convolutional layer.

7. The cognitive function evaluation system as described in claim 6, The evaluation unit further includes a multiplier, which multiplies the adjacency matrix representing the positional relationships of the plurality of key points by the output of the first convolutional layer. The output of the multiplier is input into the second convolutional layer.

8. The cognitive function evaluation system as described in claim 6, The evaluation unit further includes a residual neural network, which performs calculations on the difference between the output of the second convolutional layer and the input data.

9. The cognitive function evaluation system as described in claim 5, The evaluation unit includes multiple convolutional blocks connected in series. Each of the plurality of convolutional blocks performs convolution on the data in the spatial dimension, the data in the phase dimension, and the data in the period dimension.

10. The cognitive function assessment system as described in any one of claims 1 to 3, The cognitive function assessment system also includes a response detection unit, which is used to detect the responses given by the assessed individual to cognitive tasks. The response detection unit detects at least one of the responses of the evaluated person performing the dual task and the responses of the evaluated person performing a cognitive task with a given cognitive topic. The evaluation unit also extracts features of the responses detected by the response detection unit, and evaluates the cognitive function of the person being evaluated based on the features of the person's actions and the features of the responses.

Images