Information processing device, information processing method, program, and information processing system

Abstract

To provide an information processing device, an information processing method, a program, and an information processing system for evaluating a motor function of a person to be evaluated easily and accurately, proposing effective rehabilitation, and improving the quality of the rehabilitation.SOLUTION: An information processing device includes: a moving image acquisition unit 1031 for acquiring a moving image for evaluation capturing an entire body of a person to be evaluated taking a single leg standing posture and period information on the single leg standing posture; a moving image processing unit 1032 for estimating a key point including a joint point and a center of gravity of the person to be evaluated shown in the moving image for evaluation; and an evaluation unit 1033 for generating an evaluation result for the balance ability of the person to be evaluated using a mathematical model from the key point and period information on the single leg standing posture. The single leg standing posture includes a predictive posture control phase, a posture maintaining phase and / or a reactive posture control phase as motion phases. The evaluation unit generates the evaluation result for each motion phase.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Social security costs related to nursing care in Japan continue to rise, and the government is working to transition to a sustainable social security system by promoting support for independence and prevention of serious illness. In the future, the nursing care insurance system will accelerate the shift to an outcome-based remuneration system that emphasizes results, and nursing care facilities will need to improve the quality of rehabilitation in order to achieve results. However, there are few nursing care facilities that employ rehabilitation professionals, and in particular, about 90% of day care facilities do not have rehabilitation professionals, making it difficult to provide "effective rehabilitation." In proposing effective rehabilitation, evaluation to properly grasp the issues and conditions is important, but motion analysis requires the highly specialized skills of rehabilitation professionals.

[0003] In response to such problems in the assessment of rehabilitation, Patent Document 1 discloses a technique for objectively assessing the subject's ability to control center of gravity by calculating the speed of reaction movements until the center of gravity starts to move, and for diagnosing the risk of falling in advance. Patent Document 2 also discloses a technique for taking videos of the subject and estimating the subject's movements and skeletal information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-185557 A [Patent Document 2] JP 2017-080199 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 has a problem in that it is not possible to consider features other than the center of gravity. Also, the technology disclosed in Patent Document 2 still requires a rehabilitation professional to assess the subject's problems and condition appropriately from the estimated subject's movements and skeletal information.

[0006] Therefore, the present disclosure has been made to solve the above-mentioned problems, and its purpose is to provide an information processing device, an information processing method, a program, and an information processing system that can easily and accurately evaluate the motor function of the person being evaluated, suggest effective rehabilitation, and improve the quality of rehabilitation. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device of the present invention comprises a video acquisition unit that acquires an evaluation video that captures the entire body of a person to be evaluated standing on one leg and duration information of the one-leg standing position, a video processing unit that estimates key points including joint points and center of gravity of the person to be evaluated that appear in the evaluation video, and an evaluation unit that generates an evaluation result of the motor function of the person to be evaluated using a mathematical model from the key points and the duration information of the one-leg standing position, wherein the one-leg standing position includes, as operation phases, a predictive postural control phase, a posture maintenance phase and / or a reactive postural control phase, and the evaluation unit generates the evaluation result for each operation phase.

[0008] In order to achieve the above-mentioned object, an information processing method according to the present invention is an information processing method executed by an information processing device having a processor and a memory unit, and causes the processor to execute the steps of acquiring an evaluation video capturing an image of the entire body of a person to be evaluated standing on one leg and information on the duration of the one-legged position, estimating key points including joint points and center of gravity of the person to be evaluated appearing in the evaluation video, and generating an evaluation of the motor function of the person to be evaluated using a mathematical model from the key points and the elapsed time of the one-legged position, wherein the one-legged position includes, as movement phases, a predictive postural control phase, a posture maintenance phase and / or a reactive postural control phase, and the step of generating the evaluation is a step of generating the evaluation result for each movement phase.

[0009] In order to achieve the above-mentioned object, the program of the present invention is a program to be executed on a computer having a processor and a memory unit, the program causing the processor to execute the following steps: acquiring an evaluation video capturing the entire body of a subject standing on one leg and information on the duration of the one-legged position; estimating key points including joint points and center of gravity of the subject appearing in the evaluation video; and generating an evaluation of the motor function of the subject using a mathematical model from the key points and the elapsed time of the one-legged position, the one-legged position including a predictive postural control phase, a posture maintenance phase and / or a reactive postural control phase as movement phases, and the step of generating the evaluation is a step of generating the evaluation result for each movement phase.

[0010] In order to achieve the above-mentioned object, the information processing system of the present invention is an information processing system consisting of an information processing device having a processor and a memory unit, and a user terminal, and causes the processor to execute the steps of acquiring an evaluation video that captures the entire body of the subject standing on one leg and information on the duration of the one-legged position, estimating key points including joint points and center of gravity of the subject appearing in the evaluation video, and generating an evaluation of the motor function of the subject using a mathematical model from the key points and the elapsed time of the one-legged position, wherein the one-legged position includes, as operation phases, a predictive postural control phase, a posture maintenance phase and / or a reactive postural control phase, and the step of generating the evaluation is a step of generating the evaluation result for each operation phase. Effect of the Invention

[0011] According to the present disclosure using the above means, it is possible to easily and accurately evaluate the motor function of the person being evaluated, suggest effective rehabilitation, and improve the quality of rehabilitation. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an overall configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of an information processing server. [Diagram 3] FIG. 4 is a diagram illustrating an example of information stored in a storage unit of the information processing server. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of a user terminal. [Diagram 5] FIG. 13 is a diagram showing an example of measuring motor function using an information processing system. [Figure 6] 13A and 13B are diagrams illustrating an example of a guide displayed when capturing a moving image. [Figure 7] 13 is a flowchart showing the operation of a motor function evaluation process executed in the information processing system. [Figure 8] 13 is a flowchart showing the operation of an inference process. [Figure 9] FIG. 13 is a diagram showing an example of correspondence between an evaluation subject and estimated key points. [Figure 10] FIG. 13 is a diagram illustrating an example of an evaluation report. [Figure 11] FIG. 13 is a diagram showing an example of a list display of a presented exercise menu. [Figure 12] FIG. 13 is a diagram showing an example of the contents of a presented exercise menu. [Figure 13] FIG. 1 is a block diagram showing a basic hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] <System configuration> The information processing system 1 in the present disclosure is an information processing system that provides a motor function evaluation service that films a subject performing a specified functional movement related to motor function, evaluates the subject's motor function based on the filmed video, and suggests an exercise menu for improving the subject's motor function based on the evaluation results.

[0015] The subjects of the evaluation are mainly elderly people. The users of the motor function evaluation service are mainly businesses that provide nursing care and assistance services to the subjects of the evaluation, local governments, individuals, etc. In the following explanation, the users are described as nursing care businesses that provide nursing care services.

[0016] 1 is a diagram showing an overall configuration of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 includes information processing devices, an information processing server 10, and a user terminal 20, which are connected via a network NW.

[0017] Each information processing device is configured by a computer having an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later.

[0018] The network NW is, for example, a network such as the Internet, a VPN (Virtual Private Network), an intranet, or a short-distance wireless communication. For the sake of simplicity, one user terminal 20 is shown in FIG. 1, but the information processing server 10 can be connected to two or more user terminals 20 via the network NW. In addition, one user terminal 20 may be used by multiple users or evaluation subjects.

[0019] An operator of a motor function evaluation service provides a motor function evaluation service to a user using an information processing system 1. Specifically, the information processing system 1 provides an application that runs on a user terminal 20 used by the user. The user uses the application to measure the motor function of the person to be evaluated. In the measurement, the user captures an image of the person to be evaluated performing a predetermined functional movement related to the motor function, and transmits the captured evaluation video and information on the person to be evaluated to the information processing server 10. The information processing server 10 evaluates the motor function of the person to be evaluated based on the received evaluation video, etc. In addition, the information processing server 10 selects an appropriate exercise menu based on the evaluation result. The evaluation result and the selected exercise menu are transmitted to the user terminal 20. The motor function includes balance ability, walking ability, functional mobility, flexibility, whole-body coordination, whole-body durability, muscle strength, etc. The balance ability includes static balance ability and dynamic balance ability. In this embodiment, balance ability is evaluated as the motor function.

[0020] FIG. 2 is a block diagram showing the functional configuration of the information processing server 10. As shown in FIG. The information processing server 10 (hereinafter simply referred to as the server 10) is an information processing device that provides a motor function evaluation service. The server 10 includes a storage unit 101 and a control unit 103.

[0021] <Configuration of the storage unit of the information processing server> The storage unit 101 of the server 10 is a storage medium or storage device that stores information used to execute management by the information processing system 1. Although the storage unit 101 is illustrated as being integrated with the server 10 in FIG. 2, it may be a storage device that is physically independent from the server 10. The storage unit 101 includes an application program 1011, a video table 1012, an evaluation table 1013, an exercise information table 1014, a user table 1015, an evaluation subject table 1016, a video analysis mathematical model 1021, an evaluation mathematical model 1022, and a proposed mathematical model 1024. The storage unit 101 also stores other information that does not belong to these storage units.

[0022] 3 is a diagram showing an example of information stored in the storage unit 101 of the information processing server 10. Each table in the storage unit 101 will be described below with reference to FIG.

[0023] The video table 1012 is a table that stores and manages evaluation videos acquired by the video acquisition unit 1031. The video table 1012 includes the following items of information to be stored: video ID, video data, basic video information, creating user, imaging subject information, converted video, and evaluated video.

[0024] The moving image ID is an item for storing moving image identification information for identifying an evaluation moving image. The moving image identification information is an item in which a unique value is set for each evaluation moving image. The moving image ID is generated for each evaluation moving image obtained. The video data is an item for storing the file body of the evaluation video. The file body may be stored in another server, and the path to the file may be stored. The video basic information is an item for storing basic information about the evaluation video, such as the shooting date and time, file format, compression method, capacity, resolution, frame rate, bit rate, shooting time, and the like. The creating user is an item for storing the user ID of the user who shoots the evaluation video. This allows the evaluation video to be linked to the user. The imaging subject information is an item for storing imaging subject information, which is information relating the evaluation subject, who is the subject of the evaluation video, and the evaluation video. The imaging subject information includes a functional operation type, a total operation period Ta, and an evaluation subject ID, which will be described later. By including the evaluation subject ID, the evaluation video and the evaluation subject can be linked. The functional operations included in the functional operation type are, for example, a Timed UP&Go Test, a Functional Reach Test, tandem walking, a one-legged standing test, and a Berg Balance Scale. Each functional operation is associated with a specific motor function. Specifically, the Timed UP&Go Test is associated with muscle strength, balance ability, walking ability, and functional mobility ability, the Functional Reach Test is associated with dynamic balance ability, the tandem walking is associated with dynamic balance ability, the one-legged standing test is associated with static balance ability, and the Berg Balance Scale is associated with balance ability, muscle strength, endurance, flexibility, and interval, respectively. That is, the Timed UP&Go Test, Functional Reach Test, Tandem Walking, One-Leg Standing Test, and Berg Balance Scale are all related to balance ability and can be used individually or in combination as tests to determine the risk of falls in elderly people. The total movement period Ta is time period information that indicates the period (time) from the start to the end of such functional movements as a numerical value. The converted moving image is an item for storing a converted moving image generated by executing preprocessing on the evaluation moving image acquired by the moving image acquisition unit 1031. The evaluated moving image is an item for storing and managing the evaluated moving image generated by the inference process of the evaluation unit 1033. The evaluated moving image includes a report moving image, which will be described later.

[0025] The evaluation table 1013 is a table that stores and manages evaluation information related to the analysis and evaluation of the motor function of the evaluation subject. The evaluation table 1013 stores the following items of information: evaluation information ID, analysis target moving image, joint point information, phase period information, evaluation information, and report information. The evaluation information ID is an item for storing evaluation identification information for identifying evaluation information. The evaluation identification information is an item in which a unique value is set for each piece of evaluation information. Evaluation information is generated for each measurement of motor function. The video to be analyzed is an item in which the video ID of the evaluation video to be analyzed is stored. This links the evaluation information to the evaluation video. When the functional operation type is "standing on one leg," two video IDs can be stored: one for an evaluation video in which the left leg is used as the pivot foot, and the other for an evaluation video in which the right leg is used as the pivot foot. The joint point information is a table that stores and manages pairs of key points and coordinates generated by the inference process of the evaluation unit 1033 described later for each frame of the converted video, in association with the frame. The key points include joint points and centers of gravity. The correspondence between the human body and the inferred key points will be described later. The phase period information is an item for storing period information of the converted moving image to be analyzed. The period information of the converted moving image includes a delimiter position and an n-th phase period Tn, which will be described later. In FIG. 3, the first phase period T1 to the third phase period T3 are shown as the n-th phase period Tn when the functional operation type is one-legged standing. The evaluation information is an item for storing evaluation information that is a result of analyzing the period information and the converted video. The evaluation information includes a total movement period judgment result, a phase period judgment result, a phase evaluation result, a phase part evaluation result, a compensatory movement evaluation result, and an overall evaluation, which will be described later. The evaluation report information is an item for storing evaluation report information as information related to the evaluation report described later. The evaluation report information includes a report video described later, feedback data described later, and suggested exercise menu information described later.

[0026] The exercise information table 1014 is an item that stores information about an exercise menu to be proposed according to the evaluation result of the motor function of the evaluation subject. The exercise information table 1014 includes items of exercise information ID, basic exercise information, and exercise content as items of information to be stored. The exercise menu is divided into two categories: balance exercises and exercises to improve muscle strength and flexibility. The exercises to improve muscle strength and flexibility include exercises for the trunk, hip joints, ankles, and feet. Examples of the exercise menu will be described later.

[0027] The exercise information ID is an item for storing exercise identification information for identifying an exercise menu. The exercise identification information is an item for which a unique value is set for each exercise menu. The basic exercise information is an item for storing basic information about the exercise menu, such as the name of the exercise, the target body part, the difficulty level, the number of times or sets to be performed, points to note, and other notes. The exercise content is an item that stores content that explains the details of the exercise, which is composed of images, explanatory text, video, explanatory audio, or a combination thereof, related to the exercise menu.

[0028] The user table 1015 is a table that stores and manages information on users (user members) who use the motor function evaluation service. The user table 1015 includes user ID and user basic information as items of information to be stored. When a user registers to use the service, the user's information is stored in a new record in the user table 1015. This allows the user to use the motor function evaluation service according to the present disclosure. The user ID is an item for storing user identification information for identifying a user. The user identification information is an item for which a unique value is set for each user. The user basic information is an item for storing basic information such as the user's name, address, contact information, and password.

[0029] The evaluation subject table 1016 is a table that stores and manages information on evaluation subjects who are to undergo a motor function evaluation using the motor function evaluation service. The evaluation subject table 1016 includes, as items of information to be stored, an evaluation subject ID, evaluation subject basic information, a management user, and evaluation history. Information on evaluation subjects is mainly registered by users of the motor function evaluation service. In this embodiment, a care provider registers information on evaluation subjects, and links information on evaluation subjects registered in other systems or services used by the care provider. The evaluation target ID is an item for storing evaluation target identification information for identifying the evaluation target. The evaluation target identification information is an item for which a unique value is set for each evaluation target. The basic information of the subject of evaluation includes items such as the subject's name, date of birth, age, sex, height, weight, level of care required, health condition including medical history, physical and mental functions, ADL / IAIL, and environmental and personal factors (use of welfare equipment, home environment). The management user is an item that stores the user ID of the care provider to which the person to be evaluated belongs. This allows the information of the person to be evaluated to be linked to the information of the user. The evaluation history is an item in which the evaluation information ID is stored as the evaluation history by the information processing system 1. This makes it possible to link the motor function evaluation result with the evaluation subject.

[0030] Returning to FIG. 2, each mathematical model in the storage unit 101 will be described.

[0031] The moving image analysis mathematical model 1021 is a mathematical model that uses moving images as input data and estimates joint points, center of gravity, skeleton, posture, etc. of a human body or animal that appears in the moving image. Details of moving image processing using the moving image analysis mathematical model 1021 will be described later.

[0032] The video analysis mathematical model 1021 of this embodiment is a deep learning model using a deep neural network in deep learning. The video analysis mathematical model 1021 can be realized using a general-purpose skeleton estimation model that has undergone deep learning. Examples of the general-purpose skeleton estimation model that can be used include Openpose, PoseNet, AlphaPose, HRNet, MMPose, and BlazePose. The video analysis mathematical model 1021 does not necessarily have to be a deep learning model, and may be any machine learning or artificial intelligence model. The video analysis mathematical model 1021 may be generated for each functional operation, gender, age, level of care required, and health condition.

[0033] The evaluation mathematical model 1022 is a mathematical model that uses key points estimated for the converted video as input data and outputs the quality of the functional movement performed by the subject of evaluation appearing in the converted video. Details of the inference process using the evaluation mathematical model 1022 will be described later. The evaluation mathematical model 1022 is, for example, a type of machine learning, artificial intelligence, deep learning model, rule-based, or a combination thereof. The evaluation mathematical model 1022 does not need to be a single mathematical model, and may be realized by combining or switching between multiple independent types of mathematical models. The evaluation mathematical model 1022 may be generated for each motor function or functional movement to be evaluated, gender, age, level of care required, and health condition.

[0034] The proposed mathematical model 1023 is a mathematical model that receives period information and evaluation results as input data, selects one or more exercise menus, and outputs them. The proposed mathematical model 1023 is, for example, a type of machine learning, artificial intelligence, deep learning model, rule-based, or a combination thereof. The proposed mathematical model 1023 does not need to be a single mathematical model, and may be realized by combining or switching between multiple independent types of mathematical models. The proposed mathematical model 1023 may be generated for each motor function or functional movement to be evaluated, gender, age, level of care required, and health condition.

[0035] <Configuration of the control unit of the information processing server> The control unit 103 of the server 10 includes, as functional units, a video acquisition unit 1031, a video processing unit 1032, an evaluation unit 1033, an exercise suggestion unit 1034, an evaluation result output unit 1035, and a mathematical model management unit 1036. The control unit 103 is a computer equipped with a CPU. The control unit 103 executes an application program 1011 stored in the storage unit 101 to realize each functional unit. The control unit 103 executes a motor function evaluation process described later by linking each functional unit. Although not shown, the control unit 103 also includes a function of controlling communication with an external device connected to the server 10 via a network NW or the like.

[0036] The video acquisition unit 1031 acquires, from a user terminal 20 connected to the server 10 via the network NW, an evaluation video capturing an image of a person to be evaluated performing a specified functional operation, and duration information of the specified functional operation.

[0037] The video processing unit 1032 preprocesses the evaluation video and stores the preprocessed video as a converted video in the storage unit 101. In addition, the video processing unit 1032 estimates key points, including joint points and a center of gravity, for the person to be evaluated appearing in the converted video.

[0038] The evaluation unit 1033 executes an inference process to infer the evaluation results of the predetermined functional actions and the motor functions related to the predetermined functional actions of the person to be evaluated from the key points and the period information of the predetermined functional actions using a mathematical model, and stores the inference results as evaluation information in the storage unit 101. Details of the inference process will be described later.

[0039] The exercise suggestion unit 1034 selects an exercise menu suitable for improving, maintaining, and enhancing the motor function of the person to be evaluated, based on the period information of the predetermined functional operation and the evaluation information.

[0040] The evaluation result output unit 1035 generates evaluation report information based on the evaluation information and the selected exercise menu, and outputs it to the user terminal 20.

[0041] The mathematical model management unit 1036 generates and updates each mathematical model in the storage unit 101. Note that the generation of a mathematical model also includes the meaning of learning.

[0042] 4 is a block diagram showing the functional configuration of the user terminal 20. The user terminal 20 includes a storage unit 201, a control unit 203, an input device 205, and an output device 207 as functional units. The user terminal 20 is an information processing device having an imaging function, which is operated by a user who uses the service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or a tablet, a stationary PC (Personal Computer), or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal.

[0043] <Configuration of the storage unit of the user terminal> The storage unit 201 of the user terminal 20 is a storage medium or storage device that can store information used for providing the motor function evaluation service by the information processing system 1. Although it is illustrated as being integrated with the user terminal 20 in FIG. 4, it may be a storage device that is physically independent from the user terminal 20. The storage unit 201 includes an application program 2011 and a user ID 2012. The storage unit 201 also stores other information that does not belong to these storage units.

[0044] The application program 2011 may be stored in advance in the storage unit 201, or may be downloaded from a web server operated by a service provider via a communication IF. The application program 2011 includes an application such as a web browser application. The application program 2011 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the user terminal 20.

[0045] The user ID 2012 stores the user ID of the user using the user terminal 20 as identification information for the user to use the motor function evaluation service. The user transmits the user ID from the user terminal 20 to the server 10. The server 10 identifies the user by consulting the user table 1015 with the received user ID, and provides the user with the service according to the present disclosure. The user ID 2012 may include information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20.

[0046] <Configuration of the control unit of the user terminal> The control unit 203 of the user terminal 20 includes an input control unit 2031 and an output control unit 2032. The control unit 203 is a computer equipped with a CPU. The control unit 203 realizes each functional unit by executing an application program 2011 stored in the storage unit 201. Although not shown, the control unit 203 also includes a function of controlling communication with an external device connected to the user terminal 20 via a network NW or the like.

[0047] The input control unit 2031 controls the input device 205 . The output control unit 2032 controls the output device 207 .

[0048] The input device 205 of the user terminal 20 includes a camera 2051, a microphone 2052, a position information sensor 2053, a motion sensor 2054, and a touch device 2055. The input device 205 is a type of input device that allows the user to input or select information by operating it. The input and selection of information include, for example, input by entering numerical values, characters, or symbols, selection of an item, input to determine processing, and the like.

[0049] The camera 2051 is, for example, a visible light camera, and is an input device having a function of detecting light reflected from a subject and generating image (still image or video) information. The camera 2051 may be built into the user terminal 20, or may be an independent camera connected to the user terminal 20.

[0050] The microphone 2052 is an input device having a function of collecting sounds generated near the installation location, converting them into electrical signals, and generating sound information. For example, the microphone 2052 may be used to accept voice input of instructions from a user.

[0051] The position information sensor 2053 is an input device having a function of generating position information of the user terminal 20 .

[0052] The motion sensor 2054 has a function of generating posture information of the user terminal 20 by detecting and tracking the movement of the user terminal 20 .

[0053] The touch device 2055 is an input device having a function of generating operation information of an operation performed on the screen by a user of the user terminal 20 using a finger, a stylus pen, or the like.

[0054] The output device 207 of the user terminal 20 includes a display 2071 and a speaker 2072. The output device 207 is capable of displaying, reproducing, and notifying the user of information and the like.

[0055] The display 2071 is a display output device that displays characters, figures, images, videos, etc. The display 2071 may be a display output device independent of the user terminal 20, or may be a display output device such as a liquid crystal display or an organic EL display in a smartphone or tablet.

[0056] The speaker 2072 is an audio output device that converts an electrical signal into sound and outputs audio to the surroundings. The speaker 2072 may be built into the user terminal 20, or may be an independent audio output device connected to the user terminal 20.

[0057] <Measurement procedure> Fig. 5 is a diagram showing an example of measuring motor function using an information processing system according to an embodiment of the present disclosure. Fig. 6 is a diagram showing an example of a guide displayed when capturing an evaluation video. The procedure for measuring motor function, i.e., a specific method for capturing an evaluation video, will be described below with reference to Figs. 5 and 6.

[0058] As shown in FIG. 5, the user fixes the photographing position and photographing posture of the user terminal 20 to a predetermined position and posture using a tripod or the like. The user selects (inputs) and transmits, as information about the person to be evaluated, the registered identification information (such as name) of the person to be evaluated and the type of functional operation to be evaluated, on the display 2071 of the user terminal 20. Then, the user guides the person to be evaluated to a position (hereinafter referred to as the operation position) that is a predetermined distance away from the user terminal 20. At this time, the user may use a guide shown in FIG. 6 that is displayed when the video for evaluation is captured. For example, the user can follow the guide shown in FIG. 6 and set a position where the whole body of the person to be evaluated fits within a predetermined imaging area as the operation position where the person to be evaluated is to be made to perform an operation.

[0059] When the person to be evaluated is in a standby state at the operating position, the user starts imaging on the user terminal 20 and instructs the person to be evaluated to start the functional operation of the type selected on the display 2071. The person to be evaluated starts the functional operation at his / her own timing. The user times the period from the start to the end of the functional operation of the person to be evaluated using a stopwatch or the like. When the functional operation type is "standing on one leg," an evaluation video is captured for each of the left leg and right leg, and the total operation period Ta is timed.

[0060] When a predetermined time has elapsed since the start of video capture, the capture automatically stops. If the state of the functional operation of the person being evaluated meets a predetermined end condition before the predetermined time has elapsed, the user stops the capture on the user terminal 20 and transmits the evaluation video to the server 10. The user inputs and transmits the timing result as the total operation period Ta on the display screen of the user terminal 20. Note that at this timing, the functional operation type and the identification information of the person being evaluated may be selected and transmitted together. The predetermined end conditions are, when the functional operation type is one-legged standing, (1) the raised foot touches the floor or the opposite foot, (2) the supporting foot shifts, (3) the hand touches a handrail or wall, or (4) the measurement time exceeds 60 seconds.

[0061] <Motor function evaluation processing> FIG. 7 is a flowchart showing the operation of the motor function evaluation process. The motor function evaluation process is a series of processes for obtaining an evaluation video of the subject performing a predetermined functional movement related to the motor function, evaluating the motor function of the subject based on the obtained evaluation video, selecting an appropriate exercise menu based on the evaluation result, and providing the evaluation result and the exercise menu. In the following explanation, it is assumed that "standing on one leg" is selected as the predetermined functional movement to evaluate balance ability as a motor function. The motor function evaluation process includes a video acquisition step, a subject information acquisition step, a video pre-processing step, an inference processing step, a report video generation step, a feedback data generation step, a motion judgment step, and an evaluation result output step. The operation of the motor function evaluation process will be described below with reference to this flowchart.

[0062] In step S101, the moving image acquisition unit 1031 of the server 10 acquires an evaluation moving image from the user terminal 20 (moving image acquisition step). Specifically, the video acquisition unit 1031 acquires, from the user terminal 20 , an evaluation video of the evaluation subject performing a predetermined functional operation, and stores the video in the video table 1012 of the storage unit 101 .

[0063] In step S102, the video acquisition unit 1031 of the server 10 acquires image capture subject information from the user terminal 20 (image capture subject information acquisition step). Specifically, the video acquisition unit 1031 acquires from the user terminal 20, as imaging subject information, identification information of the subject to be evaluated (the subject to be evaluated ID or information linked to the subject to be evaluated ID) for identifying the subject to be evaluated, the functional operation type entered by the user, and the total operation period Ta of that functional operation, and stores them in the video table 1012 of the memory unit 101.

[0064] In step S103, the video processing unit 1032 of the server 10 performs pre-processing of the evaluation video acquired in step S101 (video pre-processing step). Specifically, the video processing unit 1032 performs video conversion as pre-processing to set the video to a predetermined frame rate and a predetermined format, and stores the converted video in the video table 1012 of the storage unit 101.

[0065] In step S104, the evaluation unit 1033 of the server 10 executes an inference process based on the converted moving image generated in step S103 to generate an evaluation result (inference process). The inference process will be described in detail later.

[0066] In step S105, the video processing unit 1032 of the server 10 generates a report video based on the converted video generated in step S103 and the evaluation result in step S104, and stores the report video in the evaluation report information item of the evaluation table 1013 in the storage unit 101 (report video creation process). The report video may be either a video or a still image. Two types of report videos are generated: one for the user and one for the evaluation target. The report video for the user is mainly information to be shared with rehabilitation professionals, and is mainly an extracted partial video or frame image related to the evaluation result, and key points are superimposed and displayed. In this way, by adding key points to the report video for the user, it becomes possible to request a detailed motion analysis from a rehabilitation professional (physiotherapist, occupational therapist). The report video for the evaluation target is mainly information to be shared with the evaluation target and his / her family and caregiver, and is displayed together with key points in the evaluation report described later. This will facilitate understanding of the posture during measurement (when video was taken) and the evaluation results of the subjects who have been provided with the evaluation report, thereby improving the quality of rehabilitation suggestions and the motivation of the subjects.

[0067] In step S106, the evaluation unit 1033 of the server 10 analyzes the motor function of the person to be evaluated based on the evaluation result (evaluation information) of step S104 and the past evaluation results (evaluation information) stored in the storage unit 101, generates feedback data, and stores it in the evaluation report information item of the evaluation table 1013 in the storage unit 101 (feedback data creation process). For example, a rule base or a general-purpose text generation AI model can be used to generate the feedback data. The feedback data includes information corresponding to each item displayed in the evaluation report. An example of an evaluation report when the functional operation type is "standing on one leg" will be described later.

[0068] In step S107, the exercise suggestion unit 1034 of the server 10 executes exercise judgment based on the evaluation result in step S104 (exercise judgment step). Specifically, the exercise suggestion unit 1034 applies the evaluation result of step S104 as input data to the proposed mathematical model 1023, acquires (selects) one or more exercise menus as output data as proposed exercise menu information, and stores the output data in the report information item of the evaluation table 1013 in the storage unit 101. It is preferable that at least one output data is selected from the balance exercise category. Additionally, past evaluation results may be added as input data.

[0069] In step S108, the evaluation result output unit 1035 of the server 10 outputs an evaluation report based on the report information stored in the evaluation table 1013 of the storage unit 101 (evaluation result output step).

[0070] <Inference processing> The inference process is mainly a process for generating an evaluation result by analyzing the moving image and evaluating the motor function of the evaluation subject. The inference process is executed for each measurement. FIG. 8 is a flowchart showing the operation of the inference process. The inference process is a series of processes including a joint point estimation process, a motion phase inference process, a total motion period determination process, a phase period determination process, a phase evaluation process, a phase part evaluation process, a compensation evaluation process, and an evaluation result integration process. In the following explanation, it is assumed that "one-leg standing" is selected as the predetermined functional motion. The operation of the inference process will be described below with reference to this flowchart.

[0071] In step S201, the video processing unit 1032 of the server 10 executes a joint point estimation process for each frame of a converted video (joint point estimation step). If two or more videos are shot in one measurement, the joint point estimation step is executed for each video. Specifically, the video processing unit 1032 applies each frame image of the converted video as input data to the video analysis mathematical model 1021, obtains sets of key points and their coordinates (hereinafter simply referred to as key point coordinates, joint point coordinates, and center of gravity coordinates) estimated for each frame as output data, and stores them in the joint point information item of the evaluation table 1013 of the storage unit 101 of the server 10 in association with the identification information of each frame. Note that the center of gravity may be calculated separately from the coordinates of the joint points. 9 is a diagram showing an example of correspondence between a human body (evaluation subject) shown in a video and estimated key points (joint points and center of gravity). With reference to FIG. 9, the key points estimated by the inference process will be described. In this embodiment, the coordinates of each key point shown in Fig. 9 can be estimated for each frame by the video analysis mathematical model 1021. Note that the numbers displayed near each key point in Fig. 9 are used only to indicate that they correspond to the numbers in the list of key points in the same figure, and are unrelated to the reference numbers in other figures. In FIG. 9, 33 points are indicated by black circles as estimated key points. The human body is roughly divided into the head and neck (area indicated by dashed lines in FIG. 9), the trunk (not shown), and the limbs (not shown). The limbs are divided into the upper limbs and the lower limbs (not shown). The key points belonging to the head and neck region are the nose, the inner edge of the left eye, the left eye, the outer edge of the left eye, the inner edge of the right eye, the right eye, the outer edge of the right eye, the left ear, the right ear, the left edge of the mouth, and the right edge of the mouth. The key points belonging to the trunk region are the left shoulder, the right shoulder, the left hip, and the right hip. The key points belonging to the upper limb region are the left elbow, the right elbow, the left wrist, the right wrist, the left little finger, the right little finger, the left index finger, the right index finger, the left thumb, and the right thumb. The key points belonging to the lower limb region are the left knee, the right knee, the left ankle, the right ankle, the left heel, the right heel, the left index finger, and the right index finger.

[0072] In step S202, the video processing unit 1032 of the server 10 divides the converted video into motion phases based on the time change of the key point coordinates acquired in step S201 (motion phase inference step). If two or more videos are taken in one measurement, the motion phase inference step is executed for each converted video. A functional operation includes multiple operation phases from the start to the end of the functional operation, and transitions occur between the operation phases over time. The number of included operation phases, the coupling relationships between the operation phases, and the transition directions vary depending on the type of functional operation. When the functional operation type is "standing on one leg," there are three operation phases: Phase 1, Phase 2, and Phase 3, and as time passes, the first phase transitions to Phase 3. In this case, the combination patterns of operation phases that can be included in one video (converted video) are 1) only Phase 1, 2) Phase 1 and Phase 2, and 3) Phase 1, Phase 2, and Phase 3. Specifically, the video processing unit 1032 applies the time transition of the coordinates of one or more predetermined key points in all frames of the converted video as input data to the video analysis mathematical model 1021, and obtains the delimiter position (time or frame number) for each estimated operation phase as output data. In this process, the number of the one or more predetermined key points is preferably 3 to 8. When the functional operation type is "single-leg standing", it is preferable to include two or more key points in the lower limb area. In addition, the video processing unit 1032 calculates the phase period of each operation phase based on the delimiter positions. Furthermore, the video processing unit 1032 stores the estimated delimiter positions and the calculated phase periods in the phase period information item of the evaluation table 1013 of the storage unit 101. A phase period is the period (time) elapsed from the start to the end of an operation phase, and when the functional operation type is "standing on one leg", it is the first phase period T1, the second phase period T2, and the third phase period T3. For simplicity, in the following description, n is a natural number, the phase period corresponding to the nth phase is denoted as the nth phase period Tn, and the video of the period corresponding to the nth phase is denoted as the nth phase video. If the functional operation type is "standing on one leg" and the video contains all the above three operation phases, in step S202, the video processing unit 1032 acquires the division position between the first phase and the second phase, the division position between the second phase and the third phase, the first phase period T1, the second phase period T2, and the third phase period T3, and stores them in the phase period information item of the evaluation table 1013 in the storage unit 101. The phase period information to be acquired and stored is determined according to the combination pattern described above. In the following specific examples of steps, a case where the video contains all the three phases will be described.

[0073] In step S203, the evaluation unit 1033 of the server 10 judges whether the total movement period Ta acquired from the user terminal 20 in step S101 of the motor function evaluation process is good or bad (total movement period judgment process). When two or more evaluation videos are shot in one measurement, the total movement period judgment process is executed for the total movement period Ta acquired for each evaluation video. Specifically, the evaluation unit 1033 judges the total operating period Ta to be "good" if it is equal to or greater than the standard (average) value Taa for the same age and sex, to be "passable" if it is equal to or greater than a predetermined cutoff value Tac, and to be "caution" if it is less than the predetermined cutoff value Tac. The evaluation unit 1033 stores the judgment result as an entire-period operation judgment result in the item of evaluation information in the evaluation table 1013 in the storage unit 101. The entire-period operation judgment may be a two-level pass / fail evaluation or a four-level or more step-wise evaluation.

[0074] In step S204, the evaluation unit 1033 of the server 10 judges the pass / fail of each n-th phase period Tn calculated in step S202 (phase period judgment process). If two or more evaluation videos are shot in one measurement, the phase period judgment process is executed for each phase period calculated for each evaluation video. Specifically, the evaluation unit 1033 judges the n-th phase period Tn as "good" if it is equal to or greater than the standard (average) value Tna for the same age and sex, as "fair" if it is equal to or greater than a predetermined cutoff value Tnc, and as "caution" if it is less than the predetermined cutoff value Tnc. The evaluation unit 1033 then stores the judgment result in the item of phase period judgment result in the evaluation table 1013 of the storage unit 101. When the functional operation type is "standing on one leg", the evaluation unit 1033 judges the first phase period T1 as "good" if it is equal to or greater than the standard (average) value T1a for the same age and sex, as "fair" if it is equal to or greater than a predetermined cutoff value T1c, and as "caution" if it is less than the predetermined cutoff value T1c. The second phase period T2 and the third phase period T3 are judged in the same manner. In addition, judgments are made for the left leg and the right leg respectively. The evaluation unit 1033 stores the judgment results for the left leg and the right leg in the item of evaluation information in the evaluation table 1013 of the storage unit 101, for example, as "first phase left leg judgment result" and "second phase right leg judgment result". The phase period operation judgment may be a two-level good / bad evaluation, or a four-level or more gradual evaluation.

[0075] In step S205, the evaluation unit 1033 of the server 10 infers the quality of the evaluation subject's movement in each movement phase based on the key point coordinates acquired in step S201 (phase evaluation step). If two or more videos are taken in one measurement, the phase evaluation step is performed for each video. Specifically, the evaluation unit 1033 calculates, for a predetermined part, physical quantities including coordinate positions, statistical values ​​of the physical quantities, and calculation results of the physical quantities and / or statistical values ​​as features using the keypoint coordinates of each frame of the n-th phase moving image. The evaluation unit 1033 applies the calculated features as input data to the evaluation mathematical model 1022, and obtains the quality of the inferred motion as output data. The evaluation unit 1033 then associates the quality of the inferred motion with the motion phase and stores it in the phase evaluation result field of the evaluation table 1013 in the storage unit 101. The predetermined parts in this step are selected from the above-mentioned 33 key points and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb), and the number of selected parts is preferably in the range of 3 to 8. The selected predetermined parts are set and registered in the server 10 in advance for each motion phase. The physical quantity of a specific part includes a position, a distance, and an angle. The angle may be a keypoint angle, which is an angle formed by two lines connecting a certain keypoint with two adjacent keypoints. For example, the keypoint angle of the right knee refers to the angle formed by a line connecting the right knee with the right hip and a line connecting the right knee with the right ankle. The angle may also be a quantitative angle, which is an angle formed by two lines connecting a certain region with two adjacent regions. For example, the trunk angle refers to the angle formed by a line connecting the center of gravity with the center of the head and neck (or the nose) and a line connecting the center of gravity with the tip of one lower limb (or the ankle). Hereinafter, the keypoint angle and the region angle are collectively referred to as a part angle. The statistical value of the physical quantity includes a movement amount, a movement width, an average value, a maximum value, and a minimum value. The calculation includes the four arithmetic operations. The feature quantity calculated for the selected specific part is set and registered in the server 10 in advance for each motion phase. The feature amount is defined by selecting from a predetermined part, physical amount, statistical value, or calculation, such as "amount of movement of the center of gravity within the period - width of movement of the center of gravity within the period." When the functional movement type is "standing on one leg," the first phase is also called the predictive postural control phase, the second phase the postural maintenance phase, and the third phase the reactive postural control phase. The predictive postural control phase is the period in which the central nervous system predicts and adjusts the activity of the sensory and motor systems by using previous experience to determine the posture required for the upcoming movement based on the information seen with the eyes. The postural maintenance phase is the period in which the center of gravity movement converges and the head and upper trunk are kept in the midline position. The reactive postural control phase is the period in which information is obtained from the somatosensory, visual, and vestibular systems during movement to restore stability when something unexpected occurs during movement, and the posture is adjusted to a better position so that there is no deviation from the planned movement. In the evaluation report, these are expressed in other words as "smoothness," "stability," and "strength." In the evaluation report, "smoothness" is an index that mainly indicates the ability to shift the center of gravity smoothly. This index shows whether the user can control the balance when walking, stepping over, or performing other intended movements. In addition, "stability" is an index showing the ability to maintain balance stably. How to maintain balance varies depending on the person being evaluated. Ideally, you should be able to balance with your hip joints and ankles, but if this is difficult, you may compensate by using your trunk, arms, or raised leg to maintain balance. "Bearing strength" is an index showing the ability to brace oneself when one loses balance. This index shows whether one can react and deal with situations in daily life where one loses balance. As with "stability," it is ideal to brace oneself with the hip joints and ankles, but if this is difficult, one may compensate by bracing oneself with the core, arms, or raised leg. When the functional operation type is "standing on one leg", it is preferable to use the statistical value of one or more part angles or the calculation result thereof as input data for the first phase. Similarly, it is preferable to use the statistical value of the center of gravity or the calculation result thereof and the statistical value of one or more part angles or the calculation result thereof as input data for the second phase. Similarly, it is preferable to use the statistical value of multiple part angles or the calculation result thereof as input data for the third phase.

[0076] In step S206, the evaluation unit 1033 of the server 10 infers whether the movement of the specific part of the subject in each movement phase is good or bad based on the key point coordinates acquired in step S201 (phase-part evaluation process). If two or more videos are taken in one measurement, the phase evaluation process is performed for each video. Specifically, the evaluation unit 1033 uses the keypoint coordinates of each frame of the n-th operation phase video to calculate, for a specific part, physical quantities including coordinate positions, statistical values ​​of the physical quantities, and calculation results of the physical quantities and / or statistical values ​​as features. The evaluation unit 1033 then applies the calculated features as input data to the evaluation mathematical model 1022, and obtains the quality of the inferred operation of the specific part as output data. The evaluation unit 1033 then associates the quality of the inferred operation of the specific part with the operation phase and stores it in the item of phase-part evaluation result in the evaluation table 1013 of the storage unit 101. The predetermined site in this step is different for each specific site to be evaluated. The predetermined site in this step is selected from the above-mentioned 33 key points and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb), and the number of selected regions is preferably in the range of 2 to 8. The selected predetermined site is set and registered in advance in the server 10 for each specific site to be evaluated. When the functional movement type is "standing on one leg", the movements of the specific parts "trunk", "hip joint" and "ankle" are evaluated as good or bad. As input data, it is preferable to use one or more part angle statistics and center of gravity statistics, or the calculation results thereof. As input data for the evaluation of the "ankle", physical quantities, statistics and calculation results of key points of the lower limbs may be excluded.

[0077] In step S207, the evaluation unit 1033 of the server 10 evaluates the presence or absence of compensatory movements in the functional movements of the person to be evaluated based on the converted moving images (compensatory evaluation process). The compensatory movements include upper limb compensatory movements and lower limb compensatory movements. The evaluation is performed for each movement phase. If two or more moving images are taken in one measurement, a total movement period determination process is performed for each converted moving image corresponding to each moving image. Specifically, the evaluation unit 1033 calculates, for a predetermined part, physical quantities including coordinate positions, statistics of the physical quantities, and calculation results of the physical quantities and / or the statistics as features, using the keypoint coordinates of each frame of the n-th operation phase video. The evaluation unit 1033 applies the calculated features as input data to the evaluation mathematical model 1022, and obtains the presence or absence of an estimated upper limb compensatory movement and / or lower limb compensatory movement as output data. The evaluation unit 1033 then stores the presence or absence of the obtained compensatory movement in the evaluation result item of the evaluation table 1013 of the storage unit 101 of the server 10 as an n-phase compensatory movement evaluation result. When the functional movement type is "standing on one leg", the evaluation unit 1033 first applies the feature amount for a predetermined part of the first phase video as input data to the evaluation mathematical model 1022, obtains the presence or absence of an estimated upper limb compensatory movement and / or lower limb compensatory movement as output data, and stores it as a first phase compensatory movement evaluation result in the evaluation result item of the evaluation table 1013 in the storage unit 101 of the server 10. The same is performed for the second phase and the third phase. The evaluation is also performed for each of the left leg and the right leg. The specified areas in this process are selected from the 33 key points mentioned above and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb). The number of specified areas selected is preferably in the range of 2 to 5, and is preset for each upper limb compensatory movement and lower limb compensatory movement. Examples of the feature amount of the upper limb compensatory movement include the amount of movement of a predetermined part selected from the upper limb region, and statistics of part angles of a plurality of predetermined parts selected from the upper limb region. Examples of the feature amount of the compensatory movement of the lower limbs include a statistical value of the center of gravity coordinates, a statistical value of a plurality of part angles, etc. The feature amount of the compensatory movement of the lower limbs may exclude key points in the lower limb region.

[0078] In step S208, the evaluation unit 1033 of the server 10 generates a comprehensive evaluation from the judgment result of the total movement period Ta in step S203, the judgment result of each phase period Tn in step S204, the evaluation result of each phase in step S205, the evaluation result by each part in each phase in step S206, and the compensatory movement evaluation result in step S207 (evaluation result integration process). The comprehensive evaluation is an index showing the evaluation of the quality of the motor function, generated, for example, by scoring each result and performing tabulation and statistical processing. The comprehensive evaluation may be generated using the evaluation mathematical model 1022.

[0079] <Output example> Fig. 10 is a diagram showing an example of an evaluation report. Fig. 11 is a diagram showing an example of a list display of a presented exercise menu. Fig. 12 is a diagram showing an example of the contents of a presented exercise menu. These may be displayed on the display 2071 of the user terminal 20, or may be displayed on the display of another user terminal (not shown) used by the elderly person or a third party, or may be printed out on paper. By providing these evaluation reports and the like to the person to be evaluated, his / her family, and / or caregiver, the user can improve the degree of trust and facilitate communication.

[0080] The evaluation report in FIG. 10 illustrates an example of the evaluation results (indicated as measurement results in FIG. 10) for "single-leg standing balance (standing on one leg)" among the functional operation types. "Balance ability" displays the current and previous results of the overall evaluation of the evaluation information in evaluation table 1013, with "good," "fair," and "caution." "Smoothness" displays the phase period judgment results of the first phase and the current and previous contents of the phase evaluation results. Similarly, "stability" is displayed for the second phase, and "strength" for the third phase. "Number of seconds" displays the history of the judgment results for the entire operation period in a graph. "Summary of results" displays feedback data generated based on the evaluation information.

[0081] In the example of the exercise menu list in FIG. 11, a list of exercise menus is shown for each category of balance exercises and strength / flexibility improvement exercises as recommended exercises. The displayed exercise menu corresponds to the suggested exercise menu information in the report information of evaluation table 1013. If the person being evaluated is elderly, it is preferable that at least one exercise be presented in the balance exercise category. When the user or person being evaluated selects an exercise menu on user terminal 20, the contents of the exercise menu in FIG. 12 are displayed.

[0082] Two examples of two exercise menus are shown in Fig. 12. Items displayed in the exercise menu include basic information including "Points to be aware of," "Difficulty level," "Number of times (number of times and set)," "Caution," and "Free description," as well as exercise content that explains the details of the exercise. The contents of the exercise may be a combination of one or more images and explanatory text as shown in Fig. 12, or may be a video.

[0083] As described above, the information processing system 1 according to the present disclosure provides the user or the person being evaluated with an evaluation of motor functions such as balance function, an analysis of the evaluation, and an exercise menu selected based on the evaluation of motor functions and the analysis of the evaluation, by simply taking a video and inputting the minimum necessary information. That is, the information processing system 1 can easily and accurately evaluate motor functions, suggest effective rehabilitation, and improve the quality of rehabilitation. Furthermore, the information processing system 1 allows the user to easily measure the motor functions, such as the balance ability, of the person being evaluated from the user terminal 20.

[0084] <Program> 13 is a schematic block diagram showing the configuration of a computer 801. The computer 801 includes a CPU 802 (processor), a main storage device 803, an auxiliary storage device 804, an interface 805, and a graphics processing unit (GPU) 806.

[0085] Here, the program for realizing each function constituting the server 10 according to the above embodiment will be described in detail.

[0086] The server 10 is implemented in a computer 801. The operation of each component of the server 10 is stored in the form of a program in an auxiliary storage device 804. The CPU 802 reads the program from the auxiliary storage device 804, loads it in the main storage device 803, and executes the above-mentioned processing in accordance with the program. The CPU 802 also secures a storage area corresponding to the above-mentioned storage unit 101 in the main storage device 803 or the auxiliary storage device 804 in accordance with the program.

[0087] Specifically, the program: A program to be executed by a computer having a processor and a memory unit, the program causing the processor to execute the following steps: acquiring an evaluation video capturing the entire body of a person to be evaluated standing on one leg and information about the duration of the one-legged position; estimating key points, including joint points and center of gravity, of the person to be evaluated appearing in the evaluation video; and generating an evaluation of the motor function of the person to be evaluated using a mathematical model from the estimated joint points and center of gravity and the elapsed time of the one-legged position, wherein the one-legged position includes, as movement phases, a predictive postural control phase, a posture maintenance phase and / or a reactive postural control phase, and the step of generating the evaluation is a step of generating the evaluation result for each movement phase.

[0088] The auxiliary storage device 804 is an example of a non-transient tangible medium. Other examples of non-transient tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, and a semiconductor memory connected via an interface 805. When this program is distributed to the computer 801 via a network NW, the computer 801 that has received the program may load the program into the main storage device 803 and execute the above process.

[0089] The program may be for implementing part of the above-mentioned functions. Furthermore, the program may be for implementing the above-mentioned functions in combination with other programs already stored in the auxiliary storage device 804, i.e., a so-called differential file (differential program). may be also possible.

[0090] Although the embodiments of the present invention have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention.

[0091] Although the embodiment of the present invention has been described above, the aspects of the present invention are not limited to this embodiment.

[0092] For example, in the measurement procedure in the above embodiment, the video processing unit 1032 may recognize specific parts of the body of the person being evaluated that appear in the evaluation video and objects around the person being evaluated while the evaluation video is being captured, and the video acquisition unit 1031 may automatically terminate capturing the evaluation video when the positional relationship between the specific body part and the surrounding objects satisfies a predetermined condition.

[0093] In addition, in the procedure for measuring the motor function in the above embodiment, the user measures the total movement period Ta, but the video acquisition unit 1031 may recognize the movement of the subject in real time and calculate the total movement period Ta. This makes the measurement easier.

[0094] In step S208 of the above embodiment, the overall evaluation is generated from the judgment result of the entire movement period, the judgment result of each phase period, the overall evaluation result of each phase, the evaluation result by body part of each phase, and the compensatory movement evaluation result. However, one of these judgment results and evaluation results may be used as the overall evaluation, or a combination of two or more may be used. When the functional movement is one-legged standing, the overall evaluation may be simply the judgment result of the entire movement period. Specifically, the "balance ability" is judged into three stages (for example, good, fair, caution) as an index showing the overall balance ability based on the number of seconds of the entire movement period Ta. "Good" is the total movement period Ta is equal to or greater than the cutoff value, "fair" is equal to or greater than the standard (average) of the same age / sex, and "caution" is less than the standard (average) of the same age / sex.

[0095] In addition, in step S202 of the above embodiment, the boundary positions of the operation phases were estimated using all frames of the converted video, but it is not necessary to use all frames of the converted video as long as they cover the range from the start to the end of the functional operation; only some of the frames may be used.

[0096] In addition, the judgment of the total operation period Ta in step S203 of the above embodiment may be "good" if it is equal to or greater than the standard value Taa for the same age and sex, "passable" if it is equal to or greater than a predetermined cutoff value Tac, and "caution" if it is less than the predetermined cutoff value Tac. Similarly, the judgment of the n-th phase period Tn in step S204 of the above embodiment may be "good" if it is equal to or greater than a predetermined cutoff value Tac, "passable" if it is equal to or greater than the standard value aa for the same age and sex, and "caution" if it is less than the standard value aa for the same age and sex. Furthermore, although the standard values ​​for the same age and sex are used, standard values ​​for each group of groups with the same care level, environment and individual factors, health condition, height, and weight, or standard values ​​of the union set, intersection set, difference set, and target difference set of these may be used.

[0097] <Additional Notes> An example of the configuration of this embodiment is as follows. [1] a video acquisition unit that acquires an evaluation video obtained by capturing an entire body of the subject performing a predetermined functional movement while standing on one leg and duration information of the predetermined functional movement while standing on one leg; a video processing unit that estimates key points including joint points and a center of gravity of the person to be evaluated that appears in the evaluation video; An evaluation unit that generates an evaluation result of the balance ability of the subject based on the key points and the period information of the one-leg standing functional movement by using a mathematical model; Equipped with The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The evaluation unit generates the evaluation result for each of the operation phases. Information processing device. This allows the assessment of balance ability as a motor function to be performed for each movement phase of the motor function with different features while taking into account both the center of gravity and the joint points, improving the accuracy of the assessment of motor function. In addition, since the motor function can be measured with only the simple operation of taking a video, the measurement can be labor-saving. In other words, the motor function of the person being assessed can be assessed easily and more accurately. [2] Further comprising a storage unit for storing the evaluation result, the evaluation unit analyzes the current evaluation result and the past evaluation results to generate report information; [1] An information processing device according to the present invention. This makes it possible to confirm the effect of rehabilitation. [3] The storage unit stores a plurality of exercise menus, An exercise suggestion unit that selects one or more of the exercise menus based on the evaluation result and the period information. [2] An information processing device according to the present invention. This makes it possible to present an exercise menu appropriate for the current state of motor function. [4] the video acquisition unit acquires, with respect to the evaluation videos, a left evaluation video in a one-leg standing position with the left leg as a pivot foot and a right evaluation video in a one-leg standing position with the right leg as a pivot foot; The evaluation unit generates the evaluation result based on the left evaluation video and the right evaluation video. The information processing device according to any one of [1] to [3], This makes it possible to present evaluations of the left and right sides separately, as well as evaluations comparing the left and right sides. [5] the evaluation unit inputs at least one of a physical quantity of the key point during the predictive posture control phase, a statistical value of the physical quantity, and a calculation result of the physical quantity and the statistical value as input data to the mathematical model, and obtains the evaluation result in the predictive posture control phase as output data from the mathematical model. An information processing device according to any one of [1] to [4]. This allows the motor function to be evaluated taking into account the feature quantities that affect the predictive postural control phase, making it possible to provide evaluation results specialized for the predictive postural control phase. [6] the evaluation unit inputs the physical quantities of the key points during the predictive posture control phase and / or statistics of the physical quantities of the key points as input data to the mathematical model, and obtains the evaluation result during the posture maintenance phase as output data from the mathematical model. An information processing device according to any one of [1] to [5]. This allows the motor function to be evaluated taking into account features that affect the posture maintenance phase, making it possible to provide evaluation results that are specialized for the posture maintenance phase. [7] the evaluation unit inputs physical quantities of the key points during the reactive postural control phase and / or statistical values ​​of the physical quantities of the key points as input data to the mathematical model, and obtains the evaluation result during the reactive postural control phase as output data from the mathematical model. An information processing device according to any one of [1] to [6]. This allows the motor function to be evaluated taking into account the feature quantities that affect the reactive postural control phase, making it possible to provide evaluation results specialized for the reactive postural control phase. [8] The evaluation unit inputs physical quantities of the key points and / or statistical values ​​of the physical quantities of the key points related to the upper limb compensatory movement and / or the lower limb compensatory movement into the mathematical model as input data, and obtains the presence or absence of the upper limb compensatory movement and / or the lower limb compensatory movement as output data from the mathematical model. An information processing device according to [1] or [7]. This allows for a more specific evaluation result to be presented since motor function is evaluated taking into account compensatory movements of the upper and lower limbs. Furthermore, it allows for a more appropriate exercise menu to be proposed for the compensatory movements of the upper and lower limbs. [9] An information processing method executed by an information processing device including a processor and a storage unit, the information processing method comprising: A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation of the motor function of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; An information processing method, wherein the step of generating an evaluation is a step of generating the evaluation result for each of the operation phases.

[10] A program to be executed by a computer having a processor and a storage unit, The program causes the processor to: A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation of the motor function of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The step of generating the evaluation is a step of generating the evaluation result for each of the operation phases. program.

[11] An information processing system including an information processing device having a processor and a storage unit, and a user terminal, The processor, A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation of the motor function of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The step of generating the evaluation is a step of generating the evaluation result for each of the operation phases. Information processing system.

[12] the video processing unit recognizes specific body parts of the person to be evaluated and objects around the person to be evaluated that appear in the evaluation video, the video acquisition unit generates an instruction to end capturing the evaluation video when a positional relationship between the specific body part and the surrounding object satisfies a predetermined condition. An information processing device according to any one of [1] to [8]. As a result, when the measurement meets a predetermined end condition, imaging is automatically stopped, thereby reducing the burden on the user during measurement. [Explanation of symbols]

[0098] 1: Information processing system 10: Information processing server 101: Storage section 103: Control unit 1031: Video acquisition unit 1032: Video processing section 1033: Evaluation section 1034: Exercise proposal department 1035: Evaluation result output section 1036: Mathematical Model Management Department 20: User terminal 201: Storage section 203: Control section 205: Input device 207: Output device

Claims

1. a video acquisition unit that acquires an evaluation video obtained by capturing an image of the entire body of the evaluation subject performing one-leg standing and period information of the one-leg standing; a video processing unit that estimates key points including joint points and a center of gravity of the person to be evaluated that appears in the evaluation video; an evaluation unit that generates an evaluation result of the balance ability of the evaluation subject using a mathematical model based on the key points and the period information of the one-leg standing position; Equipped with The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The evaluation unit generates the evaluation result for each of the operation phases. Information processing device.

2. Further comprising a storage unit for storing the evaluation result, the evaluation unit analyzes the current evaluation result and the past evaluation results to generate report information; The information processing device according to claim 1 .

3. The storage unit stores a plurality of exercise menus, An exercise suggestion unit that selects one or more of the exercise menus based on the evaluation result and the period information. The information processing device according to claim 2 .

4. the video acquisition unit acquires, with respect to the evaluation videos, a left evaluation video in a one-leg standing position with the left leg as a pivot foot and a right evaluation video in a one-leg standing position with the right leg as a pivot foot; The evaluation unit generates the evaluation result based on the left evaluation video and the right evaluation video. The information processing device according to claim 1 .

5. the evaluation unit inputs at least one of a physical quantity of the key point during the predictive posture control phase, a statistical value of the physical quantity, and a calculation result of the physical quantity and the statistical value as input data to the mathematical model, and obtains the evaluation result in the predictive posture control phase as output data from the mathematical model. The information processing device according to claim 1 .

6. the evaluation unit inputs physical quantities of the key points during the posture maintenance phase and / or statistical values ​​of the physical quantities of the key points as input data to the mathematical model, and obtains the evaluation result during the posture maintenance phase as output data from the mathematical model. The information processing device according to claim 1 .

7. the evaluation unit inputs physical quantities of the key points during the reactive postural control phase and / or statistical values ​​of the physical quantities of the key points as input data to the mathematical model, and obtains the evaluation result during the reactive postural control phase as output data from the mathematical model. The information processing device according to claim 1 .

8. The evaluation unit inputs physical quantities of the key points and / or statistical values ​​of the physical quantities of the key points related to the upper limb compensatory movement and / or the lower limb compensatory movement into the mathematical model as input data, and obtains the presence or absence of the upper limb compensatory movement and / or the lower limb compensatory movement as output data from the mathematical model.

8. The information processing device according to claim 6 or 7.

9. An information processing method executed by an information processing device including a processor and a storage unit, the information processing method comprising: A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation result of the balance ability of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The step of generating the evaluation result is a step of generating the evaluation result for each of the operation phases. Information processing methods.

10. A program to be executed by a computer having a processor and a storage unit, The program causes the processor to: A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation result of the balance ability of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The step of generating the evaluation result is a step of generating the evaluation result for each of the operation phases. program.

11. An information processing system including an information processing device having a processor and a storage unit, and a user terminal, The processor, A step of acquiring an evaluation video of an evaluation subject performing one-leg standing and period information of the one-leg standing; estimating key points including joint points and a center of gravity of the person to be evaluated appearing in the evaluation video; generating an evaluation result of the balance ability of the subject using a mathematical model based on the key points and the elapsed time of the one-leg standing position; Run the command, The one-leg stance includes, as movement phases, a predictive postural control phase, and a posture maintenance phase and / or a reactive postural control phase; The step of generating the evaluation result is a step of generating the evaluation result for each of the operation phases. Information processing system.

Images