Information processing device, information processing method, program, and information processing system

Abstract

To propose effective rehabilitation, and improve the quality of the rehabilitation by evaluating a motor function of a person to be evaluated easily.SOLUTION: An information processing device includes: a moving image acquisition unit for acquiring a moving image for evaluation capturing a whole body of a person to be evaluated who performs a standing-sitting motion from a seated state; a moving image processing unit for estimating a key point of the body including a center of gravity of the person to be evaluated shown in the moving image for evaluation using the moving image for evaluation, performing inference to divide motion phases at least into a first phase, which is a standing-phase, and a second phase, which is a sitting phase, of the standing-sitting motion using a temporal change of the key point, and calculating a phase period from the start to the end of the motion phase; and an evaluation unit for generating an evaluation result on the motion of the person to be evaluated using the center of gravity and the key points including two or more key points of the superior limb region and including two or more key points of the inferior limb region.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Social security costs related to nursing care in Japan continue to rise, and the government is working to transition to a sustainable social security system by promoting support for independence and prevention of serious illness. The long-term care insurance system will accelerate its shift to an outcome-based remuneration system that emphasizes results, and nursing care facilities will need to improve the quality of rehabilitation in order to achieve results. However, few nursing care facilities employ rehabilitation professionals, and approximately 80% of day care facilities, in particular, lack such professionals, making it difficult to provide "effective rehabilitation." Proposing effective rehabilitation requires assessment to properly understand issues and conditions, but motion analysis requires the highly specialized skills of rehabilitation professionals.

[0003] To address these issues in assessing rehabilitation, Patent Document 1 discloses a technology that calculates the speed of reaction movements up to the start of center-of-gravity movement to objectively evaluate the subject's ability to control center-of-gravity and diagnoses the risk of falling in advance. Also, Patent Document 2 discloses a technology that captures video of the subject and estimates the subject's movements and skeletal information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-185557 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-080199 [Patent Document 3] Patent No. 7454326 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 does not take into account feature quantities other than the center of gravity. Furthermore, the technology disclosed in Patent Document 2 still requires a rehabilitation professional to evaluate the subject's issues and condition from the estimated subject's movements and skeletal information. The technology disclosed in Patent Document 3 does not include the center of gravity as a feature point, nor does it evaluate the standing-up phase and the sitting-down phase separately.

[0006] However, with conventional evaluation methods, it is difficult to diagnose subjects with reduced motor function, such as those unable to maintain an upright posture, and it is not possible to properly grasp the challenges and conditions of such subjects. As a result, it is not possible to propose effective rehabilitation. In addition, some evaluation methods use tools (e.g., chairs, bedding, and other furniture), but to perform an accurate evaluation using such methods, it is necessary to consider the relationship between the tool and the subject.

[0007] Therefore, the present disclosure aims to provide an information processing device, information processing method, program, and information processing system that can be used to propose effective rehabilitation and improve the quality of rehabilitation by easily evaluating the motor function of the subject. Note that problems that are obvious to those skilled in the art and can be read from the embodiments and explanations characteristic of the present disclosure described in the specification, drawings, etc. of the present disclosure may also be problems to be solved by the divided invention if a divisional application based on the present disclosure is filed. [Means for solving the problem]

[0008] The information processing device according to the present disclosure includes a video acquisition unit that acquires an evaluation video capturing the entire body of the person being evaluated performing standing and sitting movements from a seated position; a video processing unit that uses the evaluation video to estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity, and uses the changes in the key points over time to make inferences that divide the standing and sitting movements into at least a first phase, which is the standing phase, and a second phase, which is the sitting phase, as movement phases, and calculates the phase period from the start to the end of the movement phase; and an evaluation unit that generates evaluation results regarding the movements of the person being evaluated using the center of gravity and key points that include two or more key points in the upper limb region and two or more key points in the lower limb region.

[0009] In addition, the information processing method according to the present disclosure is an information processing method executed by an information processing device having a processor and a memory unit, and includes the steps of: the processor acquiring an evaluation video capturing an image of the entire body of the person being evaluated performing standing-up and sitting-down movements from a seated position; the video processing unit using the evaluation video to estimate key points of the body of the person being evaluated appearing in the evaluation video, including the center of gravity; the video processing unit using the changes in the key points over time to infer that the standing-up and sitting-down movements are divided into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as movement phases; the video processing unit calculating the phase period from the start to the end of the movement phase; and the evaluation unit generating an evaluation result regarding the movement of the person being evaluated using the center of gravity and key points including two or more key points in the upper limb region and two or more key points in the lower limb region.

[0010] In addition, in order to achieve the above-mentioned object, the program of the present disclosure is a program to be executed by a computer having a processor and a memory unit, and causes the computer to execute the following steps: the processor acquires an evaluation video that captures the entire body of the person being evaluated performing standing-up and sitting-down movements from a seated position; the video processing unit uses the evaluation video to estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; the video processing unit uses changes in the key points over time to infer that the standing-up and sitting-down movements are divided into at least a first phase, which is the standing phase, and a second phase, which is the sitting phase, as movement phases; the video processing unit calculates the phase period from the start to the end of the movement phase; and the evaluation unit generates an evaluation result regarding the movement of the person being evaluated using key points that include the center of gravity and two or more key points in the upper limb region and two or more key points in the lower limb region.

[0011] In addition, in order to achieve the above-mentioned object, the information processing system of the present disclosure is an information processing system including an information processing device having a processor and a memory unit, and a user terminal that captures the entire body of the person being evaluated performing standing and sitting movements from a seated position, wherein the information processing terminal device includes a video acquisition unit that acquires an evaluation video that captures the entire body of the person being evaluated performing standing and sitting movements from a seated position, a video processing unit that uses the evaluation video to estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity, and uses the changes in the key points over time to make inferences that divide the standing and sitting movements into at least a first phase, which is the standing phase, and a second phase, which is the sitting phase, as movement phases, and calculates the phase period from the start to the end of the movement phase, and an evaluation unit that generates an evaluation result regarding the movement of the person being evaluated using the center of gravity and key points that include two or more key points in the upper limb region and two or more key points in the lower limb region. [Effects of the Invention]

[0012] According to the present disclosure, the motor function of the person being evaluated can be easily evaluated. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an overall configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of an information processing server. [Figure 3] FIG. 2 is a diagram illustrating an example of information stored in a storage unit of the information processing server. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of a user terminal. [Figure 5] FIG. 10 is a diagram showing an example of measuring motor function using an information processing system. [Figure 6] FIG. 1 is a diagram showing the procedure for measuring motor function and key points estimated during measurement. [Figure 7] 10 is a flowchart showing the operation of a motor function evaluation process executed in the information processing system. [Figure 8] 10 is a flowchart showing the operation of an inference process. [Figure 9] FIG. 10 is a diagram showing an example of correspondence between an evaluation subject and estimated key points. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a phase period. [Figure 11] FIG. 10 is a diagram illustrating an example of an evaluation report. [Figure 12] FIG. 1 is a block diagram showing the basic hardware configuration of a computer. [Figure 13] FIG. 10 is a flowchart showing the flow of processing related to a low load evaluation function. [Figure 14] FIG. 10 is a flowchart showing the flow of processing related to a low load evaluation function. [Figure 15] FIG. 10 is a flowchart showing the flow of processing related to a partial posture evaluation function. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] <System configuration> The information processing system 1 in the present disclosure is an information processing system that provides a motor function evaluation service by filming a subject performing predetermined functional movements related to motor function, evaluating the subject's motor function using the filmed video, and using the evaluation results to propose an exercise menu to improve the subject's motor function.

[0016] The subjects of evaluation are mainly elderly people. Users of the motor function evaluation service are mainly businesses that provide nursing care and assistance services to the subjects of evaluation, local governments, individuals, etc. In the following explanation, an example of a user will be a nursing care provider that provides nursing care services.

[0017] 1 is a diagram showing an overall configuration of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 includes information processing devices, an information processing server 10, and a user terminal 20, which are connected via a network NW.

[0018] Each information processing device is configured by a computer having an arithmetic unit and a storage unit. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later.

[0019] The network NW is, for example, a network such as the Internet, a VPN (Virtual Private Network), an intranet, or a short-range wireless communication. For simplicity of explanation, one user terminal 20 is shown in FIG. 1, but the information processing server 10 can be connected to two or more user terminals 20 via the network NW. Furthermore, one user terminal 20 may be used by multiple users or evaluation subjects.

[0020] An operator of a motor function evaluation service provides a motor function evaluation service to a user using an information processing system 1. Specifically, the information processing system 1 provides an application that runs on a user terminal 20 used by the user. The user uses the application to measure the motor function of a person to be evaluated. In the measurement, the user captures an image of the person to be evaluated performing a predetermined functional movement related to motor function and transmits the captured evaluation video and information about the person to the information processing server 10. The information processing server 10 evaluates the motor function of the person to be evaluated based on the received evaluation video, etc. The information processing server 10 also selects an appropriate exercise menu based on the evaluation results. The evaluation results and the selected exercise menu are transmitted to the user terminal 20. Motor function includes balance ability, walking ability, functional mobility, flexibility, whole-body coordination, whole-body endurance, muscle strength, etc. Balance ability includes static balance ability and dynamic balance ability. In this embodiment, muscle strength and balance ability are evaluated as motor function.

[0021] FIG. 2 is a block diagram showing the functional configuration of the information processing server 10. As shown in FIG. The information processing server 10 (hereinafter simply referred to as the server 10) is an information processing device that provides a motor function evaluation service. The server 10 includes a storage unit 101 and a control unit 103.

[0022] <Configuration of the storage unit of the information processing server> The storage unit 101 of the server 10 is a storage medium or storage device that stores information used to execute management by the information processing system 1. Although the storage unit 101 is illustrated as being integrated with the server 10 in FIG. 2, it may be a storage device that is physically independent from the server 10. The storage unit 101 includes an application program 1011, a video table 1012, an evaluation table 1013, an exercise information table 1014, a user table 1015, an evaluation subject table 1016, a video analysis mathematical model 1021, an evaluation mathematical model 1022, and a proposed mathematical model 1023. The storage unit 101 also stores other information that does not belong to these storage units.

[0023] 3 is a diagram showing an example of information stored in the storage unit 101 of the information processing server 10. Each table in the storage unit 101 will be described below with reference to FIG.

[0024] The video table 1012 is a table that stores and manages the evaluation videos acquired by the video acquisition unit 1031. The video table 1012 stores information items such as a video ID, video data, basic video information, creating user, imaging subject information, converted video, and evaluated video.

[0025] The video ID is an item that stores video identification information for identifying an evaluation video. The video identification information is an item in which a unique value is set for each evaluation video. A video ID is generated for each acquired evaluation video. The video data is an item for storing the file body of the evaluation video. The file body may be stored on another server, and the path to the file may be stored. The basic video information is an item for storing basic video information such as the shooting date and time of the evaluation video, file format, compression method, capacity, resolution, frame rate, bit rate, shooting time, etc. The creating user is an item that stores the user ID of the user who shoots the evaluation video. This allows the evaluation video to be linked to the user. The imaging subject information is an item that stores imaging subject information, which is information relating the evaluation subject, who is the subject of the evaluation video, to the evaluation video. The imaging subject information includes a functional operation type, a total operation period Ta, and an evaluation subject ID, which will be described later. By including the evaluation subject ID, the evaluation video can be linked to the evaluation subject. Functional operations included in the functional operation type include, for example, the Timed Up & Go Test, the Berg Balance Scale, the Stand-to-Sit Test, and the Single or Double Leg Stand Test. Each functional operation is associated with a specific motor function. Specifically, the Timed Up & Go Test is associated with muscle strength, balance ability, walking ability, and functional mobility; the Berg Balance Scale is associated with balance ability, muscle strength, endurance, flexibility, and interval; the Stand-to-Sit Test is associated with lower limb muscle strength and balance ability; and the Single or Double Leg Stand Test is associated with lower limb muscle strength and mobility ability. In other words, the Timed Up & Go Test, Berg Balance Scale, Sit-to-Stand Test, and Single or Double Leg Stand Test are all related to balance ability and muscle strength and can be used alone or in combination as tests to determine the risk of falls in elderly people. The total movement period Ta is duration information that numerically indicates the period (time) from the start to the end of such functional movement. The converted moving image is an item for storing a converted moving image generated by executing preprocessing on the evaluation moving image acquired by the moving image acquisition unit 1031. The evaluated moving image is an item for storing and managing evaluated moving images generated by the inference processing of the evaluation unit 1034. The evaluated moving images include a report moving image, which will be described later.

[0026] The evaluation table 1013 is a table that stores and manages evaluation information related to the analysis and evaluation of the motor function of the person being evaluated. The evaluation table 1013 stores the following information items: evaluation information ID, analysis target video, joint point information, environment information, phase period information, evaluation information, and evaluation report information.

[0027] The evaluation information ID is an item that stores evaluation identification information for identifying evaluation information. The evaluation identification information is an item that has a unique value set for each piece of evaluation information. Evaluation information is generated for each measurement of motor function. The analysis target video is an item in which the video ID of the evaluation video to be analyzed is stored, thereby linking the evaluation information with the evaluation video. The joint point information is a table that stores and manages pairs of key points and coordinates generated by the inference process of the evaluation unit 1034 (described later) for each frame of the converted video, linking them to the frame. Key points include joint points and centers of gravity. The correspondence between the human body and the inferred key points will be described later. The joint point information is not limited to key points related to joint points, but may also include feature points of parts that can be used to reconstruct or express the posture of the human body, such as the eyes and ears, as shown in FIG. 9 (described later). The environmental information is a table that stores and manages information about the furniture 30 used by the subject for each measurement, which is acquired by the environmental information acquisition unit 1033 described below. The information about the furniture 30 includes the type of furniture, dimensional information about the furniture, whether it has a backboard, and whether it has armrests. The type of furniture is, for example, a chair or a bed, and the dimensional information about the furniture is the height and depth of the seat. The phase period information is an item for storing period information of the converted video to be analyzed. The period information of the converted video includes a delimiter position and an n-th set phase period Tn, which will be described later. In FIG. 3, the first set phase period T1 to the fifth set phase period T5 are shown as the n-th set phase period Tn when the function operation type is the "standing-sitting test." The evaluation information is a field that stores evaluation information that is the result of analyzing the period information and the converted video. The evaluation information includes the overall movement period determination result, the phase period determination result, the phase evaluation result, the phase part evaluation result, the compensatory movement evaluation result, and the overall evaluation, which will be described later. The evaluation report information is an item for storing evaluation report information as information related to the evaluation report described later. The evaluation report information includes a report video described later, feedback data described later, and suggested exercise menu information described later.

[0028] The exercise information table 1014 stores information about an exercise menu to be proposed based on the evaluation results of the subject's motor function. The exercise information table 1014 stores the following items of information: exercise information ID, basic exercise information, and exercise content. The exercise menu is divided into two categories: balance exercises and movement training, and strength and flexibility improvement exercises. Strength and flexibility improvement exercises include exercises for the trunk, hip joints, thighs, ankles and feet. Examples of exercise menus are described below.

[0029] The exercise information ID is an item for storing exercise identification information for identifying an exercise menu. The exercise identification information is an item for which a unique value is set for each exercise menu. The basic exercise information is an item for storing basic information about the exercise menu, such as the name of the exercise, target body part, difficulty level, number of times or sets to be performed, points to note, and other notes. The exercise content is an item that stores content that explains the details of the exercise, which is composed of images, explanatory text, video, explanatory audio, or a combination thereof related to the exercise menu.

[0030] The user table 1015 is a table that stores and manages information about users (user members) who use the motor function evaluation service. The user table 1015 stores information such as a user ID and basic user information. When a user registers to use the service, the user's information is stored in a new record in the user table 1015. This allows the user to use the motor function evaluation service according to the present disclosure.

[0031] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item that is set with a unique value for each user. The user basic information is an item for storing basic information such as the user's name, address, contact information, password, etc.

[0032] The evaluation subject table 1016 is a table that stores and manages information about evaluation subjects who are to undergo motor function evaluation using the motor function evaluation service. The evaluation subject table 1016 stores information items such as evaluation subject ID, evaluation subject basic information, management user, and evaluation history. The information about evaluation subjects is mainly registered by users of the motor function evaluation service. In this embodiment, a care provider registers information about evaluation subjects, or links information about evaluation subjects registered in other systems or services used by the care provider.

[0033] The evaluation target ID is an item that stores evaluation target identification information for identifying the evaluation target. The evaluation target identification information is an item in which a unique value is set for each evaluation target. The basic information of the person being evaluated includes the person's name, date of birth, age, sex, height, weight, level of care required, health condition including medical history, physical and mental functions, ADL / IADL, environmental and personal factors (use of welfare equipment, home environment), etc. The administrative user is an item that stores the user ID of the care provider to which the person to be evaluated belongs. This allows the information of the person to be evaluated to be linked to the information of the user. The evaluation history is an item in which the evaluation information ID is stored as the evaluation history by the information processing system 1. This allows the motor function evaluation results to be linked to the person being evaluated.

[0034] Returning to FIG. 2, each mathematical model in the storage unit 101 will be described.

[0035] The video analysis mathematical model 1021 is a mathematical model that uses a video as input data to estimate the joint points, center of gravity, skeleton, posture, etc. of a human body or animal that appears in the video. Details of video processing using the video analysis mathematical model 1021 will be described later.

[0036] The video analysis mathematical model 1021 of this embodiment is a deep learning model using a deep neural network in deep learning. The video analysis mathematical model 1021 can be realized using a general-purpose skeleton estimation model that has undergone deep learning. Examples of general-purpose skeleton estimation models that can be used include Openpose, PoseNet, AlphaPose, HRNet, MMPose, and BlazePose. The video analysis mathematical model 1021 does not necessarily have to be a deep learning model, and may be any machine learning or artificial intelligence model. The video analysis mathematical model 1021 may be generated for each functional operation, gender, age, level of care required, and health condition.

[0037] The evaluation mathematical model 1022 is a mathematical model that uses key points estimated from the converted video as input data and outputs the quality of the functional movements performed by the person being evaluated who appears in the converted video. Details of the inference process using the evaluation mathematical model 1022 will be described later. The evaluation mathematical model 1022 is, for example, one type of machine learning, artificial intelligence, deep learning model, rule-based model, or a combination thereof. The evaluation mathematical model 1022 does not need to be a single mathematical model, and may be realized by combining or switching between multiple independent types of mathematical models. The evaluation mathematical model 1022 may be generated for each motor function or functional movement to be evaluated, gender, age, level of care required, and health condition.

[0038] The proposed mathematical model 1023 is a mathematical model that receives period information and evaluation results as input data, selects one or more exercise menus, and outputs them. The proposed mathematical model 1023 may be, for example, a type of machine learning, artificial intelligence, deep learning model, rule-based model, or a combination thereof. The proposed mathematical model 1023 does not need to be a single mathematical model, but may be realized by combining or switching between multiple independent types of mathematical models. The proposed mathematical model 1023 may be generated for each motor function or functional movement to be evaluated, gender, age, level of care required, and health condition.

[0039] <Configuration of the control unit of the information processing server> The control unit 103 of the server 10 includes functional units such as a video acquisition unit 1031, a video processing unit 1032, an environmental information acquisition unit 1033, an evaluation unit 1034, an exercise suggestion unit 1035, an evaluation result output unit 1036, and a mathematical model management unit 1037. The control unit 103 is a computer equipped with a CPU. The control unit 103 realizes each functional unit by executing an application program 1011 stored in the storage unit 101. The control unit 103 links each functional unit to execute a motor function evaluation process, which will be described later. Although not shown, the control unit 103 also has a function of controlling communication with external devices connected to the server 10 via a network NW or the like.

[0040] The video acquisition unit 1031 acquires, from a user terminal 20 connected to the server 10 via the network NW, an evaluation video of the person being evaluated performing a specified functional operation, and period information of the specified functional operation.

[0041] The video processing unit 1032 preprocesses the evaluation video and stores the preprocessed video as a converted video in the storage unit 101. The video processing unit 1032 also estimates key points, including joint points and centers of gravity, for the person to be evaluated who appears in the converted video.

[0042] The environment information acquisition unit 1033 acquires environment information including information about furniture 30 used by the person to be evaluated to perform a predetermined functional operation from the user terminal 20 connected to the server 10 via the network NW. For example, the environment information is obtained by inputting information about the furniture (e.g., type of furniture, dimensional information of the furniture, etc.) into the user terminal 20 by the user. Note that the acquisition of the environment information may be performed by using a mathematical model from the converted video to perform indirect measurement based on the age, sex, height, etc. of the person to be evaluated, and acquiring the measurement results as the environment information.

[0043] The evaluation unit 1034 executes an inference process to infer the evaluation results of the predetermined functional operations and the motor functions related to the predetermined functional operations of the person to be evaluated from the key points, the period information of the predetermined functional operations, and the environmental information using a mathematical model, and stores the inference results as evaluation information in the storage unit 101. Details of the inference process will be described later.

[0044] The exercise suggestion unit 1035 selects an exercise menu suitable for improving, maintaining, and enhancing the motor function of the person to be evaluated, based on the period information of the predetermined functional operation and the evaluation information.

[0045] The evaluation result output unit 1036 generates evaluation report information based on the evaluation information and the selected exercise menu, and outputs it to the user terminal 20.

[0046] The mathematical model management unit 1037 generates and updates each mathematical model in the storage unit 101. Note that the generation of a mathematical model also includes the meaning of learning.

[0047] 4 is a block diagram showing the functional configuration of the user terminal 20. The user terminal 20 includes a storage unit 201, a control unit 203, an input device 205, and an output device 207 as functional units. The user terminal 20 is an information processing device with an imaging function that is operated by a user who uses the service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or tablet, a stationary personal computer (PC) or a laptop PC, or a wearable terminal such as a head mounted display (HMD) or a wristwatch terminal.

[0048] <Configuration of the storage unit of the user terminal> The storage unit 201 of the user terminal 20 is a storage medium or storage device that can store information used for providing the motor function evaluation service by the information processing system 1. Although it is illustrated as being integrated with the user terminal 20 in FIG. 4, it may be a storage device that is physically independent from the user terminal 20. The storage unit 201 includes an application program 2011 and a user ID 2012. The storage unit 201 also stores other information that does not belong to these storage units.

[0049] The application program 2011 may be stored in advance in the storage unit 201, or may be downloaded from a web server operated by a service provider via a communication interface. The application program 2011 includes an application such as a web browser application. The application program 2011 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the user terminal 20.

[0050] The user ID 2012 stores the user ID of the user using the user terminal 20 as identification information for the user to use the motor function evaluation service. The user transmits the user ID from the user terminal 20 to the server 10. The server 10 identifies the user by referencing the received user ID in the user table 1015, and provides the user with the service according to the present disclosure. The user ID 2012 may include information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20.

[0051] <Configuration of the control unit of the user terminal> The control unit 203 of the user terminal 20 includes an input control unit 2031 and an output control unit 2032. The control unit 203 is a computer equipped with a CPU. The control unit 203 realizes each functional unit by executing an application program 2011 stored in the storage unit 201. Although not shown, the control unit 203 also includes a function of controlling communication with external devices connected to the user terminal 20 via a network NW or the like.

[0052] The input control unit 2031 controls the input device 205 . The output control unit 2032 controls the output device 207 .

[0053] The input device 205 of the user terminal 20 includes a camera 2051, a microphone 2052, a position information sensor 2053, a motion sensor 2054, and a touch device 2055. The input device 205 is a variety of input devices that can be operated by the user to input and select information. The input and selection of information include, for example, input by entering numbers, letters, or symbols, selection of items, and input to determine processing.

[0054] The camera 2051 is, for example, a visible light camera, and is an input device that has the function of detecting light reflected from a subject and generating image (still image or video) information. The camera 2051 may be built into the user terminal 20, or may be an independent camera connected to the user terminal 20.

[0055] The microphone 2052 is an input device that has the function of collecting sounds generated near the installation location, converting them into electrical signals, and generating sound information. For example, it may be used to accept voice input of instructions from the user.

[0056] The position information sensor 2053 is an input device having a function of generating position information of the user terminal 20.

[0057] The motion sensor 2054 has a function of detecting and tracking the movement of the user terminal 20 to generate posture information of the user terminal 20 .

[0058] The touch device 2055 is an input device having a function of generating operation information of an operation performed on the screen by the user of the user terminal 20 using a finger, a stylus pen, or the like.

[0059] The output device 207 of the user terminal 20 includes a display 2071 and a speaker 2072. The output device 207 is an output device that can display, play, and notify information and the like to the user.

[0060] The display 2071 is a display output device that displays characters, figures, images, videos, etc. The display 2071 may be a display output device independent of the user terminal 20, or may be a display output device such as a liquid crystal display or organic EL display in a smartphone or tablet.

[0061] The speaker 2072 is an audio output device that converts electrical signals into sound and outputs audio to the surrounding area. The speaker 2072 may be built into the user terminal 20, or may be an independent audio output device connected to the user terminal 20.

[0062] <Measurement procedure> Fig. 5 is a diagram showing an example of measuring motor function using an information processing system according to an embodiment of the present disclosure. Fig. 6 is a diagram showing the procedure for measuring motor function and key points estimated during measurement. The procedure for measuring motor function, i.e., a specific method for capturing a video for evaluation, will be described below with reference to Figs. 5 and 6.

[0063] As shown in FIG. 5, the user fixes the shooting position and shooting posture of the user terminal 20 to a predetermined position and posture using a tripod or the like. The user selects (inputs) and transmits, on the display 2071 of the user terminal 20, the registered identification information (such as the name) of the person to be evaluated and the type of functional operation to be evaluated as information about the person to be evaluated. The user then guides the person to a position (hereinafter referred to as the operation position) that is a predetermined distance away from the user terminal 20. At this time, the user may use a guide displayed when capturing the evaluation video. If the type of functional operation is a "standing-sitting test," the user guides the person to the furniture 30 used for the evaluation and places the person to be evaluated in a seated position on the furniture 30. At this time, on the display 2071 of the user terminal 20, a position that can fit the entire body of the person to be evaluated and at least a part of the furniture 30 used for the evaluation is displayed as the operation position, i.e., the position where the person to be evaluated is to perform the operation. If the functional operation type is a "stand-up and sit-down test," the subject is photographed from the right or left side (see Figure 5).

[0064] When the person to be evaluated is in a standby state at the operation position, the user starts capturing images on the user terminal 20 and instructs the person to start the functional operation of the type selected on the display 2071. The person to be evaluated starts the functional operation at their own timing. The user measures the period from the start to the end of the functional operation of the person to be evaluated using a stopwatch or the like. If the type of functional operation is a "standing-up and sitting-down test," the action of standing up from a seated position and the action of sitting down from a standing-up position are counted as one action, and an evaluation video is captured of this standing-up and sitting-down action repeated a predetermined number of times, and the total operation period Ta is measured.

[0065] When a predetermined time has elapsed since the start of video capture, the video capture automatically stops. If the functional operation status of the person being evaluated meets a predetermined termination condition before the predetermined time has elapsed, the user stops capturing video on the user terminal 20 and transmits the evaluation video to the server 10. The user inputs and transmits the timing result as the total operation period Ta on the display screen of the user terminal 20. At this timing, the functional operation type and the identification information of the person being evaluated may also be selected and transmitted together.

[0066] In this embodiment, when the functional operation type is a "stand-up and sit-down test," the test involves repeating the stand-up and sit-down movements five times. The predetermined end condition in this embodiment is when the total measurement time exceeds a predetermined time (e.g., 60 seconds). The predetermined end condition in this embodiment is not limited to when the predetermined time is exceeded, and may be changed depending on the person being evaluated, etc. The predetermined end condition differs depending on the functional operation type. Furthermore, even when the functional operation type is a "stand-up and sit-down test," the predetermined end condition also differs depending on the evaluation conditions, such as the number of times the stand-up and sit-down movements are repeated and the number of times the stand-up and sit-down movements are performed within a predetermined time (e.g., how many times the stand-up and sit-down movements were performed within 30 seconds (or 60 seconds)).

[0067] <Motor function evaluation processing> FIG. 7 is a flowchart showing the operation of the motor function evaluation process. The motor function evaluation process is a series of processes that mainly involves acquiring an evaluation video of the subject performing predetermined functional movements related to motor function, evaluating the subject's motor function based on the acquired evaluation video, selecting an appropriate exercise menu based on the evaluation results, and providing the evaluation results and the exercise menu. In the following explanation, it is assumed that a "sit-stand test" is selected as the predetermined functional movement to evaluate balance ability and muscle strength as motor functions. The motor function evaluation process includes a video acquisition process, a subject information acquisition process, a video preprocessing process, an environmental information acquisition process, an inference processing process, a report video generation process, a feedback data generation process, a motion judgment process, and an evaluation result output process. The operation of the motor function evaluation process will be described below with reference to the flowchart.

[0068] In step S101, the video acquisition unit 1031 of the server 10 acquires an evaluation video from the user terminal 20 (video acquisition step). Specifically, the video acquisition unit 1031 acquires, from the user terminal 20, an evaluation video of the person to be evaluated performing a predetermined functional operation, and stores the video in the video table 1012 of the storage unit 101.

[0069] In step S102, the video acquisition unit 1031 of the server 10 acquires imaging subject information from the user terminal 20 (imaging subject information acquisition step). Specifically, the video acquisition unit 1031 acquires, as imaging subject information, from the user terminal 20, identification information of the evaluation subject for identifying the evaluation subject (evaluation subject ID or information linked to the evaluation subject ID), the functional operation type input by the user, and the total operation period Ta of the functional operation, and stores them in the video table 1012 of the storage unit 101.

[0070] In step S103, the video processing unit 1032 of the server 10 performs preprocessing of the evaluation video acquired in step S101 (video preprocessing step). Specifically, as preprocessing, the video processing unit 1032 performs video conversion to set a predetermined frame rate and a predetermined format, and stores the converted video in the video table 1012 of the storage unit 101.

[0071] In step S104, the environmental information acquisition unit 1033 of the server 10 acquires environmental information from the user terminal 20 (environmental information acquisition step). Specifically, the environmental information acquisition unit 1033 acquires, as environmental information, information on the furniture used for evaluation (such as the type of furniture and seat dimension information) from the user terminal 20, and stores the information in the environmental information field of the evaluation table 1013 in the storage unit 101.

[0072] In step S105, the evaluation unit 1034 of the server 10 performs inference processing based on the converted video generated in step S103 and the environmental information acquired in step S104 to generate an evaluation result (inference processing step). Details of the inference processing will be described later.

[0073] In step S106, the video processing unit 1032 of the server 10 generates a report video based on the converted video generated in step S103 and the evaluation result of step S105, and stores the report video in the evaluation report information section of the evaluation table 1013 in the storage unit 101 (report video creation process). The report video may be either a video or a still image. Two types of report videos are generated: one for the user and one for the person being evaluated. The report video for the user is primarily information to be shared with rehabilitation professionals, and is mainly an extracted partial video or frame image related to the evaluation result, with key points superimposed. In this way, by adding key points to the report video for the user, it becomes possible to request a detailed movement analysis from a rehabilitation professional (physical therapist, occupational therapist). The report video for the person being evaluated is primarily information to be shared with the person being evaluated and their family and caregivers, and is displayed together with key points in the evaluation report, which will be described later. This will help those being evaluated who have been provided with the evaluation report to understand the posture during measurement (when video recording) and the evaluation results, thereby improving the quality of rehabilitation proposals and the motivation of those being evaluated.

[0074] In step S107, the evaluation unit 1034 of the server 10 analyzes the motor function of the person to be evaluated based on the evaluation result (evaluation information) of step S105 and the past evaluation results (evaluation information) stored in the storage unit 101, generates feedback data, and stores the data in the evaluation report information item of the evaluation table 1013 in the storage unit 101 (feedback data creation step). For example, a rule base or a general-purpose text generation AI model can be used to generate the feedback data. The feedback data includes information corresponding to each item displayed in the evaluation report. An example of the evaluation report when the functional operation type is "stand-up and sit-down test" will be described later.

[0075] In step S108, the exercise suggestion unit 1035 of the server 10 executes exercise determination based on the evaluation result in step S105 (exercise determination step). Specifically, the exercise suggestion unit 1035 applies the evaluation result of step S105 as input data to the proposed mathematical model 1023, acquires (selects) one or more exercise menus as output data as suggested exercise menu information, and stores the output data in the evaluation report information field of the evaluation table 1013 in the storage unit 101. It is preferable that the output data is selected from at least one category of balance exercise and movement practice. Incidentally, past evaluation results may be added as input data.

[0076] In step S109, the evaluation result output unit 1036 of the server 10 outputs an evaluation report based on the evaluation report information stored in the evaluation table 1013 of the storage unit 101 (evaluation result output step).

[0077] <Inference processing> The inference process is primarily a process for generating an evaluation result by analyzing the video and evaluating the motor function of the subject. The inference process is performed for each measurement. FIG. 8 is a flowchart showing the operation of the inference process. The inference process is a series of processes consisting of a joint point estimation process, a movement phase inference process, a total movement period determination process, a phase period determination process, a phase evaluation process, a phase part evaluation process, a compensation evaluation process, and an evaluation result integration process. In the following explanation, it is assumed that the "sit-to-stand test" is selected as the predetermined functional operation. The operation of the inference process will be explained below with reference to the flowchart.

[0078] In step S201, the video processing unit 1032 of the server 10 executes a joint point estimation process for each frame of the converted video (joint point estimation step). If two or more videos are taken in one measurement, the joint point estimation step is executed for each video. Specifically, the video processing unit 1032 applies each frame image of the converted video as input data to the video analysis mathematical model 1021, obtains sets of estimated key points and their coordinates for each frame (hereinafter simply referred to as key point coordinates, joint point coordinates, and center of gravity coordinates) as output data, and stores them in the joint point information item of the evaluation table 1013 in the storage unit 101 of the server 10 in association with the identification information of each frame. Note that the center of gravity may be calculated separately from the coordinates of the joint points.

[0079] 9 is a diagram showing an example of correspondence between a human body (evaluation subject) shown in a video and estimated key points (joint points and center of gravity). With reference to FIG. 9, the key points estimated by the inference process will be described. In this embodiment, the coordinates of each key point shown in Fig. 9 can be estimated for each frame by the video analysis mathematical model 1021. Note that the numbers displayed near each key point in Fig. 9 are used only to indicate that they correspond to the numbers in the key point list in the same figure, and are unrelated to the reference numbers in other figures. In Figure 9, 33 points are marked with black circles as estimated keypoints. The human body is roughly divided into the head and neck (area enclosed by dashed lines in Figure 9), the trunk (not shown), and the limbs (not shown). The limbs are divided into the upper limbs and lower limbs (not shown). Keypoints belonging to the head and neck region are the nose, the inner corner of the left eye, the left eye, the outer corner of the left eye, the inner corner of the right eye, the right eye, the outer corner of the right eye, the left ear, the right ear, the left edge of the mouth, and the right edge of the mouth. Keypoints belonging to the trunk region are the left shoulder, the right shoulder, the left hip, and the right hip. Keypoints belonging to the upper limb region are the left elbow, the right elbow, the left wrist, the right wrist, the left little finger, the right little finger, the left index finger, the right index finger, the left thumb, and the right thumb. Keypoints belonging to the lower limb region are the left knee, the right knee, the left ankle, the right ankle, the left heel, the right heel, the left index finger, and the right index finger.

[0080] In step S202, the video processing unit 1032 of the server 10 divides the converted video into motion phases based on the time change of the key point coordinates acquired in step S201 (motion phase inference process). If two or more videos are taken in one measurement, the motion phase inference process is executed for each converted video. A functional operation includes multiple operational phases from the start to the end of the functional operation, and transitions occur between these phases over time. The number of operational phases included, as well as the connection relationships and transition directions between the operational phases, vary depending on the type of functional operation. When the function operation type is a "standing-sitting test," as shown in FIG. 6, there are two operation phases, a first phase and a second phase, and the operation (set operation) of transitioning from the first phase to the second phase is counted as one set operation, and this set operation is repeated multiple times. In this embodiment, five repetitions of the set operation constitute one video (converted video). Note that the number of times the set operation is repeated in one video (converted video) is not limited to five. Also, even if the set operation cannot be completed five times within the specified time, a video that meets the specified termination conditions will be counted as one video (converted video). Note that, as mentioned above, the number of set operations and termination conditions for the "standing-sitting test" are not limited to these.

[0081] Specifically, the video processing unit 1032 applies the time transition of the coordinates of one or more predetermined key points in all frames of the converted video as input data to the video analysis mathematical model 1021, and obtains the delimiter positions (time or frame number) for each estimated movement phase as output data. In this process, the number of the one or more predetermined key points is preferably between three and eight. When the functional movement type is a "stand-sit test," it is preferable that the model includes two or more key points in the upper limb region and two or more key points in the lower limb region. In addition, the video processing unit 1032 calculates the phase period of each movement phase based on the delimiter positions. Furthermore, the video processing unit 1032 stores the estimated delimiter positions and the calculated phase periods in the phase period information field of the evaluation table 1013 in the storage unit 101. A phase period is the period (time) elapsed from the start to the end of an operation phase. Figure 10 shows the periods of each phase when the functional operation type is a "standing-sitting test." As shown in Figure 10, the elapsed time for the first setting operation is the first setting phase period T1, the elapsed time for the second setting operation is the second setting phase period T2, the elapsed time for the third setting operation is the third setting phase period T3, the elapsed time for the fourth setting operation is the fourth setting phase period T4, and the elapsed time for the fifth setting operation is the fifth setting phase period T5. For simplicity, in the following description, n is a natural number, and the phase period corresponding to the nth phase will be referred to as the nth phase period Tn, and the animation for the period corresponding to the nth phase will be referred to as the nth phase animation. When the functional operation type is a "standing-sitting test," the phase period corresponding to the nth set operation will be referred to as the nth set phase period Tn, and the animation for the period corresponding to the nth animation phase will be referred to as the nth set phase animation.

[0082] If the function operation type is a "stand-up and sit-down test" and the video includes five set movements, in step S202, the video processing unit 1032 acquires the division positions between the first and second phases of the set movement, the first set phase period T1, the second set phase period T2, the third set phase period T3, the fourth set phase period T4, and the fifth set phase period T5, and stores them in the phase period information field of the evaluation table 1013 in the storage unit 101. The acquired and stored phase period information is determined depending on the number of repetitions of the set movement described above. In the specific example of the following steps, the set movement is repeated five times.

[0083] In step S203, the evaluation unit 1034 of the server 10 judges the total movement period Ta acquired from the user terminal 20 in step S101 of the motor function evaluation process using a five-level scoring system (total movement period judgment process). If two or more evaluation videos are shot in one measurement, the total movement period judgment process is executed for the total movement period Ta acquired for each evaluation video. Specifically, the evaluation unit 1034 judges the total operation period Ta on a five-point scale based on a preset reference value (reference range). For example, the evaluation unit 1034 judges the total operation period Ta to be "3 points" when it is equal to the reference value (reference range) Taa, "4 points" when it is slightly better than the reference value (reference range) Taa, "5 points" when it is better than the reference value (reference range) Taa, and conversely, "2 points" when it is slightly worse than the reference value (reference range) Taa, and "1 point" when it is worse than the reference value (reference range) Taa. The evaluation unit 1034 stores the judgment result as the total operation period judgment result in the item of evaluation information in the evaluation table 1013 in the storage unit 101. Note that the judgment of the total operation period is not limited to a five-level scoring system, and may be, for example, only a two-level good / bad evaluation, or a three-level good / bad evaluation.

[0084] In step S204, the evaluation unit 1034 of the server 10 makes a judgment for each n-th phase period Tn calculated in step S202 (phase period judgment step). If two or more evaluation videos are shot in one measurement, the evaluation unit 1034 executes the phase period judgment step for each phase period calculated for each evaluation video. Specifically, the evaluation unit 1034 judges the n-th phase period Tn using a five-level scoring system based on a preset reference value (reference range). Then, the evaluation unit 1034 stores the judgment result in the item of phase period judgment result in the evaluation table 1013 of the storage unit 101.

[0085] When the functional operation type is the "standing-sitting test," the evaluation unit 1034 evaluates the first set phase period T1 on a five-point scale based on a preset reference value (reference range). For example, a reference value (reference range) T1a is evaluated as "3 points," a value slightly better than the reference value (reference range) T1a is evaluated as "4 points," a value better than the reference value (reference range) T1a is evaluated as "5 points," and conversely, a value slightly worse than the reference value (reference range) T1a is evaluated as "2 points," and a value worse than the reference value (reference range) T1a is evaluated as "1 point." The same evaluation is performed for the second set phase period T2 through the fifth set phase period T5. Evaluation is also performed for the first and second phases of each operation set. Evaluation may also be performed for the first and second phases of only the nth operation set. For example, evaluation may also be performed for the first and second phases of the first or last operation set. In this case, the evaluation unit 1034 stores the judgment results of the first phase and the second phase in the evaluation information items of the evaluation table 1013 in the storage unit 101, for example, as "nth first phase judgment result" and "nth second phase judgment result." Note that the phase period operation judgment is not limited to a five-level scoring system, and may be, for example, a two-level pass / fail evaluation, or a three-level or more level evaluation.

[0086] In step S205, the evaluation unit 1034 of the server 10 infers whether the evaluation subject's movement in each movement phase is good or bad based on the key point coordinates acquired in step S201 (phase evaluation step). If two or more videos are taken in one measurement, the phase evaluation step is performed for each video. Specifically, the evaluation unit 1034 uses the keypoint coordinates of each frame of the n-th phase video to calculate, for a predetermined body part, physical quantities including coordinate positions, statistical values ​​of the physical quantities, and calculation results of the physical quantities and / or statistical values ​​as feature quantities. The evaluation unit 1034 applies the calculated feature quantities as input data to the evaluation mathematical model 1022 and obtains the pass / fail of the inferred movement as output data. The evaluation unit 1034 then associates the pass / fail of the inferred movement with the n-th movement phase and stores it in the phase evaluation result field of the evaluation table 1013 in the storage unit 101. When the functional operation type is a "sit-to-stand test," evaluation results based on the variability in the evaluation of multiple set movements may also be included.

[0087] The predetermined parts in this step are selected from the above-mentioned 33 key points and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb), and the number of selected parts is preferably in the range of 3 to 8. The selected predetermined parts are set and registered in advance in the server 10 for each movement phase. The physical quantity of a predetermined body part includes position, distance, and angle. The angle may be a keypoint angle, which is the angle formed by two lines connecting a certain keypoint to two adjacent keypoints. For example, the keypoint angle of the right knee refers to the angle formed by the line connecting the right knee to the right hip and the line connecting the right knee to the right ankle. The angle may also be the angle formed by two lines connecting a certain region to two adjacent regions. For example, the trunk angle refers to the angle formed by the line connecting the center of gravity to the center of the head and neck (which may be the nose) and the line connecting the center of gravity to the tip of one lower limb (which may be the ankle). Hereinafter, the keypoint angle and the region angle are collectively referred to as the body part angle. The statistical values ​​of the physical quantity include the amount of movement, the movement width, the average value, the maximum value, and the minimum value. The calculation includes arithmetic operations. The feature values ​​calculated for the selected predetermined body part are set and registered in the server 10 in advance for each movement phase. The feature amount is defined by selecting from a predetermined part, physical amount, statistical value, or calculation, such as "amount of movement of the center of gravity within a period - width of movement of the center of gravity within a period." In other words, the evaluation result regarding the behavior of the subject can be generated using any of the values ​​or statistical values ​​of the positions of the keypoints, the distance between the keypoints, and the angle formed by the line segments connecting two or more keypoints that connect any of three or more keypoints, or a combination of these.

[0088] When the functional operation type is the "stand-to-sit test," the first phase is also called the standing phase, and the second phase is called the sitting phase. The standing phase is the period in which the subject adjusts their standing posture by tilting their upper body and knees when standing up from a seated position, while the sitting phase is the period in which the subject adjusts their sitting posture by changing the speed of their center of gravity when sitting down from a standing position. In the evaluation report, the tilt of the upper body is referred to as "bowing," the tilt of the knees as "dragging the feet," and the speed of center of gravity movement as "slumping."

[0089] In the evaluation report, "bowing" is an index that indicates the angle at which the trunk tilts forward when standing up. This index indicates whether the standing-up movement can be performed smoothly. Specifically, as shown in Figure 6, if the angle at which the trunk tilts forward (trunk forward tilt angle, or so-called bowing angle) is less than the first forward tilt angle (e.g., a specified angle between 10° and 30°), it becomes difficult to stand up smoothly. If the angle is equal to or greater than the second forward tilt angle (e.g., a specified angle between 30° and 50°), there is a possibility that the subject is compensating for a decrease in lower body muscle strength or knee or hip pain. Ideally, the trunk should be tilted between 15° and 40° when standing up. Furthermore, "pullback" is an index that indicates the angle at which the lower leg is tilted toward the furniture when standing up (the angle at which the lower leg is tilted forward, or the so-called pullback angle). As with "bowing," this index indicates whether the person can smoothly perform the standing-up motion. Specifically, as shown in Figure 6, if the angle of inclination from the heel to the knee with respect to the ground (pullback angle) is equal to or greater than the first pullback angle (e.g., a predetermined angle between 60° and 80°), the ankle is considered stiff, making it difficult to stand up smoothly. Ideally, when standing up, the dorsiflexion angle should be less than a predetermined dorsiflexion angle (e.g., a predetermined angle between 60° and 80°). The "thud sitting" is an index showing the speed at which the center of gravity moves when sitting down. This index shows whether or not a person has the ability to control the speed at which the center of gravity moves when sitting down.

[0090] When the functional operation type is "standing-sitting test," it is preferable to use one or more part angle statistics or their calculation results as input data for the first phase. Similarly, for example, it is preferable to use the center of gravity statistics or their calculation results and one or more part angle statistics or their calculation results as input data for the second phase.

[0091] In step S206, the evaluation unit 1034 of the server 10 infers whether the movement of the specific part of the person to be evaluated in each movement phase is good or bad based on the key point coordinates acquired in step S201 (phase-part evaluation process). If two or more videos are taken in one measurement, the phase evaluation process is performed for each video. Specifically, the evaluation unit 1034 uses the keypoint coordinates of each frame of the nth movement phase moving image to calculate, for a specific body part, physical quantities including coordinate positions, statistical values ​​of the physical quantities, and calculation results of the physical quantities and / or statistical values ​​as feature quantities.The evaluation unit 1034 then applies the calculated feature quantities as input data to the evaluation mathematical model 1022 and obtains the quality of the inferred movement of the specific body part as output data.The evaluation unit 1034 then associates the quality of the inferred movement of the specific body part with the movement phase and stores it in the item "phase-part evaluation result" in the evaluation table 1013 of the storage unit 101. The predetermined site in this step differs for each specific site to be evaluated. The predetermined site in this step is selected from the 33 key points and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb) described above, and the number of selected regions is preferably in the range of 2 to 8. The selected predetermined site is set and registered in advance in server 10 for each specific site to be evaluated.

[0092] When the functional operation type is the "standing and sitting test," in this embodiment, the movements of the specific parts, "shoulder," "hip joint," "knee," "ankle," "heel," and "index finger," are evaluated as good or bad. It is preferable to use, as input data, statistical values ​​of one or more part angles and statistical values ​​of the center of gravity, or their calculation results. For example, using keypoint coordinates of each frame of the nth set of movement phase video, physical quantities including the shoulder position of the person being evaluated, statistical values ​​of those physical quantities, and calculation results of those physical quantities and / or those statistical values ​​are calculated as feature quantities. The feature quantities obtained from the shoulder position are applied as input data to the evaluation mathematical model 1022, and the evaluation unit 1034 obtains the quality of the inferred shoulder movement as output data. The input data for the evaluation of the "ankle" may exclude the physical quantities, statistical values, and calculation results of keypoints of the lower limbs. For example, as shown in Figure 6, in the first phase of "bow," the statistical value of the angle of the line connecting the "shoulder" and "hip joint" with respect to the vertical virtual line P, the statistical value of the center of gravity, or the calculation results of these, is used. Also, in "pull your foot," the statistical value of the angle of the line connecting the "knee" and "heel" with respect to the horizontal virtual line Q, the statistical value of the center of gravity, or the calculation results of these, is used. Note that the key points and six regions used are not limited to these.

[0093] In step S207, the evaluation unit 1034 of the server 10 evaluates the presence or absence of compensatory movements in the functional movements of the person being evaluated based on the converted video (compensatory evaluation process). To be clear, compensatory movements refer to movements that are made using other parts of the body that are different from the original movements to compensate for the insufficient function of muscles or joints that should be used. This compensation occurs due to various factors, such as muscle weakness, limited range of motion of joints, pain, and nerve damage. Therefore, it is generally considered better to have fewer compensatory movements, and this can be included in the evaluation. Compensatory movements include upper limb compensatory movements and lower limb compensatory movements. The evaluation is performed for each movement phase. If two or more videos are taken in one measurement, the total movement period determination process is performed for each converted video corresponding to each video. Specifically, the evaluation unit 1034 calculates, for a predetermined body part, physical quantities including coordinate positions, statistical values ​​of the physical quantities, and calculation results of the physical quantities and / or statistical values ​​as feature quantities using the keypoint coordinates of each frame of the nth movement phase moving image. The evaluation unit 1034 applies the calculated feature quantities as input data to the evaluation mathematical model 1022 and obtains, as output data, the estimated presence or absence of upper limb compensatory movement and / or lower limb compensatory movement. The evaluation unit 1034 then stores the presence or absence of the obtained compensatory movement in the evaluation information field of the evaluation table 1013 in the storage unit 101 of the server 10 as the nth phase compensatory movement evaluation result.

[0094] When the functional operation type is a "standing-sitting test," the evaluation unit 1034 first applies, as input data, feature amounts for a predetermined part of the first set phase video to the evaluation mathematical model 1022, and obtains the estimated presence or absence of upper limb compensatory movement as output data. The obtained output data is stored in the evaluation result item of the evaluation table 1013 of the storage unit 101 of the server 10 as a first phase compensatory movement evaluation result. The second phase is obtained in the same way, and stored in the evaluation result item of the evaluation table 1013 as a second phase compensatory movement evaluation result. The evaluation is also performed for both standing up and sitting down.

[0095] The predetermined areas in this process are selected from the 33 key points mentioned above and six regions (head and neck, trunk, left upper limb, right upper limb, left lower limb, and right lower limb). The number of predetermined areas to be selected is preferably in the range of 2 to 5, and is set in advance for each upper limb compensatory movement and lower limb compensatory movement. Examples of the feature quantity of the upper limb compensatory movement include the movement amount of a predetermined part selected from the upper limb region and statistics of part angles of multiple predetermined parts selected from the upper limb region. Examples of the feature quantity of the lower limb compensatory movement include statistics of the center of gravity coordinates and statistics of multiple part angles. Furthermore, the feature quantity of the lower limb compensatory movement may exclude key points in the lower limb region. When the functional movement type is a "standing and sitting test," the features of the upper limb compensatory movement include selecting at least one part from the upper limb region, such as the "shoulder," "elbow," or "wrist," and statistical values ​​of the position and angle of the selected part.

[0096] In step S208, the evaluation unit 1034 of the server 10 evaluates the relative positional relationship between the person to be evaluated and the environmental information based on the converted video (environment evaluation step). The relative positional relationship is the positional relationship between the person to be evaluated and the furniture. Specifically, the evaluation unit 1034 calculates the positional relationship based on the environmental information acquired by the environmental information acquisition unit 1033 and the coordinate positions of predetermined parts of the person being evaluated that appear in the converted video. The calculated positional relationship is applied as input data to the evaluation mathematical model 1022, and an estimated relative positional relationship is acquired as output data. The evaluation unit 1034 then stores the acquired relative positional relationship in the evaluation information item of the evaluation table 1013 in the storage unit 101 of the server 10 as a relative positional relationship evaluation result. When the functional operation type is a "sit-stand test," the relative positional relationship refers to the subject's seated position, i.e., whether the subject is sitting shallowly or deeply. For example, the distance between the legs of the furniture and the area below the knees can be used to assess whether the subject is sitting shallowly or deeply. In this case, feature points on the edge or edge of the furniture are extracted from the converted video, and the horizontal distance from the feature points to the knee key point is calculated. If the distance is greater than a predetermined distance, the subject is considered to be sitting shallowly, and if the distance is less than the predetermined distance, the subject is considered to be sitting deeply. The seating depth may be determined as three or more levels, such as shallow, normal, or deep, by setting a threshold. If the furniture has a backboard, it is possible to evaluate whether the subject is leaning on the backboard based on the distance between the backboard and the subject's back. In this case, feature points of the backboard are extracted from the converted video, and the horizontal distance from the feature points to the shoulder key points is calculated. If the distance is less than a predetermined distance, it is determined that the subject is leaning on the backboard, and if the distance is greater than the predetermined distance, it is determined that the subject is not leaning on the backboard.

[0097] In step S209, the evaluation unit 1034 of the server 10 generates a comprehensive evaluation from the determination result of the total movement period Ta in step S203, the determination result of each phase period Tn in step S204, the evaluation result of each phase in step S205, the evaluation result by each phase part in step S206, and the compensatory movement evaluation result in step S207 (evaluation result integration process). The comprehensive evaluation is an index showing the evaluation of the quality of motor function, generated, for example, by scoring each result and performing tabulation and statistical processing. The comprehensive evaluation may be generated using the evaluation mathematical model 1022.

[0098] <Example output> 11 is a diagram showing an example of an evaluation report. The evaluation report may be displayed on the display 2071 of the user terminal 20, or on the display of another user terminal (not shown) used by the elderly person or a third party, or may be printed out on paper. By providing these evaluation reports, etc. to the person being evaluated, their family, and caregivers, the user can improve trust and facilitate communication.

[0099] The evaluation report in FIG. 11 illustrates an example of the evaluation results (indicated as "measurement results" in FIG. 11) for the "standing and sitting movement test" among the functional operation types. The evaluation report in FIG. 11 is the evaluation result when a set of standing and sitting movements is repeated five times. The "Leg Score" displays the current result as a score from 1 to 5 points, which is the overall evaluation of the evaluation information in the evaluation table 1013. "1 point" indicates that the set movement could not be performed, "2 points" indicates that the set movement was performed at least once but less than five times, "3 points" indicates that there was hand assistance but the set movement was completed five times and reached the standard (average) value Tna, "4 points" indicates that there was no hand assistance but the set movement was completed five times and reached the standard (average) value Tna, and "5 points" indicates that the arm was folded in front of the chest and was better than the standard (average) value Tna. The "Overall Comment" displays feedback data generated based on the evaluation information. The "leg score" displays an overall evaluation using a score from 1 to 5, but is not limited to a 5-point scoring system, and may be a 2-point evaluation of good or bad, or a 3-point or higher level evaluation. Also, although the evaluation is based on the presence or absence of hand assistance, the evaluation may be performed without the presence or absence of hand assistance, and the results may be displayed. "Change in leg score" and "Change in seconds" are displayed in graph form, showing the history of the assessment results for the entire movement period. The results of "Change in leg score" and "Change in seconds" are displayed as "Decrease," "Maintain," or "Improvement" compared to the previous results. "Standing up state" displays the current and previous results of "light bow," "appropriate," and "deep bow" in the evaluation information of "bow" in the evaluation table 1013. Similarly, it displays the current and previous results of "insufficient pulling" and "sufficient pulling" in the evaluation information of "bowing." "Comments" displays feedback data generated based on the evaluation information of "bow" and "pull." In the "improvement advice" section, feedback data generated for "functional aspects" and "environmental aspects" based on the overall evaluation of the evaluation information in the evaluation table 1013 is displayed.

[0100] A list of exercise menus is also displayed (not shown) based on the overall evaluation of the evaluation information in the evaluation table 1013. For example, the list of exercise menus shows recommended exercises categorized into balance exercises and movement training and strength and flexibility training. The displayed exercise menu corresponds to the suggested exercise menu information in the report information in the evaluation table 1013. If the person being evaluated is elderly, it is preferable that at least one exercise be presented in the balance exercise and movement training category. When the user or person being evaluated selects an exercise menu on the user terminal 20, the contents of the exercise menu are displayed. Items displayed in the exercise menu include basic information such as "Points to consider," "Difficulty level," "Number of repetitions (number of repetitions and sets)," "Cautions," and "Free description," as well as exercise content that explains the exercise content. The exercise content may be a combination of one or more images and explanatory text, or may be a video.

[0101] As described above, the information processing system 1 according to the present disclosure provides a user or a subject with an evaluation of motor function such as balance, an analysis of the evaluation, and an exercise menu selected based on the evaluation and the analysis of the evaluation, simply by capturing a video and providing minimal input. In other words, the information processing system 1 can easily and accurately evaluate motor function, suggest effective rehabilitation, and improve the quality of rehabilitation. Furthermore, the information processing system 1 allows a user to easily measure the subject's motor function, such as balance, from the user terminal 20.

[0102] <Program> 12 is a schematic block diagram showing the configuration of a computer 801. The computer 801 includes a CPU 802 (processor), a main storage device 803, an auxiliary storage device 804, an interface 805, and a graphics processing unit (GPU) 806.

[0103] Here, the programs for realizing the functions constituting the server 10 according to the above embodiment will be described in detail.

[0104] The server 10 is implemented in a computer 801. The operation of each component of the server 10 is stored in the form of a program in an auxiliary storage device 804. The CPU 802 reads the program from the auxiliary storage device 804, loads it into the main storage device 803, and executes the above-mentioned processing in accordance with the program. The CPU 802 also allocates a storage area corresponding to the above-mentioned storage unit 101 in the main storage device 803 or the auxiliary storage device 804 in accordance with the program.

[0105] Specifically, the program is a program to be executed by a computer having a processor and a memory unit, and the program causes the processor to execute the following steps: acquire an evaluation video that captures the entire body of the person being evaluated performing standing-up and sitting-down movements and duration information of the standing-up and sitting-down movements; estimate key points, including joint points and center of gravity, of the person being evaluated that appear in the evaluation video; and generate evaluation results of the muscle strength and balance ability of the person being evaluated using a mathematical model from the key points and the elapsed time of the standing-up and sitting-down movements, wherein the standing-up and sitting-down movements include a standing phase and a sitting phase as movement phases, and the step of generating the evaluation results is a step of generating the evaluation results for each movement phase.

[0106] The auxiliary storage device 804 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, and a semiconductor memory connected via the interface 805. When this program is distributed to the computer 801 via the network NW, the computer 801 that receives the program may load the program into the main storage device 803 and execute the above-described processing.

[0107] The program may also be one that realizes part of the above-mentioned functions. Furthermore, the program may realize the above-mentioned functions in combination with other programs already stored in the auxiliary storage device 804, so-called differential files (differential programs). may be.

[0108] Although the embodiments of the present invention have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0109] Although the description of the embodiment of the present invention has been completed above, the aspects of the present invention are not limited to this embodiment.

[0110] For example, in the measurement procedure in the above embodiment, the video processing unit 1032 may recognize specific body parts of the person being evaluated that appear in the evaluation video and objects around the person being evaluated while the evaluation video is being captured, and the video acquisition unit 1031 may automatically end the capture of the evaluation video when the positional relationship between the specific body part and the surrounding objects satisfies a predetermined condition.

[0111] In addition, in the procedure for measuring motor function in the above embodiment, the user measures the total movement period Ta, but the video acquisition unit 1031 may recognize the movement of the subject in real time and calculate the total movement period Ta, which makes the measurement easier.

[0112] In addition, in step S209 of the above embodiment, the overall evaluation was generated from the judgment results for the entire movement period, the judgment results for each phase period, the overall evaluation result for each phase, the part-by-part evaluation results for each phase, and the compensatory movement evaluation result, but the overall evaluation may be one of these judgment results or evaluation results, or may be generated from a combination of two or more of them.

[0113] Furthermore, in step S202 of the above embodiment, the division positions of the operation phases were estimated using all frames of the converted video, but it is not necessary to use all frames of the converted video as long as they cover the range from the start to the end of the functional operation; only a portion of the frames may be used.

[0114] Furthermore, in the above embodiment, no score is provided for the relative positional relationship evaluation result, but a relative position score with respect to the furniture corresponding to the relative positional relationship evaluation result may be provided.

[0115] <Low load evaluation function 1> However, the conventional five-time sit-to-stand test places a significant physical burden on the subject. In particular, it may not always produce appropriate evaluation results for people with limited mobility, such as dementia patients and the elderly. Therefore, if it were possible to evaluate motor ability with a fewer number of sit-to-stand tests, such as one or two, more people would be able to undergo the evaluation more easily. Therefore, the inventors investigated whether it would be possible to perform an evaluation equivalent to that using a video with a regular number of exercises using a video with a smaller number of exercises.

[0116] According to this, if it is possible to prepare (1) video data (full video data) of a standing-sitting test in which the exercise is repeated five or more times, with the specified number of times being five (five cycles), (2) video data (partial video data) cut out from the video of a small number of times, such as one or two times, from (1), and (3) an evaluation (evaluation data) calculated based on (1), then by machine learning the correlation between the partial video data and the evaluation data, evaluation data equivalent to that which can be obtained from the full video data can be obtained from the partial video data.

[0117] This will be explained in detail with reference to Fig. 13. Fig. 13 is a flowchart for explaining the above method. In step S301, the video acquisition unit 1031, video processing unit 1032, and evaluation unit 1034 operate to acquire video data (full video data = normal load video data) of the (5-cycle) standing-sitting test exercise, and use this to obtain an evaluation (evaluation data). In step S302, for example, the video processing unit 1032 cuts out a video of a small number of movements (at least one but less than five) from the full video data to create partial video data. This partial video data is video data linked to the full video data. Therefore, the evaluation data of the full video data can be linked to the partial video data cut out from the full video data to create learning data. Note that the partial video data may include key point information estimated by the video processing unit 1032, as well as physical quantities and period information thereof. In step S303, machine learning is performed using learning data that is a set of partial video data and evaluation data, to obtain a trained model, low load estimation model 1024. With this low load estimation model 1024, by inputting video data of a small number of exercises, that is, one or more times but less than five times, evaluation data can be obtained as output data. This low load estimation model 1024 basically only needs to be trained once, and the trained model data can be stored in storage unit 101, or it can be called up via an API or the like. In step S304, low-load exercise evaluation data with improved evaluation accuracy can be obtained by machine learning by inputting a video of a small number of exercises (less than five times, but at least one time) into the low-load estimation model 1024. In actual operation, there is no need to perform machine learning each time, and evaluation data can be obtained by the processing from step S304 onwards.

[0118] Therefore, the video acquisition unit 1031 acquires, as the evaluation video, a low-load exercise video that captures the entire body of the person being evaluated who has performed standing up and sitting down movements from a seated position at least once but less than five times, and the evaluation unit 1034 uses, as the evaluation video, a normal-load exercise video that captures the entire body of the person being evaluated who has performed standing up and sitting down movements from a seated position at least five times, and outputs an evaluation that is linked to the low-load exercise video and a low-load estimation model 1024, which is a trained model that has been machine-learned, and generates an evaluation result regarding the movements of the person being evaluated, using the low-load exercise video.

[0119] <Low load evaluation function 2> However, the conventional five-times sit-to-stand test requires the subject to intentionally perform high-intensity exercises, which require the subject to perform effortful exercises (effortful speeds) that place a heavier load on the subject than natural exercises (natural speeds) that are not particularly conscious of the exercise. Generally, the time required to complete one exercise session at an effortful speed is shorter than the time required to complete one exercise session at a natural speed. Therefore, the inventors investigated whether it would be possible to perform an evaluation equivalent to that of an exercise video at an effortful speed using an exercise video at a natural speed.

[0120] According to this method, a predetermined period (reference exercise time) is determined based on the length of a video of an exercise performed at a reference effort speed. A video of an exercise performed at a natural speed for a longer period is shortened, and the natural exercise video is normalized to match the duration of the effort speed video to obtain a normalized natural exercise video. The evaluation unit 1034 can then use this normalized natural exercise video to perform an evaluation equivalent to that of an exercise video performed at an effort speed. Here, because the difference in speed between the effort speed and the natural speed differs between the first phase, which is the standing phase, and the second phase, which is the sitting phase, it is preferable to perform normalization to shorten the phase duration for each phase. For example, this is because the difference in time between sitting down naturally and sitting down with effort is smaller than the difference in time between standing up naturally and standing up with effort. For time normalization, for example, a method can be used in which the natural exercise video is converted into an effort speed video using linear retiming or nonlinear retiming for each phase duration, and the converted video is used.

[0121] This will be explained in detail with reference to Fig. 14. Fig. 14 is a flowchart for explaining the above method. In step S401, the video acquisition unit 1031 acquires a long video that exceeds the length of the acquired predetermined period. In step S402, the video processing unit 1032 performs a normalization process to normalize the acquired long video whose length exceeds the predetermined period to match the length of the predetermined period. The subsequent processes are the same as those described above, and therefore will not be described further.

[0122] Therefore, the video processing unit 1032 performs a normalization process to normalize the acquired long video that exceeds the length of the specified period to match the length of the specified period, and the normalization process is performed at least for the first phase, which is the standing phase, and the second phase, which is the sitting phase.

[0123] <Partial posture evaluation function> Although the five-times-stand-sit test is basically an exercise performed with only the five-times-stand-sit test as an evaluation item, since it is a high-stress exercise as described above, it is beneficial for the subject to be able to evaluate other physical functions in parallel. An example of a physical function that can be considered for parallel evaluation is posture. Therefore, the inventors considered extracting and evaluating posture at any point during the five-times-stand-sit test, if there is anything useful to evaluate.

[0124] According to this, a specific posture can be detected using any of the values ​​or statistical values ​​of the positions of the key points, the distance between the key points, and the angle between two or more line segments connecting any of three or more key points, or a combination of these, or the posture at the start or end of the first phase, which is the standing phase, or the second phase, which is the sitting phase, can be detected, and posture evaluation can be performed for that posture using the key points.

[0125] This will be specifically explained using Figure 15. Figure 15 is a flowchart for explaining the above method. In step S501, a predetermined posture, or a posture at the start or end of the first phase, which is the standing phase, or the second phase, which is the sitting phase, is detected using any of the values ​​or statistical values ​​of the positions of the key points, the distance between the key points, and the angle formed by the line segments connecting any of the three or more key points, or a combination of these. In step S502, an evaluation result regarding the posture of the person to be evaluated is generated.

[0126] As a result, the evaluation unit generates an evaluation result regarding the posture of the person being evaluated for a specified posture or the posture at the start or end of the first phase, which is the standing phase, or the second phase, which is the sitting phase, detected using any of the values ​​or statistical values ​​of the positions of the key points, the distance between the key points, and the angle between two or more line segments connecting any of the three or more key points, or a combination of these.

[0127] <Additional Notes> An example of the configuration of this embodiment is as follows. [1] a video acquisition unit that acquires an evaluation video in which the whole body of the subject undergoing evaluation performs a standing-up and sitting-down motion from a seated state; Using the evaluation video, estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; Using the time change of the key points, an inference is made to divide the standing-up / sitting-down motion into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as motion phases; a motion picture processing unit that calculates a phase period from the start to the end of the motion phase; an evaluation unit that generates an evaluation result regarding the movement of the subject using the key points, which include the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region; Information processing device. [2] The evaluation unit generates the evaluation result regarding the behavior of the person to be evaluated using any one of values ​​or statistical values ​​of the positions of the key points, the distances between the key points, and the angles formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof. [1] The information processing device according to [1]. [3] The evaluation unit evaluates the presence or absence of a compensatory movement including an upper limb compensatory movement or a lower limb compensatory movement in the functional movement of the evaluation subject, generating the evaluation result regarding the movement of the evaluation subject using the evaluation of the presence or absence of the compensatory movement; [2] The information processing device according to [2]. [4] the evaluation unit evaluates the presence or absence of the upper limb compensatory movement using any one of values ​​or statistical values ​​of the positions of the key points in the upper limb region, the distance between the key points, and the angle formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof; or The presence or absence of the lower limb compensatory movement is evaluated using any one of the values ​​or statistical values ​​of the positions of the key points in the lower limb region, the distance between the key points, and the angle formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof. [3] The information processing device according to [3]. [5] the video acquisition unit acquires, as an evaluation video, a low-load exercise video in which the evaluation subject's whole body is captured while the subject stands up and sits down from a seated position at least once but less than five times; The evaluation unit uses a normal load exercise video, which is an image of the whole body of the evaluation subject who has performed standing and sitting movements from a seated position five or more times, as an evaluation video, and outputs an evaluation by linking the evaluation to the low load exercise video and a trained model that has been machine-learned, using the low load exercise video and the low load exercise video, The information processing device according to [1] generates the evaluation result regarding the behavior of the person being evaluated. [6] the video processing unit performs a normalization process to normalize the acquired long video exceeding a predetermined period of time to match the length of the predetermined period of time; The information processing device according to [1], wherein the normalization process is performed at least for a first phase, which is a standing phase, and a second phase, which is a sitting phase. [7] The evaluation unit detects a predetermined posture using any one of the values ​​or statistical values ​​of the positions of the key points, the distances between the key points, and the angles formed by line segments connecting any of three or more key points, or a combination thereof, or Regarding the posture at the start or end of the first phase, which is the standing phase, or the second phase, which is the sitting phase, [2] The method according to [2], wherein an evaluation result regarding the posture of the evaluation subject is generated. Information processing device. [8] An information processing method executed by an information processing device including a processor and a storage unit, The processor acquires an evaluation video capturing an entire body of the subject performing a standing-up and sitting-down motion from a seated state; the video processing unit uses the evaluation video to estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; the video processing unit performs an inference using the time change of the key points to divide the standing-up and sitting-down movements into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as movement phases; a step in which the motion picture processing unit calculates a phase period from the start to the end of the motion phase; and a step in which an evaluation unit generates an evaluation result regarding the movement of the person to be evaluated using the key points including the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region. Information processing methods. [9] A program to be executed by a computer having a processor and a storage unit, The processor acquires an evaluation video capturing an entire body of the subject performing a standing-up and sitting-down motion from a seated state; the video processing unit uses the evaluation video to estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; the video processing unit performs an inference using the time change of the key points to divide the standing-up and sitting-down movements into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as movement phases; a step in which the motion picture processing unit calculates a phase period from the start to the end of the motion phase; and a step in which an evaluation unit generates an evaluation result regarding the movement of the person to be evaluated using the key points including the center of gravity, two or more key points in the upper limb region, and two or more key points in the lower limb region, to cause a computer to execute the steps. program.

[10] An information processing system including an information processing device having a processor and a storage unit, and a user terminal that captures an image of the whole body of an evaluation subject performing a standing-up and sitting-down motion from a seated state, The information processing terminal device a video acquisition unit that acquires an evaluation video in which the whole body of the subject undergoing evaluation performs a standing-up and sitting-down motion from a seated state; Using the evaluation video, estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; Using the time change of the key points, an inference is made to divide the standing-up / sitting-down motion into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as motion phases; a motion picture processing unit that calculates a phase period from the start to the end of the motion phase; an evaluation unit that generates an evaluation result regarding the movement of the subject using the key points, which include the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region; Information processing system. [Explanation of symbols]

[0128] 1. Information Processing Systems 10 Information Processing Server 20 User terminal 30 Furniture 101 Storage section 103 Control Unit 201 Storage section 203 Control Unit 205 Input Device 207 Output Device 801 Computer 802 CPU 803 Main storage 804 Auxiliary storage 805 Interface 1011 Application Program 1012 Video Table 1013 Evaluation Table 1014 Exercise Information Table 1015 User Table 1016 Evaluation Target Table 1021 Mathematical Model for Video Analysis 1022 Evaluation Mathematical Model 1023 Proposed mathematical model 1031 Video Acquisition Unit 1032 Video Processing Unit 1033 Environmental Information Acquisition Department 1034 Evaluation Department 1035 Exercise proposal department 1036 Evaluation result output unit 1037 Mathematical Model Management Department 2011 Application Program 2012 User ID 2031 Input control section 2032 Output control section 2051 Camera 2052 Mike 2053 Location Sensor 2054 Motion Sensor 2055 Touch Device 2071 Display 2072 Speaker NW Network P vertical virtual line Q Horizontal virtual line

Claims

1. a video acquisition unit that acquires an evaluation video in which the whole body of the subject undergoing evaluation performs a standing-up and sitting-down motion from a seated state; Using the evaluation video, estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; Using the time change of the key points, an inference is made to divide the standing-up / sitting-down motion into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as motion phases; a motion picture processing unit that calculates a phase period from the start to the end of the motion phase; an evaluation unit that generates an evaluation result regarding the movement of the subject using the key points, which include the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region; Information processing device.

2. The evaluation unit generates the evaluation result regarding the behavior of the person to be evaluated using any one of values ​​or statistical values ​​of the positions of the key points, the distances between the key points, and the angles formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof. The information processing device according to claim 1 .

3. The evaluation unit evaluates the presence or absence of a compensatory movement including an upper limb compensatory movement or a lower limb compensatory movement in the functional movement of the evaluation subject, generating the evaluation result regarding the movement of the evaluation subject using the evaluation of the presence or absence of the compensatory movement; The information processing device according to claim 2 .

4. the evaluation unit evaluates the presence or absence of the upper limb compensatory movement using any one of values ​​or statistical values ​​of the positions of the key points in the upper limb region, the distance between the key points, and the angle formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof; or The presence or absence of the lower limb compensatory movement is evaluated using any one of values ​​or statistical values ​​of the positions of the key points in the lower limb region, the distance between the key points, and the angle formed by line segments connecting any of three or more key points between two or more of the key points, or a combination thereof. The information processing device according to claim 3 .

5. the video acquisition unit acquires, as an evaluation video, a low-load exercise video in which the evaluation subject's whole body is captured while the subject stands up and sits down from a seated position at least once but less than five times; The evaluation unit uses a normal load exercise video, which is an image of the whole body of the evaluation subject who has performed standing-up and sitting-down movements from a seated position five or more times, as an evaluation video, and outputs an evaluation by linking the evaluation to the low load exercise video and a trained model that has been machine-learned, using the low load exercise video and the low load exercise video, The information processing device according to claim 1 , wherein the evaluation result is generated regarding the behavior of the person to be evaluated.

6. the video processing unit performs a normalization process to normalize the acquired long video exceeding a predetermined period of time to match the length of the predetermined period of time; The information processing apparatus according to claim 1 , wherein the normalization process is performed at least for a first phase that is a standing phase and a second phase that is a sitting phase.

7. The evaluation unit detects a predetermined posture using any one of the values ​​or statistical values ​​of the positions of the key points, the distances between the key points, and the angles formed by line segments connecting any of three or more key points, or a combination thereof, or Regarding the posture at the start or end of the first phase, which is the standing phase, or the second phase, which is the sitting phase, The method according to claim 2, wherein an evaluation result regarding the posture of the person to be evaluated is generated. Information processing device.

8. An information processing method executed by an information processing device including a processor and a storage unit, The processor acquires an evaluation video capturing an entire body of the subject performing a standing-up and sitting-down motion from a seated state; the video processing unit uses the evaluation video to estimate key points of the body of the person to be evaluated that appear in the evaluation video, including the center of gravity; the video processing unit performs an inference using the time change of the key points to divide the standing-up and sitting-down movements into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as movement phases; a step in which the motion picture processing unit calculates a phase period from the start to the end of the motion phase; and a step in which an evaluation unit generates the evaluation result regarding the movement of the person to be evaluated using the key points including the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region. Information processing methods.

9. A program to be executed by a computer having a processor and a storage unit, The processor acquires an evaluation video capturing an entire body of the subject performing a standing-up and sitting-down motion from a seated state; the video processing unit uses the evaluation video to estimate key points of the body of the person to be evaluated that appear in the evaluation video, including the center of gravity; the video processing unit performs an inference using the time change of the key points to divide the standing-up and sitting-down movements into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as movement phases; a step in which the motion picture processing unit calculates a phase period from the start to the end of the motion phase; and a step in which an evaluation unit generates the evaluation result regarding the movement of the person to be evaluated using the key points including the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region, to cause a computer to execute the step. program.

10. An information processing system including an information processing device having a processor and a storage unit, and a user terminal that captures an image of the whole body of an evaluation subject performing a standing-up and sitting-down movement from a seated state, The information processing terminal device a video acquisition unit that acquires an evaluation video in which the whole body of the subject undergoing evaluation performs a standing-up and sitting-down motion from a seated state; Using the evaluation video, estimate key points of the body of the person being evaluated that appear in the evaluation video, including the center of gravity; Using the time change of the key points, an inference is made to divide the standing-up / sitting-down motion into at least a first phase, which is a standing-up phase, and a second phase, which is a sitting-down phase, as motion phases; a motion picture processing unit that calculates a phase period from the start to the end of the motion phase; an evaluation unit that generates the evaluation result regarding the movement of the subject of evaluation using the key points, which include the center of gravity, two or more key points in an upper limb region, and two or more key points in a lower limb region; Information processing system.

Images